2023-10-30T13:19:00.811 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T13:19:00.812 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --name leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi --archive /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046 --verbose --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/orig.config.yaml 2023-10-30T13:19:00.862 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T13:19:00.945 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046 branch: wip-lusov-balance-after-stop description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7441046' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-balance-after-stop ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-balance-after-stop sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 owner: scheduled_leonidus@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 8397 sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 sleep_before_teardown: 0 subset: 464/1024 suite: fs suite_branch: wip-lusov-balance-after-stop suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVLU+jSgOeJeW7UG/jbVLLJdv0luZt06WSrUBK/zLu/CZnGWK3S1K+PNeK43Y3u0/bIANzRVtLcV7EpzeK5vxA= smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGasMAKSbdhk/Ddh2VZPS5CPuVlwKIWRX7nVm+pw5RdOWvQBxabSh4+BgLboPQkMclKOfNZsf1BkjclIvpQzn0M= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_06:34:16 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T13:19:00.946 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa; will attempt to use it 2023-10-30T13:19:00.947 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks 2023-10-30T13:19:00.947 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T13:19:00.949 INFO:teuthology.task.internal:Checking packages... 2023-10-30T13:19:00.972 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '00c1c13cea21a686b046584596fbeac5315a7d71' 2023-10-30T13:19:00.972 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T13:19:00.973 INFO:teuthology.packaging:ref: None 2023-10-30T13:19:00.973 INFO:teuthology.packaging:tag: None 2023-10-30T13:19:00.973 INFO:teuthology.packaging:branch: wip-lusov-balance-after-stop 2023-10-30T13:19:00.974 INFO:teuthology.packaging:sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:19:00.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-balance-after-stop 2023-10-30T13:19:01.199 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:19:01.199 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T13:19:01.206 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T13:19:01.206 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T13:19:01.228 INFO:teuthology.task.internal:Saving configuration 2023-10-30T13:19:01.246 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T13:19:01.251 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T13:19:01.276 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:13:36.472144', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMsSGOc9ewzseY4MR+/6lDIuHCMKosmQNtS3MfHoBoZN'} 2023-10-30T13:19:01.301 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:13:36.473675', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICIwJMhtm+2neEpa7Ce3U93SPc4yHOk1dBXwXbHIaBt7'} 2023-10-30T13:19:01.302 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T13:19:01.308 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-30T13:19:01.309 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-30T13:19:01.309 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T13:19:01.377 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f164938cc10>, signals=[15]) 2023-10-30T13:19:01.378 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T13:19:01.383 INFO:teuthology.task.internal:Opening connections... 2023-10-30T13:19:01.384 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T13:19:01.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:19:01.461 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T13:19:01.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:19:01.538 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T13:19:01.544 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-10-30T13:19:01.550 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-10-30T13:19:01.551 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-10-30T13:19:01.596 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T13:19:01.596 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Ubuntu" 2023-10-30T13:19:01.596 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="22.04" 2023-10-30T13:19:01.597 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T13:19:01.597 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_CODENAME=jammy 2023-10-30T13:19:01.597 INFO:teuthology.orchestra.run.smithi031.stdout:ID=ubuntu 2023-10-30T13:19:01.597 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE=debian 2023-10-30T13:19:01.597 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T13:19:01.598 INFO:teuthology.orchestra.run.smithi031.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T13:19:01.598 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T13:19:01.598 INFO:teuthology.orchestra.run.smithi031.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T13:19:01.598 INFO:teuthology.orchestra.run.smithi031.stdout:UBUNTU_CODENAME=jammy 2023-10-30T13:19:01.599 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-10-30T13:19:01.619 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2023-10-30T13:19:01.624 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2023-10-30T13:19:01.625 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2023-10-30T13:19:01.676 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T13:19:01.676 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2023-10-30T13:19:01.677 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="22.04" 2023-10-30T13:19:01.677 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T13:19:01.677 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=jammy 2023-10-30T13:19:01.677 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2023-10-30T13:19:01.677 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2023-10-30T13:19:01.678 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T13:19:01.678 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T13:19:01.678 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T13:19:01.678 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T13:19:01.678 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=jammy 2023-10-30T13:19:01.679 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-10-30T13:19:01.702 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T13:19:01.711 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T13:19:01.719 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T13:19:01.719 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-10-30T13:19:01.722 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2023-10-30T13:19:01.726 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T13:19:01.731 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T13:19:01.732 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-10-30T13:19:01.769 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2023-10-30T13:19:01.780 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T13:19:01.828 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T13:19:01.843 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-30T13:19:01.844 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-10-30T13:19:01.844 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-10-30T13:19:01.845 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T13:19:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:19:01.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2023-10-30T13:19:02.098 DEBUG:teuthology.task.kernel:sha1 for client.0 is eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:19:02.098 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "eacf00cc51e204f401b4d5d4f84825a68f373c74"... 2023-10-30T13:19:02.098 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-10-30T13:19:02.104 INFO:teuthology.orchestra.run.smithi031.stdout:5.15.0-56-generic 2023-10-30T13:19:02.104 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:19:02.104 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-30T13:19:02.105 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2023-10-30T13:19:02.105 DEBUG:teuthology.orchestra.run.smithi031:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-30T13:19:02.163 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-30T13:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-30T13:19:02.322 INFO:teuthology.orchestra.run.smithi031.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-30T13:19:02.322 INFO:teuthology.orchestra.run.smithi031.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-30T13:19:02.322 INFO:teuthology.orchestra.run.smithi031.stderr:hint: 2023-10-30T13:19:02.322 INFO:teuthology.orchestra.run.smithi031.stderr:hint: git config --global init.defaultBranch 2023-10-30T13:19:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:hint: 2023-10-30T13:19:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-30T13:19:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-30T13:19:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:hint: 2023-10-30T13:19:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:hint: git branch -m 2023-10-30T13:19:02.324 INFO:teuthology.orchestra.run.smithi031.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-30T13:19:02.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-30T13:19:02.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-30T13:19:02.461 DEBUG:teuthology.orchestra.run.smithi031:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-30T13:19:56.280 INFO:teuthology.orchestra.run.smithi031.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-30T13:19:56.281 INFO:teuthology.orchestra.run.smithi031.stderr: * [new branch] main -> origin/main 2023-10-30T13:19:56.304 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20190312 -> 20190312 2023-10-30T13:19:56.305 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20190416 -> 20190416 2023-10-30T13:19:56.305 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20190514 -> 20190514 2023-10-30T13:19:56.305 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20190618 -> 20190618 2023-10-30T13:19:56.306 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20190717 -> 20190717 2023-10-30T13:19:56.306 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20190815 -> 20190815 2023-10-30T13:19:56.307 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20190923 -> 20190923 2023-10-30T13:19:56.307 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20191022 -> 20191022 2023-10-30T13:19:56.307 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20191215 -> 20191215 2023-10-30T13:19:56.307 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200122 -> 20200122 2023-10-30T13:19:56.307 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200316 -> 20200316 2023-10-30T13:19:56.308 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200421 -> 20200421 2023-10-30T13:19:56.308 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200519 -> 20200519 2023-10-30T13:19:56.308 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200619 -> 20200619 2023-10-30T13:19:56.311 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200721 -> 20200721 2023-10-30T13:19:56.311 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200817 -> 20200817 2023-10-30T13:19:56.311 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20200918 -> 20200918 2023-10-30T13:19:56.311 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20201022 -> 20201022 2023-10-30T13:19:56.312 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20201118 -> 20201118 2023-10-30T13:19:56.312 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20201218 -> 20201218 2023-10-30T13:19:56.312 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20210208 -> 20210208 2023-10-30T13:19:56.312 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20210315 -> 20210315 2023-10-30T13:19:56.312 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20210511 -> 20210511 2023-10-30T13:19:56.312 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20210716 -> 20210716 2023-10-30T13:19:56.313 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20210818 -> 20210818 2023-10-30T13:19:56.313 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20210919 -> 20210919 2023-10-30T13:19:56.313 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20211027 -> 20211027 2023-10-30T13:19:56.313 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20211216 -> 20211216 2023-10-30T13:19:56.313 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220209 -> 20220209 2023-10-30T13:19:56.313 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220310 -> 20220310 2023-10-30T13:19:56.314 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220411 -> 20220411 2023-10-30T13:19:56.314 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220509 -> 20220509 2023-10-30T13:19:56.314 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220610 -> 20220610 2023-10-30T13:19:56.314 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220708 -> 20220708 2023-10-30T13:19:56.314 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220815 -> 20220815 2023-10-30T13:19:56.315 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20220913 -> 20220913 2023-10-30T13:19:56.315 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20221012 -> 20221012 2023-10-30T13:19:56.315 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20221109 -> 20221109 2023-10-30T13:19:56.315 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20221214 -> 20221214 2023-10-30T13:19:56.315 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230117 -> 20230117 2023-10-30T13:19:56.316 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230210 -> 20230210 2023-10-30T13:19:56.316 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230310 -> 20230310 2023-10-30T13:19:56.316 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230404 -> 20230404 2023-10-30T13:19:56.316 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230515 -> 20230515 2023-10-30T13:19:56.316 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230625 -> 20230625 2023-10-30T13:19:56.316 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230804 -> 20230804 2023-10-30T13:19:56.317 INFO:teuthology.orchestra.run.smithi031.stderr: * [new tag] 20230919 -> 20230919 2023-10-30T13:20:00.215 INFO:teuthology.orchestra.run.smithi031.stderr:Updating files: 49% (1572/3168) Updating files: 50% (1584/3168) Updating files: 51% (1616/3168) Updating files: 52% (1648/3168) Updating files: 53% (1680/3168) Updating files: 54% (1711/3168) Updating files: 55% (1743/3168) Updating files: 56% (1775/3168) Updating files: 57% (1806/3168) Updating files: 58% (1838/3168) Updating files: 59% (1870/3168) Updating files: 60% (1901/3168) Updating files: 61% (1933/3168) Updating files: 62% (1965/3168) Updating files: 63% (1996/3168) Updating files: 64% (2028/3168) Updating files: 64% (2042/3168) Updating files: 65% (2060/3168) Updating files: 66% (2091/3168) Updating files: 67% (2123/3168) Updating files: 68% (2155/3168) Updating files: 69% (2186/3168) Updating files: 70% (2218/3168) Updating files: 71% (2250/3168) Updating files: 72% (2281/3168) Updating files: 73% (2313/3168) Updating files: 74% (2345/3168) Updating files: 75% (2376/3168) Updating files: 76% (2408/3168) Updating files: 77% (2440/3168) Updating files: 78% (2472/3168) Updating files: 79% (2503/3168) Updating files: 80% (2535/3168) Updating files: 81% (2567/3168) Updating files: 81% (2582/3168) Updating files: 82% (2598/3168) Updating files: 83% (2630/3168) Updating files: 84% (2662/3168) Updating files: 85% (2693/3168) Updating files: 86% (2725/3168) Updating files: 87% (2757/3168) Updating files: 88% (2788/3168) Updating files: 89% (2820/3168) Updating files: 90% (2852/3168) Updating files: 91% (2883/3168) Updating files: 92% (2915/3168) Updating files: 93% (2947/3168) Updating files: 94% (2978/3168) Updating files: 95% (3010/3168) Updating files: 96% (3042/3168) Updating files: 97% (3073/3168) Updating files: 98% (3105/3168) Updating files: 99% (3137/3168) Updating files: 100% (3168/3168) Updating files: 100% (3168/3168), done. 2023-10-30T13:20:00.220 INFO:teuthology.orchestra.run.smithi031.stdout:HEAD is now at 2b304bfe Merge branch 'main' into 'main' 2023-10-30T13:20:00.241 INFO:teuthology.task.kernel:Downloading kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T13:20:00.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:20:00.405 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:20:00.406 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ --input-file=- 2023-10-30T13:20:00.409 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi031.front.sepia.ceph.com to complete... 2023-10-30T13:20:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30 13:20:02 URL:https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb [38909658/38909658] -> "/tmp/linux-image.deb" [1] 2023-10-30T13:20:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:FINISHED --2023-10-30 13:20:02-- 2023-10-30T13:20:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:Total wall clock time: 2.0s 2023-10-30T13:20:02.424 INFO:teuthology.orchestra.run.smithi031.stderr:Downloaded: 1 files, 37M in 1.8s (20.9 MB/s) 2023-10-30T13:20:02.426 INFO:teuthology.task.kernel:Installing kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T13:20:02.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo dpkg -i /tmp/linux-image.deb 2023-10-30T13:20:02.732 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-geacf00cc51e2. 2023-10-30T13:20:04.705 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-30T13:20:04.709 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-30T13:20:04.858 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T13:20:08.389 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T13:20:08.482 INFO:teuthology.orchestra.run.smithi031.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:28.789 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub' 2023-10-30T13:20:28.800 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T13:20:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-10-30T13:20:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:29.135 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:29.438 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T13:20:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T13:20:29.514 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T13:20:29.519 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T13:20:29.688 INFO:teuthology.orchestra.run.smithi031.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T13:20:29.688 INFO:teuthology.orchestra.run.smithi031.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T13:20:29.688 INFO:teuthology.orchestra.run.smithi031.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T13:20:29.733 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-10-30T13:20:29.945 DEBUG:teuthology.orchestra.run.smithi031:> dpkg-deb -c /tmp/linux-image.deb 2023-10-30T13:20:29.953 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./ 2023-10-30T13:20:29.953 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./boot/ 2023-10-30T13:20:29.953 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 4415097 2023-10-30 03:46 ./boot/System.map-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:29.965 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 169233 2023-10-30 03:46 ./boot/config-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:29.966 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11352928 2023-10-30 03:46 ./boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:29.987 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/ 2023-10-30T13:20:29.987 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/ 2023-10-30T13:20:29.987 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postinst.d/ 2023-10-30T13:20:29.988 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postrm.d/ 2023-10-30T13:20:29.988 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/preinst.d/ 2023-10-30T13:20:29.988 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/prerm.d/ 2023-10-30T13:20:29.989 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/ 2023-10-30T13:20:29.989 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/ 2023-10-30T13:20:29.989 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:20:29.990 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/ 2023-10-30T13:20:29.990 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/ 2023-10-30T13:20:29.991 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/ 2023-10-30T13:20:29.991 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ 2023-10-30T13:20:29.991 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 490233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-30T13:20:29.995 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-30T13:20:29.996 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-30T13:20:29.996 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-30T13:20:29.996 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 93113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-30T13:20:29.997 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 106729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-30T13:20:29.997 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 152241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-30T13:20:29.998 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-30T13:20:29.998 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-30T13:20:29.998 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 83889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-30T13:20:29.999 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-30T13:20:29.999 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-30T13:20:29.999 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-30T13:20:30.000 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-30T13:20:30.000 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-30T13:20:30.000 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-30T13:20:30.001 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 58585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-30T13:20:30.001 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-30T13:20:30.002 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-30T13:20:30.002 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-30T13:20:30.003 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/ 2023-10-30T13:20:30.003 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/ 2023-10-30T13:20:30.003 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/ 2023-10-30T13:20:30.004 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-30T13:20:30.004 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpuid.ko 2023-10-30T13:20:30.004 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/msr.ko 2023-10-30T13:20:30.005 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/ 2023-10-30T13:20:30.005 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 353761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-30T13:20:30.005 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 881713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-30T13:20:30.006 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 2276817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm.ko 2023-10-30T13:20:30.049 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ 2023-10-30T13:20:30.049 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/af_alg.ko 2023-10-30T13:20:30.049 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_hash.ko 2023-10-30T13:20:30.049 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_skcipher.ko 2023-10-30T13:20:30.050 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ansi_cprng.ko 2023-10-30T13:20:30.050 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/anubis.ko 2023-10-30T13:20:30.050 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/arc4.ko 2023-10-30T13:20:30.050 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/ 2023-10-30T13:20:30.050 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-30T13:20:30.051 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/ 2023-10-30T13:20:30.051 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_memcpy.ko 2023-10-30T13:20:30.051 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_pq.ko 2023-10-30T13:20:30.051 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-30T13:20:30.051 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_tx.ko 2023-10-30T13:20:30.051 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_xor.ko 2023-10-30T13:20:30.052 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/raid6test.ko 2023-10-30T13:20:30.052 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authenc.ko 2023-10-30T13:20:30.052 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authencesn.ko 2023-10-30T13:20:30.052 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blake2b_generic.ko 2023-10-30T13:20:30.052 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_common.ko 2023-10-30T13:20:30.053 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_generic.ko 2023-10-30T13:20:30.053 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/camellia_generic.ko 2023-10-30T13:20:30.053 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast5_generic.ko 2023-10-30T13:20:30.053 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast6_generic.ko 2023-10-30T13:20:30.053 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast_common.ko 2023-10-30T13:20:30.053 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ccm.ko 2023-10-30T13:20:30.054 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cmac.ko 2023-10-30T13:20:30.054 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crc32_generic.ko 2023-10-30T13:20:30.054 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cryptd.ko 2023-10-30T13:20:30.054 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_engine.ko 2023-10-30T13:20:30.054 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_simd.ko 2023-10-30T13:20:30.055 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_user.ko 2023-10-30T13:20:30.055 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cts.ko 2023-10-30T13:20:30.055 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/des_generic.ko 2023-10-30T13:20:30.055 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/dh_generic.ko 2023-10-30T13:20:30.055 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/echainiv.ko 2023-10-30T13:20:30.056 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/essiv.ko 2023-10-30T13:20:30.056 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/fcrypt.ko 2023-10-30T13:20:30.056 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/khazad.ko 2023-10-30T13:20:30.056 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lrw.ko 2023-10-30T13:20:30.056 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4.ko 2023-10-30T13:20:30.057 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4hc.ko 2023-10-30T13:20:30.057 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/md4.ko 2023-10-30T13:20:30.057 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/michael_mic.ko 2023-10-30T13:20:30.057 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcbc.ko 2023-10-30T13:20:30.057 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcrypt.ko 2023-10-30T13:20:30.058 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/rmd160.ko 2023-10-30T13:20:30.058 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/seed.ko 2023-10-30T13:20:30.058 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/serpent_generic.ko 2023-10-30T13:20:30.058 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 149193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tcrypt.ko 2023-10-30T13:20:30.058 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tea.ko 2023-10-30T13:20:30.059 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_common.ko 2023-10-30T13:20:30.059 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_generic.ko 2023-10-30T13:20:30.059 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/vmac.ko 2023-10-30T13:20:30.059 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/wp512.ko 2023-10-30T13:20:30.059 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xcbc.ko 2023-10-30T13:20:30.060 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xor.ko 2023-10-30T13:20:30.060 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xts.ko 2023-10-30T13:20:30.060 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xxhash_generic.ko 2023-10-30T13:20:30.060 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ 2023-10-30T13:20:30.060 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ 2023-10-30T13:20:30.060 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_extlog.ko 2023-10-30T13:20:30.061 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-30T13:20:30.061 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_pad.ko 2023-10-30T13:20:30.061 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/ 2023-10-30T13:20:30.061 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/einj.ko 2023-10-30T13:20:30.061 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ec_sys.ko 2023-10-30T13:20:30.062 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/platform_profile.ko 2023-10-30T13:20:30.062 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbs.ko 2023-10-30T13:20:30.062 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbshc.ko 2023-10-30T13:20:30.062 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 103025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/video.ko 2023-10-30T13:20:30.062 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ 2023-10-30T13:20:30.062 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/acard-ahci.ko 2023-10-30T13:20:30.063 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 88865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci.ko 2023-10-30T13:20:30.063 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci_platform.ko 2023-10-30T13:20:30.063 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 88289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci.ko 2023-10-30T13:20:30.063 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci_platform.ko 2023-10-30T13:20:30.063 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_acpi.ko 2023-10-30T13:20:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ali.ko 2023-10-30T13:20:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_amd.ko 2023-10-30T13:20:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_artop.ko 2023-10-30T13:20:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atiixp.ko 2023-10-30T13:20:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atp867x.ko 2023-10-30T13:20:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd640.ko 2023-10-30T13:20:30.065 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd64x.ko 2023-10-30T13:20:30.065 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cypress.ko 2023-10-30T13:20:30.065 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_efar.ko 2023-10-30T13:20:30.065 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt366.ko 2023-10-30T13:20:30.065 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt37x.ko 2023-10-30T13:20:30.066 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-30T13:20:30.066 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-30T13:20:30.066 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it8213.ko 2023-10-30T13:20:30.066 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it821x.ko 2023-10-30T13:20:30.066 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_jmicron.ko 2023-10-30T13:20:30.067 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_legacy.ko 2023-10-30T13:20:30.067 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_marvell.ko 2023-10-30T13:20:30.067 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_mpiix.ko 2023-10-30T13:20:30.067 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_netcell.ko 2023-10-30T13:20:30.067 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ninja32.ko 2023-10-30T13:20:30.068 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87410.ko 2023-10-30T13:20:30.068 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87415.ko 2023-10-30T13:20:30.068 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_oldpiix.ko 2023-10-30T13:20:30.068 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_opti.ko 2023-10-30T13:20:30.068 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_optidma.ko 2023-10-30T13:20:30.068 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pcmcia.ko 2023-10-30T13:20:30.069 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-30T13:20:30.069 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-30T13:20:30.069 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_piccolo.ko 2023-10-30T13:20:30.069 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_radisys.ko 2023-10-30T13:20:30.069 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rdc.ko 2023-10-30T13:20:30.070 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rz1000.ko 2023-10-30T13:20:30.070 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sch.ko 2023-10-30T13:20:30.070 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_serverworks.ko 2023-10-30T13:20:30.070 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sil680.ko 2023-10-30T13:20:30.070 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sl82c105.ko 2023-10-30T13:20:30.070 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_triflex.ko 2023-10-30T13:20:30.071 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_via.ko 2023-10-30T13:20:30.071 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pdc_adma.ko 2023-10-30T13:20:30.071 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_inic162x.ko 2023-10-30T13:20:30.071 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 83345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_mv.ko 2023-10-30T13:20:30.071 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_nv.ko 2023-10-30T13:20:30.072 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_promise.ko 2023-10-30T13:20:30.072 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_qstor.ko 2023-10-30T13:20:30.072 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil.ko 2023-10-30T13:20:30.072 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil24.ko 2023-10-30T13:20:30.072 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sis.ko 2023-10-30T13:20:30.072 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_svw.ko 2023-10-30T13:20:30.073 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sx4.ko 2023-10-30T13:20:30.073 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_uli.ko 2023-10-30T13:20:30.073 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_via.ko 2023-10-30T13:20:30.073 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_vsc.ko 2023-10-30T13:20:30.073 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ 2023-10-30T13:20:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ks0108.ko 2023-10-30T13:20:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/ 2023-10-30T13:20:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 124473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/bcma.ko 2023-10-30T13:20:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/ 2023-10-30T13:20:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/ 2023-10-30T13:20:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 113665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/aoe.ko 2023-10-30T13:20:30.075 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/ 2023-10-30T13:20:30.075 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 965585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/drbd.ko 2023-10-30T13:20:30.075 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 173225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/floppy.ko 2023-10-30T13:20:30.075 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/ 2023-10-30T13:20:30.075 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 94297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-30T13:20:30.076 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 115793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/nbd.ko 2023-10-30T13:20:30.076 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/ 2023-10-30T13:20:30.076 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 110449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/null_blk.ko 2023-10-30T13:20:30.076 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 98193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/pktcdvd.ko 2023-10-30T13:20:30.076 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 263769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/rbd.ko 2023-10-30T13:20:30.077 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/ 2023-10-30T13:20:30.077 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 98145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-30T13:20:30.077 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/ 2023-10-30T13:20:30.077 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/zram.ko 2023-10-30T13:20:30.077 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ 2023-10-30T13:20:30.078 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/applicom.ko 2023-10-30T13:20:30.078 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hangcheck-timer.ko 2023-10-30T13:20:30.078 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/ 2023-10-30T13:20:30.078 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-30T13:20:30.078 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-30T13:20:30.078 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-30T13:20:30.079 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/via-rng.ko 2023-10-30T13:20:30.079 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-30T13:20:30.079 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ 2023-10-30T13:20:30.079 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-30T13:20:30.079 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 148601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-30T13:20:30.080 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-30T13:20:30.080 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 139313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-30T13:20:30.080 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-30T13:20:30.080 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-30T13:20:30.080 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/lp.ko 2023-10-30T13:20:30.081 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/ 2023-10-30T13:20:30.081 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 84097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/mwave.ko 2023-10-30T13:20:30.081 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/nvram.ko 2023-10-30T13:20:30.081 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ppdev.ko 2023-10-30T13:20:30.081 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tlclk.ko 2023-10-30T13:20:30.081 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/ 2023-10-30T13:20:30.082 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-30T13:20:30.082 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-30T13:20:30.082 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-30T13:20:30.082 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-30T13:20:30.082 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-30T13:20:30.082 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-30T13:20:30.083 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-30T13:20:30.083 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/ 2023-10-30T13:20:30.083 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-30T13:20:30.083 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-30T13:20:30.083 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-30T13:20:30.084 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/ 2023-10-30T13:20:30.084 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-palmas.ko 2023-10-30T13:20:30.084 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-si5351.ko 2023-10-30T13:20:30.084 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-twl6040.ko 2023-10-30T13:20:30.084 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-wm831x.ko 2023-10-30T13:20:30.085 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/ 2023-10-30T13:20:30.085 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-30T13:20:30.085 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-30T13:20:30.085 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-30T13:20:30.085 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ 2023-10-30T13:20:30.085 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ 2023-10-30T13:20:30.086 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-30T13:20:30.086 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 243025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp.ko 2023-10-30T13:20:30.086 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/ 2023-10-30T13:20:30.086 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/ 2023-10-30T13:20:30.086 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-30T13:20:30.087 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 438761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-30T13:20:30.087 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-30T13:20:30.087 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-30T13:20:30.087 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-aes.ko 2023-10-30T13:20:30.087 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-sha.ko 2023-10-30T13:20:30.088 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/ 2023-10-30T13:20:30.088 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 68857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-30T13:20:30.088 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/ 2023-10-30T13:20:30.088 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/dca.ko 2023-10-30T13:20:30.088 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ 2023-10-30T13:20:30.089 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/ 2023-10-30T13:20:30.089 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-30T13:20:30.089 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ 2023-10-30T13:20:30.089 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 122569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-30T13:20:30.089 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ 2023-10-30T13:20:30.089 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 79953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/amd64_edac.ko 2023-10-30T13:20:30.090 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/e752x_edac.ko 2023-10-30T13:20:30.090 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 63617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/edac_mce_amd.ko 2023-10-30T13:20:30.090 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3000_edac.ko 2023-10-30T13:20:30.090 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3200_edac.ko 2023-10-30T13:20:30.090 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5100_edac.ko 2023-10-30T13:20:30.090 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5400_edac.ko 2023-10-30T13:20:30.091 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7300_edac.ko 2023-10-30T13:20:30.091 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 52041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7core_edac.ko 2023-10-30T13:20:30.091 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i82975x_edac.ko 2023-10-30T13:20:30.091 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ie31200_edac.ko 2023-10-30T13:20:30.091 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 56825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/sb_edac.ko 2023-10-30T13:20:30.092 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/x38_edac.ko 2023-10-30T13:20:30.092 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/ 2023-10-30T13:20:30.092 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-gpio.ko 2023-10-30T13:20:30.092 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max14577.ko 2023-10-30T13:20:30.092 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max77693.ko 2023-10-30T13:20:30.092 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max8997.ko 2023-10-30T13:20:30.093 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-palmas.ko 2023-10-30T13:20:30.093 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-30T13:20:30.093 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-30T13:20:30.093 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/ 2023-10-30T13:20:30.093 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-30T13:20:30.094 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/ 2023-10-30T13:20:30.094 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-30T13:20:30.094 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-30T13:20:30.094 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/ 2023-10-30T13:20:30.094 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-30T13:20:30.095 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-30T13:20:30.095 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-arizona.ko 2023-10-30T13:20:30.095 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9052.ko 2023-10-30T13:20:30.095 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9055.ko 2023-10-30T13:20:30.095 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-dln2.ko 2023-10-30T13:20:30.096 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 77849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-30T13:20:30.096 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-generic.ko 2023-10-30T13:20:30.096 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ich.ko 2023-10-30T13:20:30.096 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-30T13:20:30.096 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-kempld.ko 2023-10-30T13:20:30.097 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-30T13:20:30.097 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max7300.ko 2023-10-30T13:20:30.097 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max730x.ko 2023-10-30T13:20:30.097 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max732x.ko 2023-10-30T13:20:30.097 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-30T13:20:30.097 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-30T13:20:30.098 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-30T13:20:30.098 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-30T13:20:30.098 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch.ko 2023-10-30T13:20:30.098 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-30T13:20:30.098 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-30T13:20:30.099 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-30T13:20:30.099 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-30T13:20:30.099 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-30T13:20:30.099 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-vx855.ko 2023-10-30T13:20:30.099 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-30T13:20:30.100 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-30T13:20:30.100 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-30T13:20:30.100 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/ 2023-10-30T13:20:30.100 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-a4tech.ko 2023-10-30T13:20:30.100 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-apple.ko 2023-10-30T13:20:30.101 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-appleir.ko 2023-10-30T13:20:30.101 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-aureal.ko 2023-10-30T13:20:30.101 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-axff.ko 2023-10-30T13:20:30.101 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-belkin.ko 2023-10-30T13:20:30.101 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cherry.ko 2023-10-30T13:20:30.101 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-chicony.ko 2023-10-30T13:20:30.102 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cp2112.ko 2023-10-30T13:20:30.102 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cypress.ko 2023-10-30T13:20:30.102 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-dr.ko 2023-10-30T13:20:30.102 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elecom.ko 2023-10-30T13:20:30.102 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elo.ko 2023-10-30T13:20:30.103 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-emsff.ko 2023-10-30T13:20:30.103 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ezkey.ko 2023-10-30T13:20:30.103 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gaff.ko 2023-10-30T13:20:30.103 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-generic.ko 2023-10-30T13:20:30.103 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gt683r.ko 2023-10-30T13:20:30.104 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gyration.ko 2023-10-30T13:20:30.104 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-30T13:20:30.104 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-30T13:20:30.104 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtekff.ko 2023-10-30T13:20:30.104 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-hyperv.ko 2023-10-30T13:20:30.104 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-icade.ko 2023-10-30T13:20:30.105 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kensington.ko 2023-10-30T13:20:30.105 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-keytouch.ko 2023-10-30T13:20:30.105 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kye.ko 2023-10-30T13:20:30.105 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lcpower.ko 2023-10-30T13:20:30.105 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-led.ko 2023-10-30T13:20:30.106 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lenovo.ko 2023-10-30T13:20:30.106 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lg-g15.ko 2023-10-30T13:20:30.106 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 58185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-30T13:20:30.106 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 101369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-30T13:20:30.106 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 80993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech.ko 2023-10-30T13:20:30.107 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-magicmouse.ko 2023-10-30T13:20:30.107 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-microsoft.ko 2023-10-30T13:20:30.107 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-monterey.ko 2023-10-30T13:20:30.107 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-multitouch.ko 2023-10-30T13:20:30.107 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ntrig.ko 2023-10-30T13:20:30.107 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ortek.ko 2023-10-30T13:20:30.108 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-penmount.ko 2023-10-30T13:20:30.108 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-petalynx.ko 2023-10-30T13:20:30.108 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 72937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-picolcd.ko 2023-10-30T13:20:30.108 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-pl.ko 2023-10-30T13:20:30.108 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-plantronics.ko 2023-10-30T13:20:30.109 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-primax.ko 2023-10-30T13:20:30.109 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-rmi.ko 2023-10-30T13:20:30.109 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-30T13:20:30.109 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-common.ko 2023-10-30T13:20:30.109 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-30T13:20:30.109 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-30T13:20:30.110 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-30T13:20:30.110 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-30T13:20:30.110 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-30T13:20:30.110 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-30T13:20:30.110 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-30T13:20:30.111 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-30T13:20:30.111 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-30T13:20:30.111 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat.ko 2023-10-30T13:20:30.111 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-saitek.ko 2023-10-30T13:20:30.111 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-samsung.ko 2023-10-30T13:20:30.111 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-30T13:20:30.112 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sjoy.ko 2023-10-30T13:20:30.112 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sony.ko 2023-10-30T13:20:30.112 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-speedlink.ko 2023-10-30T13:20:30.112 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-steelseries.ko 2023-10-30T13:20:30.112 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sunplus.ko 2023-10-30T13:20:30.113 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-30T13:20:30.113 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tivo.ko 2023-10-30T13:20:30.113 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tmff.ko 2023-10-30T13:20:30.113 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-topseed.ko 2023-10-30T13:20:30.113 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-twinhan.ko 2023-10-30T13:20:30.114 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 71073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-uclogic.ko 2023-10-30T13:20:30.114 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-waltop.ko 2023-10-30T13:20:30.114 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 143473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-wiimote.ko 2023-10-30T13:20:30.114 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-xinmo.ko 2023-10-30T13:20:30.114 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zpff.ko 2023-10-30T13:20:30.115 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zydacron.ko 2023-10-30T13:20:30.115 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 267825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid.ko 2023-10-30T13:20:30.115 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/uhid.ko 2023-10-30T13:20:30.115 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/ 2023-10-30T13:20:30.115 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 125721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-30T13:20:30.115 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-30T13:20:30.116 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-30T13:20:30.116 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 217873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/wacom.ko 2023-10-30T13:20:30.116 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/ 2023-10-30T13:20:30.116 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 64201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_balloon.ko 2023-10-30T13:20:30.116 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_utils.ko 2023-10-30T13:20:30.117 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 305465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_vmbus.ko 2023-10-30T13:20:30.117 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ 2023-10-30T13:20:30.117 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru.ko 2023-10-30T13:20:30.117 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 61393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru3.ko 2023-10-30T13:20:30.117 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-30T13:20:30.118 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7414.ko 2023-10-30T13:20:30.118 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7418.ko 2023-10-30T13:20:30.118 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adc128d818.ko 2023-10-30T13:20:30.118 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1025.ko 2023-10-30T13:20:30.118 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 97977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1026.ko 2023-10-30T13:20:30.119 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1029.ko 2023-10-30T13:20:30.119 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 45553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1031.ko 2023-10-30T13:20:30.119 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm9240.ko 2023-10-30T13:20:30.119 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ads7828.ko 2023-10-30T13:20:30.119 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7410.ko 2023-10-30T13:20:30.120 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7411.ko 2023-10-30T13:20:30.120 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 74433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7462.ko 2023-10-30T13:20:30.120 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7470.ko 2023-10-30T13:20:30.120 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7475.ko 2023-10-30T13:20:30.120 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7x10.ko 2023-10-30T13:20:30.121 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/amc6821.ko 2023-10-30T13:20:30.121 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 55265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/applesmc.ko 2023-10-30T13:20:30.121 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asb100.ko 2023-10-30T13:20:30.121 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 58617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asc7621.ko 2023-10-30T13:20:30.121 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-30T13:20:30.121 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/atxp1.ko 2023-10-30T13:20:30.122 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/coretemp.ko 2023-10-30T13:20:30.122 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-30T13:20:30.122 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-30T13:20:30.122 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-30T13:20:30.122 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 99601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dme1737.ko 2023-10-30T13:20:30.123 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds1621.ko 2023-10-30T13:20:30.123 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds620.ko 2023-10-30T13:20:30.123 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc1403.ko 2023-10-30T13:20:30.123 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc2103.ko 2023-10-30T13:20:30.123 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc6w201.ko 2023-10-30T13:20:30.124 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71805f.ko 2023-10-30T13:20:30.124 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 94457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71882fg.ko 2023-10-30T13:20:30.124 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f75375s.ko 2023-10-30T13:20:30.124 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fam15h_power.ko 2023-10-30T13:20:30.124 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 55449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fschmd.ko 2023-10-30T13:20:30.124 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g760a.ko 2023-10-30T13:20:30.125 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g762.ko 2023-10-30T13:20:30.125 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl518sm.ko 2023-10-30T13:20:30.125 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 45529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl520sm.ko 2023-10-30T13:20:30.125 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hih6130.ko 2023-10-30T13:20:30.125 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-30T13:20:30.126 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5500_temp.ko 2023-10-30T13:20:30.126 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5k_amb.ko 2023-10-30T13:20:30.126 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmaem.ko 2023-10-30T13:20:30.126 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmpex.ko 2023-10-30T13:20:30.126 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina209.ko 2023-10-30T13:20:30.127 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina2xx.ko 2023-10-30T13:20:30.127 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 130449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/it87.ko 2023-10-30T13:20:30.127 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/jc42.ko 2023-10-30T13:20:30.127 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k10temp.ko 2023-10-30T13:20:30.127 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k8temp.ko 2023-10-30T13:20:30.128 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lineage-pem.ko 2023-10-30T13:20:30.128 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 48601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm63.ko 2023-10-30T13:20:30.128 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm73.ko 2023-10-30T13:20:30.128 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm75.ko 2023-10-30T13:20:30.128 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm77.ko 2023-10-30T13:20:30.128 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm78.ko 2023-10-30T13:20:30.129 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm80.ko 2023-10-30T13:20:30.129 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm83.ko 2023-10-30T13:20:30.129 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 72833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm85.ko 2023-10-30T13:20:30.129 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm87.ko 2023-10-30T13:20:30.129 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm90.ko 2023-10-30T13:20:30.130 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm92.ko 2023-10-30T13:20:30.130 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 108841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm93.ko 2023-10-30T13:20:30.130 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95234.ko 2023-10-30T13:20:30.130 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95241.ko 2023-10-30T13:20:30.130 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95245.ko 2023-10-30T13:20:30.130 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc2945.ko 2023-10-30T13:20:30.131 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4151.ko 2023-10-30T13:20:30.131 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4215.ko 2023-10-30T13:20:30.131 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4222.ko 2023-10-30T13:20:30.131 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4245.ko 2023-10-30T13:20:30.131 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4260.ko 2023-10-30T13:20:30.132 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4261.ko 2023-10-30T13:20:30.132 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max16065.ko 2023-10-30T13:20:30.132 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1619.ko 2023-10-30T13:20:30.132 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1668.ko 2023-10-30T13:20:30.132 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max197.ko 2023-10-30T13:20:30.132 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6639.ko 2023-10-30T13:20:30.133 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6650.ko 2023-10-30T13:20:30.133 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6697.ko 2023-10-30T13:20:30.133 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-30T13:20:30.133 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mcp3021.ko 2023-10-30T13:20:30.133 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-30T13:20:30.134 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6683.ko 2023-10-30T13:20:30.134 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 119817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775-core.ko 2023-10-30T13:20:30.134 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775.ko 2023-10-30T13:20:30.134 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 59737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct7802.ko 2023-10-30T13:20:30.134 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 82337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87360.ko 2023-10-30T13:20:30.135 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87427.ko 2023-10-30T13:20:30.135 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pcf8591.ko 2023-10-30T13:20:30.135 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ 2023-10-30T13:20:30.135 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-30T13:20:30.135 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-30T13:20:30.135 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-30T13:20:30.136 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-30T13:20:30.136 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-30T13:20:30.136 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-30T13:20:30.136 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-30T13:20:30.136 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 82577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-30T13:20:30.136 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-30T13:20:30.137 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-30T13:20:30.137 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-30T13:20:30.137 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-30T13:20:30.137 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/powr1220.ko 2023-10-30T13:20:30.137 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5627.ko 2023-10-30T13:20:30.138 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5636.ko 2023-10-30T13:20:30.138 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-30T13:20:30.138 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht15.ko 2023-10-30T13:20:30.138 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht21.ko 2023-10-30T13:20:30.138 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/shtc1.ko 2023-10-30T13:20:30.138 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sis5595.ko 2023-10-30T13:20:30.139 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47b397.ko 2023-10-30T13:20:30.139 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m1.ko 2023-10-30T13:20:30.139 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m192.ko 2023-10-30T13:20:30.139 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/thmc50.ko 2023-10-30T13:20:30.139 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp102.ko 2023-10-30T13:20:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp103.ko 2023-10-30T13:20:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp401.ko 2023-10-30T13:20:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp421.ko 2023-10-30T13:20:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via-cputemp.ko 2023-10-30T13:20:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via686a.ko 2023-10-30T13:20:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 52457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt1211.ko 2023-10-30T13:20:30.141 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt8231.ko 2023-10-30T13:20:30.141 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627ehf.ko 2023-10-30T13:20:30.141 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627hf.ko 2023-10-30T13:20:30.141 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 71345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83781d.ko 2023-10-30T13:20:30.141 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 63289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83791d.ko 2023-10-30T13:20:30.141 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 71321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83792d.ko 2023-10-30T13:20:30.142 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 101473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83793.ko 2023-10-30T13:20:30.142 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 119505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83795.ko 2023-10-30T13:20:30.142 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l785ts.ko 2023-10-30T13:20:30.142 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l786ng.ko 2023-10-30T13:20:30.142 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-30T13:20:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-30T13:20:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/ 2023-10-30T13:20:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/ 2023-10-30T13:20:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-30T13:20:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-30T13:20:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/ 2023-10-30T13:20:30.144 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-30T13:20:30.144 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-30T13:20:30.144 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-30T13:20:30.144 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-30T13:20:30.144 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-30T13:20:30.144 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-30T13:20:30.145 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-30T13:20:30.145 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 6929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-30T13:20:30.145 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-30T13:20:30.145 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-30T13:20:30.145 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-30T13:20:30.145 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-30T13:20:30.146 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-30T13:20:30.146 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-30T13:20:30.146 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-30T13:20:30.146 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-30T13:20:30.146 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-30T13:20:30.147 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-30T13:20:30.147 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-30T13:20:30.147 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-30T13:20:30.147 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-30T13:20:30.147 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-30T13:20:30.147 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-30T13:20:30.148 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-30T13:20:30.148 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-30T13:20:30.148 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-30T13:20:30.148 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-30T13:20:30.148 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-30T13:20:30.149 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-30T13:20:30.149 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-30T13:20:30.149 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-30T13:20:30.149 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-30T13:20:30.149 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-30T13:20:30.149 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-30T13:20:30.150 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-30T13:20:30.150 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-30T13:20:30.150 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-30T13:20:30.150 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-mux.ko 2023-10-30T13:20:30.150 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-smbus.ko 2023-10-30T13:20:30.150 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-stub.ko 2023-10-30T13:20:30.151 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/ 2023-10-30T13:20:30.151 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-30T13:20:30.151 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-30T13:20:30.151 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-30T13:20:30.151 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ 2023-10-30T13:20:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ 2023-10-30T13:20:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 273609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-30T13:20:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 879969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_core.ko 2023-10-30T13:20:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 69337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-30T13:20:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 324217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-30T13:20:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 97985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-30T13:20:30.153 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 282273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-30T13:20:30.153 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-30T13:20:30.153 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ 2023-10-30T13:20:30.153 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-30T13:20:30.153 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 493073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-30T13:20:30.153 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/ 2023-10-30T13:20:30.154 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 419737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-30T13:20:30.154 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/ 2023-10-30T13:20:30.154 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 739025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-30T13:20:30.154 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ 2023-10-30T13:20:30.154 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 296625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-30T13:20:30.155 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-30T13:20:30.155 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 153697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-30T13:20:30.155 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/ 2023-10-30T13:20:30.155 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 168969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-30T13:20:30.155 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ 2023-10-30T13:20:30.155 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-30T13:20:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 263081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-30T13:20:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ 2023-10-30T13:20:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 100241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-30T13:20:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ 2023-10-30T13:20:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 111425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-30T13:20:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ 2023-10-30T13:20:30.157 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 128753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-30T13:20:30.157 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ 2023-10-30T13:20:30.157 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 129529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-30T13:20:30.157 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ 2023-10-30T13:20:30.157 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/evbug.ko 2023-10-30T13:20:30.158 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ff-memless.ko 2023-10-30T13:20:30.158 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/ 2023-10-30T13:20:30.158 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-30T13:20:30.158 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-30T13:20:30.158 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/gameport.ko 2023-10-30T13:20:30.159 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/joydev.ko 2023-10-30T13:20:30.159 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/ 2023-10-30T13:20:30.159 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-30T13:20:30.159 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-30T13:20:30.159 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-30T13:20:30.159 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-30T13:20:30.160 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-30T13:20:30.160 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-30T13:20:30.160 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-30T13:20:30.160 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8323.ko 2023-10-30T13:20:30.160 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8333.ko 2023-10-30T13:20:30.160 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-30T13:20:30.161 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-30T13:20:30.161 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-30T13:20:30.161 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-30T13:20:30.161 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-30T13:20:30.161 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-30T13:20:30.161 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt1070.ko 2023-10-30T13:20:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt2160.ko 2023-10-30T13:20:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-30T13:20:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/stowaway.ko 2023-10-30T13:20:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-30T13:20:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-30T13:20:30.163 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-30T13:20:30.163 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-30T13:20:30.163 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-30T13:20:30.163 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/matrix-keymap.ko 2023-10-30T13:20:30.163 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/ 2023-10-30T13:20:30.163 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/appletouch.ko 2023-10-30T13:20:30.164 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/bcm5974.ko 2023-10-30T13:20:30.164 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 123681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/cyapatp.ko 2023-10-30T13:20:30.164 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-30T13:20:30.164 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-30T13:20:30.164 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 335977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/psmouse.ko 2023-10-30T13:20:30.242 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/sermouse.ko 2023-10-30T13:20:30.242 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-30T13:20:30.242 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-30T13:20:30.242 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-30T13:20:30.242 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/ 2023-10-30T13:20:30.243 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 109873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-30T13:20:30.243 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ 2023-10-30T13:20:30.243 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/altera_ps2.ko 2023-10-30T13:20:30.243 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/arc_ps2.ko 2023-10-30T13:20:30.243 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ct82c710.ko 2023-10-30T13:20:30.243 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-30T13:20:30.244 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/parkbd.ko 2023-10-30T13:20:30.244 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/pcips2.ko 2023-10-30T13:20:30.244 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ps2mult.ko 2023-10-30T13:20:30.244 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serio_raw.ko 2023-10-30T13:20:30.244 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serport.ko 2023-10-30T13:20:30.245 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/sparse-keymap.ko 2023-10-30T13:20:30.245 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/ 2023-10-30T13:20:30.245 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/ 2023-10-30T13:20:30.245 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-30T13:20:30.245 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/ 2023-10-30T13:20:30.245 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-88pm860x.ko 2023-10-30T13:20:30.246 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-adp5520.ko 2023-10-30T13:20:30.246 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 59753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-bd2802.ko 2023-10-30T13:20:30.246 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-blinkm.ko 2023-10-30T13:20:30.246 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da903x.ko 2023-10-30T13:20:30.246 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da9052.ko 2023-10-30T13:20:30.247 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-gpio.ko 2023-10-30T13:20:30.247 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3530.ko 2023-10-30T13:20:30.247 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3533.ko 2023-10-30T13:20:30.247 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm355x.ko 2023-10-30T13:20:30.247 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3642.ko 2023-10-30T13:20:30.247 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp3944.ko 2023-10-30T13:20:30.248 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp8788.ko 2023-10-30T13:20:30.248 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-max8997.ko 2023-10-30T13:20:30.248 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-mc13783.ko 2023-10-30T13:20:30.248 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-30T13:20:30.248 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca9532.ko 2023-10-30T13:20:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca955x.ko 2023-10-30T13:20:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca963x.ko 2023-10-30T13:20:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-ss4200.ko 2023-10-30T13:20:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-tca6507.ko 2023-10-30T13:20:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-30T13:20:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm8350.ko 2023-10-30T13:20:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ 2023-10-30T13:20:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-30T13:20:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-30T13:20:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-30T13:20:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-30T13:20:30.251 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-30T13:20:30.251 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-30T13:20:30.251 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-30T13:20:30.251 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-30T13:20:30.251 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/ 2023-10-30T13:20:30.251 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/ 2023-10-30T13:20:30.252 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 654969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/bcache.ko 2023-10-30T13:20:30.252 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bio-prison.ko 2023-10-30T13:20:30.252 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bufio.ko 2023-10-30T13:20:30.252 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache-smq.ko 2023-10-30T13:20:30.252 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 156009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache.ko 2023-10-30T13:20:30.252 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 106897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-crypt.ko 2023-10-30T13:20:30.253 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-delay.ko 2023-10-30T13:20:30.253 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-era.ko 2023-10-30T13:20:30.253 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-flakey.ko 2023-10-30T13:20:30.253 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log-userspace.ko 2023-10-30T13:20:30.253 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log.ko 2023-10-30T13:20:30.254 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-mirror.ko 2023-10-30T13:20:30.254 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-multipath.ko 2023-10-30T13:20:30.254 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-queue-length.ko 2023-10-30T13:20:30.254 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-raid.ko 2023-10-30T13:20:30.254 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-region-hash.ko 2023-10-30T13:20:30.255 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-round-robin.ko 2023-10-30T13:20:30.255 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-service-time.ko 2023-10-30T13:20:30.255 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-snapshot.ko 2023-10-30T13:20:30.255 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-switch.ko 2023-10-30T13:20:30.255 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 181865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-thin-pool.ko 2023-10-30T13:20:30.256 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-verity.ko 2023-10-30T13:20:30.256 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-zero.ko 2023-10-30T13:20:30.256 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/faulty.ko 2023-10-30T13:20:30.256 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/linear.ko 2023-10-30T13:20:30.256 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/multipath.ko 2023-10-30T13:20:30.256 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/ 2023-10-30T13:20:30.257 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 199937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-30T13:20:30.257 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid0.ko 2023-10-30T13:20:30.257 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 113073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid1.ko 2023-10-30T13:20:30.257 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 158681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid10.ko 2023-10-30T13:20:30.257 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 395721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid456.ko 2023-10-30T13:20:30.258 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/ 2023-10-30T13:20:30.258 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/ 2023-10-30T13:20:30.258 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 178137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptbase.ko 2023-10-30T13:20:30.258 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 63385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptctl.ko 2023-10-30T13:20:30.258 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptfc.ko 2023-10-30T13:20:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptlan.ko 2023-10-30T13:20:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 123321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptsas.ko 2023-10-30T13:20:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 79041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptscsih.ko 2023-10-30T13:20:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptspi.ko 2023-10-30T13:20:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ 2023-10-30T13:20:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm800.ko 2023-10-30T13:20:30.260 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm805.ko 2023-10-30T13:20:30.260 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm80x.ko 2023-10-30T13:20:30.260 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona-i2c.ko 2023-10-30T13:20:30.260 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 123721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona.ko 2023-10-30T13:20:30.260 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/bcm590xx.ko 2023-10-30T13:20:30.261 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-30T13:20:30.261 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/dln2.ko 2023-10-30T13:20:30.261 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/janz-cmodio.ko 2023-10-30T13:20:30.261 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/kempld-core.ko 2023-10-30T13:20:30.261 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-core.ko 2023-10-30T13:20:30.261 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-30T13:20:30.262 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lp3943.ko 2023-10-30T13:20:30.262 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_ich.ko 2023-10-30T13:20:30.262 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_sch.ko 2023-10-30T13:20:30.262 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/max8907.ko 2023-10-30T13:20:30.262 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-30T13:20:30.262 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-30T13:20:30.263 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/menf21bmc.ko 2023-10-30T13:20:30.263 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-30T13:20:30.263 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-30T13:20:30.263 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633.ko 2023-10-30T13:20:30.263 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-30T13:20:30.263 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/retu-mfd.ko 2023-10-30T13:20:30.264 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 80425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/si476x-core.ko 2023-10-30T13:20:30.264 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 48457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/sm501.ko 2023-10-30T13:20:30.264 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-30T13:20:30.264 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps65010.ko 2023-10-30T13:20:30.264 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps6507x.ko 2023-10-30T13:20:30.265 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/viperboard.ko 2023-10-30T13:20:30.265 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/vx855.ko 2023-10-30T13:20:30.265 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wl1273-core.ko 2023-10-30T13:20:30.265 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wm8994.ko 2023-10-30T13:20:30.265 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ 2023-10-30T13:20:30.265 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-30T13:20:30.266 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 48489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot.ko 2023-10-30T13:20:30.266 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/ 2023-10-30T13:20:30.266 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-30T13:20:30.266 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds9802als.ko 2023-10-30T13:20:30.266 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds990x.ko 2023-10-30T13:20:30.267 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 45593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/bh1770glc.ko 2023-10-30T13:20:30.267 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/ 2023-10-30T13:20:30.267 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-30T13:20:30.267 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/core.ko 2023-10-30T13:20:30.267 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/ 2023-10-30T13:20:30.268 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/cb710.ko 2023-10-30T13:20:30.268 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ds1682.ko 2023-10-30T13:20:30.268 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/dummy-irq.ko 2023-10-30T13:20:30.268 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/ 2023-10-30T13:20:30.268 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/echo.ko 2023-10-30T13:20:30.268 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/ 2023-10-30T13:20:30.269 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/at24.ko 2023-10-30T13:20:30.269 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-30T13:20:30.269 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-30T13:20:30.269 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/max6875.ko 2023-10-30T13:20:30.269 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/enclosure.ko 2023-10-30T13:20:30.270 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/ 2023-10-30T13:20:30.270 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 152201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-30T13:20:30.270 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hmc6352.ko 2023-10-30T13:20:30.270 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hpilo.ko 2023-10-30T13:20:30.270 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ 2023-10-30T13:20:30.270 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 72129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-30T13:20:30.271 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ics932s401.ko 2023-10-30T13:20:30.271 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29003.ko 2023-10-30T13:20:30.271 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29020.ko 2023-10-30T13:20:30.271 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/ 2023-10-30T13:20:30.271 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-30T13:20:30.272 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-30T13:20:30.272 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/ 2023-10-30T13:20:30.272 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 125209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-me.ko 2023-10-30T13:20:30.272 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 56521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-txe.ko 2023-10-30T13:20:30.272 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 310113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei.ko 2023-10-30T13:20:30.272 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/phantom.ko 2023-10-30T13:20:30.273 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/ 2023-10-30T13:20:30.273 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 74265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-30T13:20:30.273 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_7xx1.ko 2023-10-30T13:20:30.273 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_core.ko 2023-10-30T13:20:30.273 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tsl2550.ko 2023-10-30T13:20:30.274 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 48697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_balloon.ko 2023-10-30T13:20:30.274 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/ 2023-10-30T13:20:30.274 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 183609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-30T13:20:30.274 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ 2023-10-30T13:20:30.274 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/ 2023-10-30T13:20:30.275 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 490129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/bonding.ko 2023-10-30T13:20:30.275 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/ 2023-10-30T13:20:30.275 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/ 2023-10-30T13:20:30.275 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 106033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-30T13:20:30.275 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-30T13:20:30.275 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-30T13:20:30.276 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-30T13:20:30.276 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 453737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-30T13:20:30.276 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dummy.ko 2023-10-30T13:20:30.276 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/eql.ko 2023-10-30T13:20:30.276 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ 2023-10-30T13:20:30.277 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/ 2023-10-30T13:20:30.277 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 45785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-30T13:20:30.277 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-30T13:20:30.277 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 130553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-30T13:20:30.277 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 68849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-30T13:20:30.278 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ 2023-10-30T13:20:30.278 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-30T13:20:30.278 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 59689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-30T13:20:30.278 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-30T13:20:30.278 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 99113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-30T13:20:30.278 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/ 2023-10-30T13:20:30.279 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-30T13:20:30.279 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/ 2023-10-30T13:20:30.279 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 69201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-30T13:20:30.279 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/ 2023-10-30T13:20:30.279 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 59585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-30T13:20:30.280 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/ 2023-10-30T13:20:30.280 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-30T13:20:30.280 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/ 2023-10-30T13:20:30.280 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-30T13:20:30.280 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-30T13:20:30.280 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 87297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-30T13:20:30.281 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/ 2023-10-30T13:20:30.281 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-30T13:20:30.281 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 97337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-30T13:20:30.281 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-30T13:20:30.281 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 104673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-30T13:20:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-30T13:20:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 90257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-30T13:20:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-30T13:20:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 81569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-30T13:20:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 69729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-30T13:20:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/ 2023-10-30T13:20:30.283 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-30T13:20:30.283 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 176761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-30T13:20:30.283 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-30T13:20:30.283 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1421497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-30T13:20:30.283 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 128569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-30T13:20:30.284 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-30T13:20:30.284 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 132089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-30T13:20:30.284 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 312081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-30T13:20:30.284 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/ 2023-10-30T13:20:30.284 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-30T13:20:30.284 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 323513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-30T13:20:30.285 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/ 2023-10-30T13:20:30.285 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-30T13:20:30.285 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 193497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-30T13:20:30.285 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-30T13:20:30.285 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 347417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-30T13:20:30.286 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-30T13:20:30.286 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1044433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-30T13:20:30.286 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-30T13:20:30.286 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 166193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-30T13:20:30.286 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-30T13:20:30.287 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-30T13:20:30.287 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/ 2023-10-30T13:20:30.287 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-30T13:20:30.287 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 204345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-30T13:20:30.287 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/ 2023-10-30T13:20:30.288 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-30T13:20:30.288 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 69449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-30T13:20:30.288 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-30T13:20:30.288 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 141225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-30T13:20:30.288 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-30T13:20:30.288 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-30T13:20:30.289 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-30T13:20:30.289 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/ 2023-10-30T13:20:30.289 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-30T13:20:30.289 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 62833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-30T13:20:30.289 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 45721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dnet.ko 2023-10-30T13:20:30.290 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-30T13:20:30.290 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/ 2023-10-30T13:20:30.290 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-30T13:20:30.290 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 281137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-30T13:20:30.290 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ethoc.ko 2023-10-30T13:20:30.290 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fealnx.ko 2023-10-30T13:20:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/ 2023-10-30T13:20:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 45345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-30T13:20:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ 2023-10-30T13:20:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 90465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-30T13:20:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-30T13:20:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 295577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-30T13:20:30.292 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-30T13:20:30.292 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 584497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-30T13:20:30.292 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-30T13:20:30.292 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 246209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-30T13:20:30.292 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-30T13:20:30.293 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 925697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-30T13:20:30.293 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-30T13:20:30.293 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 302849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-30T13:20:30.293 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/ 2023-10-30T13:20:30.293 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 514681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-30T13:20:30.293 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-30T13:20:30.294 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 93593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-30T13:20:30.294 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-30T13:20:30.294 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 734305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-30T13:20:30.294 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-30T13:20:30.294 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 151769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-30T13:20:30.295 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 86921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/jme.ko 2023-10-30T13:20:30.295 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/ 2023-10-30T13:20:30.295 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-30T13:20:30.295 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 102289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-30T13:20:30.295 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 122649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-30T13:20:30.296 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/ 2023-10-30T13:20:30.296 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-30T13:20:30.296 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 727601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-30T13:20:30.296 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 278937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-30T13:20:30.296 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-30T13:20:30.297 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-30T13:20:30.297 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1113993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-30T13:20:30.297 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ 2023-10-30T13:20:30.297 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-30T13:20:30.297 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-30T13:20:30.297 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-30T13:20:30.298 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 95665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-30T13:20:30.298 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/ 2023-10-30T13:20:30.298 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-30T13:20:30.298 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 108649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-30T13:20:30.298 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ 2023-10-30T13:20:30.299 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 73441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-30T13:20:30.299 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-30T13:20:30.299 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/ 2023-10-30T13:20:30.299 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 150905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-30T13:20:30.299 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/ 2023-10-30T13:20:30.299 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 136721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-30T13:20:30.300 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/ 2023-10-30T13:20:30.300 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-30T13:20:30.300 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-30T13:20:30.300 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/ 2023-10-30T13:20:30.300 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-30T13:20:30.300 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 227897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-30T13:20:30.301 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 87081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-30T13:20:30.301 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-30T13:20:30.301 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 672049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-30T13:20:30.301 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/ 2023-10-30T13:20:30.301 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-30T13:20:30.301 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/ 2023-10-30T13:20:30.302 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-30T13:20:30.302 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 83985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-30T13:20:30.302 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-30T13:20:30.302 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 189801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-30T13:20:30.302 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/ 2023-10-30T13:20:30.303 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-30T13:20:30.303 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 104241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-30T13:20:30.303 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/ 2023-10-30T13:20:30.303 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1053921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-30T13:20:30.303 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/ 2023-10-30T13:20:30.303 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-30T13:20:30.304 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/ 2023-10-30T13:20:30.304 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-30T13:20:30.304 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 63337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-30T13:20:30.304 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/ 2023-10-30T13:20:30.304 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 61601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-30T13:20:30.304 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-30T13:20:30.305 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 73409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-30T13:20:30.305 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 68177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-30T13:20:30.305 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/ 2023-10-30T13:20:30.305 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-30T13:20:30.305 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-30T13:20:30.306 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-30T13:20:30.306 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-30T13:20:30.306 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 486961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-30T13:20:30.306 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/ 2023-10-30T13:20:30.306 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 103337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-30T13:20:30.307 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 166161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-30T13:20:30.307 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-30T13:20:30.307 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-30T13:20:30.307 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/ 2023-10-30T13:20:30.307 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-30T13:20:30.307 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/ 2023-10-30T13:20:30.308 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 86529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-30T13:20:30.308 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/ 2023-10-30T13:20:30.308 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 80273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-30T13:20:30.308 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 83633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-30T13:20:30.308 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/ 2023-10-30T13:20:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-30T13:20:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-30T13:20:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/ 2023-10-30T13:20:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-30T13:20:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/ 2023-10-30T13:20:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/defxx.ko 2023-10-30T13:20:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/ 2023-10-30T13:20:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 216985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-30T13:20:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 77345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/geneve.ko 2023-10-30T13:20:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/ 2023-10-30T13:20:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 235521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-30T13:20:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/ 2023-10-30T13:20:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-30T13:20:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ifb.ko 2023-10-30T13:20:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ 2023-10-30T13:20:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 83769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-30T13:20:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 62209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvlan.ko 2023-10-30T13:20:30.312 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvtap.ko 2023-10-30T13:20:30.312 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/ 2023-10-30T13:20:30.312 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-30T13:20:30.312 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-30T13:20:30.312 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-30T13:20:30.312 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-30T13:20:30.313 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio.ko 2023-10-30T13:20:30.313 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mii.ko 2023-10-30T13:20:30.313 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/netconsole.ko 2023-10-30T13:20:30.313 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/nlmon.ko 2023-10-30T13:20:30.313 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/ 2023-10-30T13:20:30.313 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-30T13:20:30.314 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-30T13:20:30.314 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ 2023-10-30T13:20:30.314 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/amd.ko 2023-10-30T13:20:30.314 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-30T13:20:30.314 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-30T13:20:30.315 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm87xx.ko 2023-10-30T13:20:30.315 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/broadcom.ko 2023-10-30T13:20:30.315 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/cicada.ko 2023-10-30T13:20:30.315 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/davicom.ko 2023-10-30T13:20:30.315 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/et1011c.ko 2023-10-30T13:20:30.315 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/icplus.ko 2023-10-30T13:20:30.316 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/lxt.ko 2023-10-30T13:20:30.316 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 82049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/marvell.ko 2023-10-30T13:20:30.316 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 76177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/micrel.ko 2023-10-30T13:20:30.316 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/national.ko 2023-10-30T13:20:30.316 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 97857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/phylink.ko 2023-10-30T13:20:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/qsemi.ko 2023-10-30T13:20:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/realtek.ko 2023-10-30T13:20:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/smsc.ko 2023-10-30T13:20:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ste10Xp.ko 2023-10-30T13:20:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/vitesse.ko 2023-10-30T13:20:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/ 2023-10-30T13:20:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/plip.ko 2023-10-30T13:20:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ 2023-10-30T13:20:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-30T13:20:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_async.ko 2023-10-30T13:20:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-30T13:20:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-30T13:20:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-30T13:20:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppoe.ko 2023-10-30T13:20:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppox.ko 2023-10-30T13:20:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pptp.ko 2023-10-30T13:20:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sb1000.ko 2023-10-30T13:20:30.320 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/ 2023-10-30T13:20:30.320 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/slip.ko 2023-10-30T13:20:30.320 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sungem_phy.ko 2023-10-30T13:20:30.320 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 64385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/tap.ko 2023-10-30T13:20:30.320 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/ 2023-10-30T13:20:30.320 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 106169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team.ko 2023-10-30T13:20:30.321 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-30T13:20:30.321 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-30T13:20:30.321 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-30T13:20:30.321 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_random.ko 2023-10-30T13:20:30.321 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-30T13:20:30.321 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 87017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/veth.ko 2023-10-30T13:20:30.322 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/ 2023-10-30T13:20:30.322 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 144297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-30T13:20:30.322 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/ 2023-10-30T13:20:30.322 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 297329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/vxlan.ko 2023-10-30T13:20:30.322 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/ 2023-10-30T13:20:30.323 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 144649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-30T13:20:30.323 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ 2023-10-30T13:20:30.323 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ntb.ko 2023-10-30T13:20:30.323 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/ 2023-10-30T13:20:30.323 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/ 2023-10-30T13:20:30.323 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 264937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme-core.ko 2023-10-30T13:20:30.324 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 102209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme.ko 2023-10-30T13:20:30.324 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/ 2023-10-30T13:20:30.324 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 133841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport.ko 2023-10-30T13:20:30.324 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_cs.ko 2023-10-30T13:20:30.324 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 84465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_pc.ko 2023-10-30T13:20:30.325 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_serial.ko 2023-10-30T13:20:30.325 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/ 2023-10-30T13:20:30.325 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/ 2023-10-30T13:20:30.325 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-30T13:20:30.325 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-30T13:20:30.325 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-30T13:20:30.326 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/pci-stub.ko 2023-10-30T13:20:30.326 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/xen-pcifront.ko 2023-10-30T13:20:30.326 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/ 2023-10-30T13:20:30.326 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/i82092.ko 2023-10-30T13:20:30.326 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 145809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia.ko 2023-10-30T13:20:30.327 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 62137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-30T13:20:30.327 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-30T13:20:30.327 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pd6729.ko 2023-10-30T13:20:30.327 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 112889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-30T13:20:30.327 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/ 2023-10-30T13:20:30.327 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/ 2023-10-30T13:20:30.328 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-30T13:20:30.328 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/ 2023-10-30T13:20:30.328 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-30T13:20:30.328 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/ 2023-10-30T13:20:30.328 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/ 2023-10-30T13:20:30.328 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-30T13:20:30.329 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/ 2023-10-30T13:20:30.329 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/ 2023-10-30T13:20:30.329 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-30T13:20:30.329 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-30T13:20:30.329 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-30T13:20:30.329 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-30T13:20:30.330 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-30T13:20:30.330 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-30T13:20:30.330 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-30T13:20:30.330 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ 2023-10-30T13:20:30.330 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-30T13:20:30.331 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acerhdf.ko 2023-10-30T13:20:30.331 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-30T13:20:30.331 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 59665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-30T13:20:30.331 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-30T13:20:30.331 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 127569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-30T13:20:30.331 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-30T13:20:30.332 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-30T13:20:30.332 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-30T13:20:30.332 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-30T13:20:30.332 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-30T13:20:30.332 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/hdaps.ko 2023-10-30T13:20:30.333 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-30T13:20:30.333 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/ 2023-10-30T13:20:30.333 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-30T13:20:30.333 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-30T13:20:30.333 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel_ips.ko 2023-10-30T13:20:30.333 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-30T13:20:30.334 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-30T13:20:30.334 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-30T13:20:30.334 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-30T13:20:30.334 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-30T13:20:30.334 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-30T13:20:30.334 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-30T13:20:30.335 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-30T13:20:30.335 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-30T13:20:30.335 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi.ko 2023-10-30T13:20:30.335 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/ 2023-10-30T13:20:30.335 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ 2023-10-30T13:20:30.336 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-30T13:20:30.336 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-30T13:20:30.336 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 55057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-30T13:20:30.336 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-30T13:20:30.336 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-30T13:20:30.337 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-30T13:20:30.337 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9030_battery.ko 2023-10-30T13:20:30.337 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9052-battery.ko 2023-10-30T13:20:30.337 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-30T13:20:30.337 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-30T13:20:30.337 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-30T13:20:30.338 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-30T13:20:30.338 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/gpio-charger.ko 2023-10-30T13:20:30.338 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-30T13:20:30.338 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-30T13:20:30.338 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max14577_charger.ko 2023-10-30T13:20:30.339 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17040_battery.ko 2023-10-30T13:20:30.339 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17042_battery.ko 2023-10-30T13:20:30.339 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8903_charger.ko 2023-10-30T13:20:30.339 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8925_power.ko 2023-10-30T13:20:30.339 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-30T13:20:30.339 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/sbs-battery.ko 2023-10-30T13:20:30.340 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/test_power.ko 2023-10-30T13:20:30.340 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-30T13:20:30.340 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-30T13:20:30.340 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_power.ko 2023-10-30T13:20:30.340 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm8350_power.ko 2023-10-30T13:20:30.340 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/ 2023-10-30T13:20:30.341 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-30T13:20:30.341 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-30T13:20:30.341 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/ 2023-10-30T13:20:30.341 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/ 2023-10-30T13:20:30.341 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-30T13:20:30.341 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-30T13:20:30.342 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps_parport.ko 2023-10-30T13:20:30.342 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/pps_core.ko 2023-10-30T13:20:30.342 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ 2023-10-30T13:20:30.342 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp.ko 2023-10-30T13:20:30.342 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp_kvm.ko 2023-10-30T13:20:30.342 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/ 2023-10-30T13:20:30.343 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-30T13:20:30.343 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-30T13:20:30.343 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-30T13:20:30.343 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9052.ko 2023-10-30T13:20:30.343 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9055.ko 2023-10-30T13:20:30.343 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9063.ko 2023-10-30T13:20:30.344 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-30T13:20:30.344 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 68577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-30T13:20:30.344 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-30T13:20:30.344 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-30T13:20:30.344 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-30T13:20:30.344 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-30T13:20:30.345 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-30T13:20:30.345 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-30T13:20:30.345 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-30T13:20:30.345 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-em3027.ko 2023-10-30T13:20:30.345 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-30T13:20:30.346 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-30T13:20:30.346 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-30T13:20:30.346 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-30T13:20:30.346 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-30T13:20:30.346 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-30T13:20:30.346 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-30T13:20:30.347 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-30T13:20:30.347 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max6900.ko 2023-10-30T13:20:30.347 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8907.ko 2023-10-30T13:20:30.347 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8925.ko 2023-10-30T13:20:30.347 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8997.ko 2023-10-30T13:20:30.348 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8998.ko 2023-10-30T13:20:30.348 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-30T13:20:30.348 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-30T13:20:30.348 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-palmas.ko 2023-10-30T13:20:30.348 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-30T13:20:30.348 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-30T13:20:30.349 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-30T13:20:30.349 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-30T13:20:30.349 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-30T13:20:30.349 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-30T13:20:30.349 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-30T13:20:30.349 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-30T13:20:30.350 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-30T13:20:30.350 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-30T13:20:30.350 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-30T13:20:30.350 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-30T13:20:30.350 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-30T13:20:30.351 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-30T13:20:30.351 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-30T13:20:30.351 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-30T13:20:30.351 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-30T13:20:30.351 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-30T13:20:30.351 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-x1205.ko 2023-10-30T13:20:30.352 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ 2023-10-30T13:20:30.352 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 77617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-9xxx.ko 2023-10-30T13:20:30.352 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-sas.ko 2023-10-30T13:20:30.352 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 55169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-xxxx.ko 2023-10-30T13:20:30.352 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 135185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/BusLogic.ko 2023-10-30T13:20:30.352 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/a100u2w.ko 2023-10-30T13:20:30.353 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/ 2023-10-30T13:20:30.353 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 272305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-30T13:20:30.353 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 100617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/advansys.ko 2023-10-30T13:20:30.353 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/ 2023-10-30T13:20:30.353 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 356265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-30T13:20:30.353 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 295801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-30T13:20:30.354 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/ 2023-10-30T13:20:30.354 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 189185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-30T13:20:30.354 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/am53c974.ko 2023-10-30T13:20:30.354 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/ 2023-10-30T13:20:30.354 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 106881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-30T13:20:30.355 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/atp870u.ko 2023-10-30T13:20:30.355 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/ 2023-10-30T13:20:30.355 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 246985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-30T13:20:30.355 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/ 2023-10-30T13:20:30.355 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1171945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/bfa.ko 2023-10-30T13:20:30.355 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/ 2023-10-30T13:20:30.356 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 237273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-30T13:20:30.356 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/ 2023-10-30T13:20:30.356 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 115369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-30T13:20:30.356 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ch.ko 2023-10-30T13:20:30.356 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/ 2023-10-30T13:20:30.356 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 308921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-30T13:20:30.357 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/ 2023-10-30T13:20:30.357 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-30T13:20:30.357 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 76145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-30T13:20:30.357 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-30T13:20:30.357 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 101449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-30T13:20:30.357 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 146689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-30T13:20:30.358 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 75553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dc395x.ko 2023-10-30T13:20:30.358 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dmx3191d.ko 2023-10-30T13:20:30.358 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/ 2023-10-30T13:20:30.358 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 212425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-30T13:20:30.358 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esp_scsi.ko 2023-10-30T13:20:30.359 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/ 2023-10-30T13:20:30.359 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-30T13:20:30.359 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 158505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-30T13:20:30.359 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fdomain.ko 2023-10-30T13:20:30.359 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/ 2023-10-30T13:20:30.359 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 263489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/fnic.ko 2023-10-30T13:20:30.360 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 201817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hpsa.ko 2023-10-30T13:20:30.360 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hptiop.ko 2023-10-30T13:20:30.360 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hv_storvsc.ko 2023-10-30T13:20:30.360 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/imm.ko 2023-10-30T13:20:30.360 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/initio.ko 2023-10-30T13:20:30.361 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 266537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ipr.ko 2023-10-30T13:20:30.361 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 101881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ips.ko 2023-10-30T13:20:30.361 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/ 2023-10-30T13:20:30.361 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 317153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/isci.ko 2023-10-30T13:20:30.361 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-30T13:20:30.361 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 52961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-30T13:20:30.362 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/ 2023-10-30T13:20:30.362 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 314865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/libfc.ko 2023-10-30T13:20:30.362 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 145585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi.ko 2023-10-30T13:20:30.362 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-30T13:20:30.362 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/ 2023-10-30T13:20:30.362 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 216665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/libsas.ko 2023-10-30T13:20:30.363 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/ 2023-10-30T13:20:30.363 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 2197249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-30T13:20:30.363 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/ 2023-10-30T13:20:30.363 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 73569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-30T13:20:30.363 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-30T13:20:30.363 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 310617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-30T13:20:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 84721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid.ko 2023-10-30T13:20:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/ 2023-10-30T13:20:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 659777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-30T13:20:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/ 2023-10-30T13:20:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 119561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-30T13:20:30.365 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvumi.ko 2023-10-30T13:20:30.365 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/ 2023-10-30T13:20:30.365 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 81625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-30T13:20:30.365 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-30T13:20:30.365 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-30T13:20:30.365 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-30T13:20:30.366 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/ 2023-10-30T13:20:30.366 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 364105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-30T13:20:30.366 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 123617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pmcraid.ko 2023-10-30T13:20:30.366 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ppa.ko 2023-10-30T13:20:30.366 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 55361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla1280.ko 2023-10-30T13:20:30.367 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/ 2023-10-30T13:20:30.367 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1876625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-30T13:20:30.367 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 85289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-30T13:20:30.367 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/ 2023-10-30T13:20:30.367 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 559033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-30T13:20:30.367 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qlogicfas408.ko 2023-10-30T13:20:30.368 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/raid_class.ko 2023-10-30T13:20:30.368 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 238825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_debug.ko 2023-10-30T13:20:30.368 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 173937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-30T13:20:30.368 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 279313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-30T13:20:30.368 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 97729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-30T13:20:30.368 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 73881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-30T13:20:30.369 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-30T13:20:30.369 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ses.ko 2023-10-30T13:20:30.369 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 115889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/st.ko 2023-10-30T13:20:30.369 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/stex.ko 2023-10-30T13:20:30.369 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-30T13:20:30.369 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 162121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-30T13:20:30.370 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/virtio_scsi.ko 2023-10-30T13:20:30.370 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-30T13:20:30.370 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/wd719x.ko 2023-10-30T13:20:30.370 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/xen-scsifront.ko 2023-10-30T13:20:30.370 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ 2023-10-30T13:20:30.371 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 129441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ssb.ko 2023-10-30T13:20:30.371 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/ 2023-10-30T13:20:30.371 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/ 2023-10-30T13:20:30.371 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 736601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-30T13:20:30.371 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/ 2023-10-30T13:20:30.371 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-30T13:20:30.372 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_file.ko 2023-10-30T13:20:30.372 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_iblock.ko 2023-10-30T13:20:30.372 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 930321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_mod.ko 2023-10-30T13:20:30.372 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_pscsi.ko 2023-10-30T13:20:30.372 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 122521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_user.ko 2023-10-30T13:20:30.372 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/ 2023-10-30T13:20:30.373 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 75593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-30T13:20:30.373 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/ 2023-10-30T13:20:30.373 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/ 2023-10-30T13:20:30.373 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-30T13:20:30.373 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-30T13:20:30.374 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-30T13:20:30.374 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-30T13:20:30.374 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-30T13:20:30.374 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-30T13:20:30.374 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-30T13:20:30.375 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-30T13:20:30.375 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-30T13:20:30.375 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-30T13:20:30.375 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-30T13:20:30.375 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-30T13:20:30.376 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-30T13:20:30.376 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-30T13:20:30.376 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-30T13:20:30.376 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-30T13:20:30.376 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-30T13:20:30.376 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ 2023-10-30T13:20:30.377 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ 2023-10-30T13:20:30.377 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-30T13:20:30.377 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/moxa.ko 2023-10-30T13:20:30.377 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 50953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/mxser.ko 2023-10-30T13:20:30.377 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 108601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_gsm.ko 2023-10-30T13:20:30.378 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_hdlc.ko 2023-10-30T13:20:30.378 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/nozomi.ko 2023-10-30T13:20:30.378 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/ 2023-10-30T13:20:30.378 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/ 2023-10-30T13:20:30.378 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-30T13:20:30.379 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 59497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-30T13:20:30.379 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-30T13:20:30.379 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_uart.ko 2023-10-30T13:20:30.379 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/arc_uart.ko 2023-10-30T13:20:30.379 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 72425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-30T13:20:30.379 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/ 2023-10-30T13:20:30.380 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 103073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-30T13:20:30.380 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/rp2.ko 2023-10-30T13:20:30.380 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 52817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-30T13:20:30.380 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/ 2023-10-30T13:20:30.380 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ 2023-10-30T13:20:30.380 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 531265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-30T13:20:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ 2023-10-30T13:20:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-30T13:20:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-30T13:20:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/ 2023-10-30T13:20:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio.ko 2023-10-30T13:20:30.382 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_aec.ko 2023-10-30T13:20:30.382 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_cif.ko 2023-10-30T13:20:30.382 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-30T13:20:30.382 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_mf624.ko 2023-10-30T13:20:30.382 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_netx.ko 2023-10-30T13:20:30.382 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pci_generic.ko 2023-10-30T13:20:30.383 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-30T13:20:30.383 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_sercos3.ko 2023-10-30T13:20:30.383 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/ 2023-10-30T13:20:30.383 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/ 2023-10-30T13:20:30.383 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 73297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-30T13:20:30.384 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ 2023-10-30T13:20:30.384 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 144377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-30T13:20:30.384 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-30T13:20:30.384 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-30T13:20:30.384 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-30T13:20:30.384 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/ 2023-10-30T13:20:30.385 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 87089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-acm.ko 2023-10-30T13:20:30.385 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-30T13:20:30.385 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usblp.ko 2023-10-30T13:20:30.385 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usbtmc.ko 2023-10-30T13:20:30.385 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ 2023-10-30T13:20:30.385 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ulpi.ko 2023-10-30T13:20:30.386 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/ 2023-10-30T13:20:30.386 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-30T13:20:30.386 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/ 2023-10-30T13:20:30.386 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-30T13:20:30.386 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-30T13:20:30.387 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 444553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-30T13:20:30.387 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/ 2023-10-30T13:20:30.387 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/ 2023-10-30T13:20:30.387 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-30T13:20:30.387 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-30T13:20:30.388 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-30T13:20:30.388 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-30T13:20:30.388 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-30T13:20:30.388 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-30T13:20:30.388 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 92489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-30T13:20:30.388 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-30T13:20:30.389 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 122081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-30T13:20:30.389 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-30T13:20:30.389 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-30T13:20:30.389 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-30T13:20:30.389 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-30T13:20:30.389 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 85609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-30T13:20:30.390 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-30T13:20:30.390 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 58113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-30T13:20:30.390 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-30T13:20:30.390 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/ 2023-10-30T13:20:30.390 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-30T13:20:30.391 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-30T13:20:30.391 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-30T13:20:30.391 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-30T13:20:30.391 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-30T13:20:30.391 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-30T13:20:30.391 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-30T13:20:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-30T13:20:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-30T13:20:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-30T13:20:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-30T13:20:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-30T13:20:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 62185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-30T13:20:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-30T13:20:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 161225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-30T13:20:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/ 2023-10-30T13:20:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-30T13:20:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-30T13:20:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 157681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-30T13:20:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-30T13:20:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 52721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-30T13:20:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 69705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-30T13:20:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 56033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-30T13:20:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 76057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-30T13:20:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 75073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-30T13:20:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 73169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-30T13:20:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 65809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-30T13:20:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-30T13:20:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 81505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-30T13:20:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 156073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-30T13:20:30.396 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ 2023-10-30T13:20:30.396 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-30T13:20:30.396 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 80113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-30T13:20:30.396 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 87353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-30T13:20:30.396 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-30T13:20:30.396 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-30T13:20:30.397 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811_cs.ko 2023-10-30T13:20:30.397 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-30T13:20:30.397 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-30T13:20:30.397 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/ 2023-10-30T13:20:30.397 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/mdc800.ko 2023-10-30T13:20:30.398 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/microtek.ko 2023-10-30T13:20:30.398 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ 2023-10-30T13:20:30.398 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/adutux.ko 2023-10-30T13:20:30.398 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/appledisplay.ko 2023-10-30T13:20:30.398 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-30T13:20:30.399 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cytherm.ko 2023-10-30T13:20:30.399 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ehset.ko 2023-10-30T13:20:30.399 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi26.ko 2023-10-30T13:20:30.399 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi62.ko 2023-10-30T13:20:30.399 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ezusb.ko 2023-10-30T13:20:30.399 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/idmouse.ko 2023-10-30T13:20:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/iowarrior.ko 2023-10-30T13:20:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-30T13:20:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ldusb.ko 2023-10-30T13:20:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/legousbtower.ko 2023-10-30T13:20:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/lvstest.ko 2023-10-30T13:20:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/ 2023-10-30T13:20:30.401 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 64841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-30T13:20:30.401 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-30T13:20:30.401 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usb3503.ko 2023-10-30T13:20:30.401 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usblcd.ko 2023-10-30T13:20:30.401 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-30T13:20:30.401 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbtest.ko 2023-10-30T13:20:30.402 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/uss720.ko 2023-10-30T13:20:30.402 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/yurex.ko 2023-10-30T13:20:30.402 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/ 2023-10-30T13:20:30.402 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 72953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/usbmon.ko 2023-10-30T13:20:30.402 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/ 2023-10-30T13:20:30.403 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 301545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-30T13:20:30.403 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/ 2023-10-30T13:20:30.403 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-generic.ko 2023-10-30T13:20:30.403 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-30T13:20:30.403 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-30T13:20:30.403 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-30T13:20:30.404 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ 2023-10-30T13:20:30.404 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/aircable.ko 2023-10-30T13:20:30.404 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ark3116.ko 2023-10-30T13:20:30.404 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-30T13:20:30.404 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ch341.ko 2023-10-30T13:20:30.404 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cp210x.ko 2023-10-30T13:20:30.405 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cyberjack.ko 2023-10-30T13:20:30.405 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 58217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-30T13:20:30.405 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-30T13:20:30.405 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/empeg.ko 2023-10-30T13:20:30.405 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/f81232.ko 2023-10-30T13:20:30.406 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 175137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-30T13:20:30.406 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-30T13:20:30.406 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 137081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-30T13:20:30.406 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 124833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_ti.ko 2023-10-30T13:20:30.406 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipaq.ko 2023-10-30T13:20:30.407 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipw.ko 2023-10-30T13:20:30.407 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ir-usb.ko 2023-10-30T13:20:30.407 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 58233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-30T13:20:30.407 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 107521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan.ko 2023-10-30T13:20:30.407 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-30T13:20:30.407 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-30T13:20:30.408 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-30T13:20:30.408 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mct_u232.ko 2023-10-30T13:20:30.408 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/metro-usb.ko 2023-10-30T13:20:30.408 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7720.ko 2023-10-30T13:20:30.408 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 78217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7840.ko 2023-10-30T13:20:30.409 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 56337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mxuport.ko 2023-10-30T13:20:30.409 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/navman.ko 2023-10-30T13:20:30.409 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/omninet.ko 2023-10-30T13:20:30.409 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/opticon.ko 2023-10-30T13:20:30.409 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 190633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/option.ko 2023-10-30T13:20:30.410 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/oti6858.ko 2023-10-30T13:20:30.410 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/pl2303.ko 2023-10-30T13:20:30.410 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcaux.ko 2023-10-30T13:20:30.410 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcserial.ko 2023-10-30T13:20:30.410 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/quatech2.ko 2023-10-30T13:20:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/safe_serial.ko 2023-10-30T13:20:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 51625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/sierra.ko 2023-10-30T13:20:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-30T13:20:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ssu100.ko 2023-10-30T13:20:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/symbolserial.ko 2023-10-30T13:20:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-30T13:20:30.412 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-30T13:20:30.412 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_debug.ko 2023-10-30T13:20:30.412 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-30T13:20:30.412 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 125609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usbserial.ko 2023-10-30T13:20:30.412 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/visor.ko 2023-10-30T13:20:30.413 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/whiteheat.ko 2023-10-30T13:20:30.413 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-30T13:20:30.413 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-30T13:20:30.413 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ 2023-10-30T13:20:30.413 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 52913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/uas.ko 2023-10-30T13:20:30.413 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-30T13:20:30.414 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-30T13:20:30.414 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-30T13:20:30.414 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-30T13:20:30.414 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-30T13:20:30.414 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-30T13:20:30.415 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-30T13:20:30.415 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-karma.ko 2023-10-30T13:20:30.415 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-30T13:20:30.415 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-30T13:20:30.415 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-30T13:20:30.416 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-30T13:20:30.416 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-30T13:20:30.416 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 179969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/usb-storage.ko 2023-10-30T13:20:30.416 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/ 2023-10-30T13:20:30.416 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-30T13:20:30.417 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 72177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-30T13:20:30.417 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 107001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-30T13:20:30.417 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/ 2023-10-30T13:20:30.417 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/ 2023-10-30T13:20:30.417 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 154297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-30T13:20:30.417 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-30T13:20:30.418 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 108737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio.ko 2023-10-30T13:20:30.418 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 77609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-30T13:20:30.418 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/ 2023-10-30T13:20:30.418 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 113873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost.ko 2023-10-30T13:20:30.418 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-30T13:20:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 58089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_net.ko 2023-10-30T13:20:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_scsi.ko 2023-10-30T13:20:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/ 2023-10-30T13:20:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/ 2023-10-30T13:20:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-30T13:20:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-30T13:20:30.420 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-30T13:20:30.420 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-30T13:20:30.420 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-30T13:20:30.420 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/apple_bl.ko 2023-10-30T13:20:30.420 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-30T13:20:30.421 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/bd6107.ko 2023-10-30T13:20:30.421 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-30T13:20:30.421 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-30T13:20:30.421 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-30T13:20:30.422 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-30T13:20:30.422 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lcd.ko 2023-10-30T13:20:30.422 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-30T13:20:30.422 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-30T13:20:30.422 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-30T13:20:30.423 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-30T13:20:30.423 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-30T13:20:30.423 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-30T13:20:30.423 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-30T13:20:30.423 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-30T13:20:30.423 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/ 2023-10-30T13:20:30.424 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ 2023-10-30T13:20:30.424 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2482.ko 2023-10-30T13:20:30.424 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2490.ko 2023-10-30T13:20:30.424 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-30T13:20:30.424 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-30T13:20:30.425 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/ 2023-10-30T13:20:30.425 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-30T13:20:30.425 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-30T13:20:30.425 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-30T13:20:30.425 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-30T13:20:30.426 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-30T13:20:30.426 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-30T13:20:30.426 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-30T13:20:30.426 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-30T13:20:30.426 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-30T13:20:30.426 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-30T13:20:30.427 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 56425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-30T13:20:30.427 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 102673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/wire.ko 2023-10-30T13:20:30.427 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ 2023-10-30T13:20:30.427 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/acquirewdt.ko 2023-10-30T13:20:30.427 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/advantechwdt.ko 2023-10-30T13:20:30.428 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-30T13:20:30.428 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-30T13:20:30.428 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-30T13:20:30.428 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-30T13:20:30.428 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-30T13:20:30.428 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-30T13:20:30.429 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/dw_wdt.ko 2023-10-30T13:20:30.429 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-30T13:20:30.429 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-30T13:20:30.429 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/hpwdt.ko 2023-10-30T13:20:30.429 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/i6300esb.ko 2023-10-30T13:20:30.429 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-30T13:20:30.430 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-30T13:20:30.430 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ib700wdt.ko 2023-10-30T13:20:30.430 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ibmasr.ko 2023-10-30T13:20:30.430 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-30T13:20:30.430 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-30T13:20:30.430 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it87_wdt.ko 2023-10-30T13:20:30.431 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-30T13:20:30.431 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/machzwd.ko 2023-10-30T13:20:30.431 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-30T13:20:30.431 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-30T13:20:30.431 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/nv_tco.ko 2023-10-30T13:20:30.431 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-30T13:20:30.432 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-30T13:20:30.432 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-30T13:20:30.432 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-30T13:20:30.432 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/retu_wdt.ko 2023-10-30T13:20:30.432 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-30T13:20:30.433 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-30T13:20:30.433 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-30T13:20:30.433 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-30T13:20:30.433 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-30T13:20:30.433 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-30T13:20:30.433 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/softdog.ko 2023-10-30T13:20:30.434 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-30T13:20:30.434 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-30T13:20:30.434 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/via_wdt.ko 2023-10-30T13:20:30.434 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-30T13:20:30.434 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-30T13:20:30.434 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-30T13:20:30.435 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-30T13:20:30.435 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wdt_pci.ko 2023-10-30T13:20:30.435 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-30T13:20:30.435 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-30T13:20:30.435 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/xen_wdt.ko 2023-10-30T13:20:30.435 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/ 2023-10-30T13:20:30.436 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-evtchn.ko 2023-10-30T13:20:30.436 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntalloc.ko 2023-10-30T13:20:30.436 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntdev.ko 2023-10-30T13:20:30.436 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/ 2023-10-30T13:20:30.436 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 176441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-30T13:20:30.437 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-privcmd.ko 2023-10-30T13:20:30.437 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 71249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-scsiback.ko 2023-10-30T13:20:30.437 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/ 2023-10-30T13:20:30.437 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-30T13:20:30.437 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ 2023-10-30T13:20:30.437 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/ 2023-10-30T13:20:30.438 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 166425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/9p.ko 2023-10-30T13:20:30.438 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/ 2023-10-30T13:20:30.438 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 61073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/adfs.ko 2023-10-30T13:20:30.438 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/ 2023-10-30T13:20:30.438 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 163593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/affs.ko 2023-10-30T13:20:30.438 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/ 2023-10-30T13:20:30.439 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 932817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/kafs.ko 2023-10-30T13:20:30.439 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/ 2023-10-30T13:20:30.439 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 101057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/autofs4.ko 2023-10-30T13:20:30.439 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/ 2023-10-30T13:20:30.439 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 64545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/befs.ko 2023-10-30T13:20:30.439 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/ 2023-10-30T13:20:30.440 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/bfs.ko 2023-10-30T13:20:30.440 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/binfmt_misc.ko 2023-10-30T13:20:30.440 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/ 2023-10-30T13:20:30.440 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 3402585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/btrfs.ko 2023-10-30T13:20:30.440 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/ 2023-10-30T13:20:30.440 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 306073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/cachefiles.ko 2023-10-30T13:20:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ 2023-10-30T13:20:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1323945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ceph.ko 2023-10-30T13:20:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/ 2023-10-30T13:20:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 100801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/coda.ko 2023-10-30T13:20:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/ 2023-10-30T13:20:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 108393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/configfs.ko 2023-10-30T13:20:30.442 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/ 2023-10-30T13:20:30.442 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 29625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/cramfs.ko 2023-10-30T13:20:30.442 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/ 2023-10-30T13:20:30.442 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 642577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/dlm.ko 2023-10-30T13:20:30.442 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/ 2023-10-30T13:20:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/efs.ko 2023-10-30T13:20:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/ 2023-10-30T13:20:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1505025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/f2fs.ko 2023-10-30T13:20:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/ 2023-10-30T13:20:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/msdos.ko 2023-10-30T13:20:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/ 2023-10-30T13:20:30.444 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/freevxfs.ko 2023-10-30T13:20:30.444 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/ 2023-10-30T13:20:30.444 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 226481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/fscache.ko 2023-10-30T13:20:30.444 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/ 2023-10-30T13:20:30.444 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/cuse.ko 2023-10-30T13:20:30.444 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/ 2023-10-30T13:20:30.445 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1231529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/gfs2.ko 2023-10-30T13:20:30.445 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/ 2023-10-30T13:20:30.445 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 141209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/hfs.ko 2023-10-30T13:20:30.445 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/ 2023-10-30T13:20:30.445 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 222817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/hfsplus.ko 2023-10-30T13:20:30.445 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/ 2023-10-30T13:20:30.446 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 192329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/hpfs.ko 2023-10-30T13:20:30.446 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/ 2023-10-30T13:20:30.446 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 74065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/isofs.ko 2023-10-30T13:20:30.446 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/ 2023-10-30T13:20:30.446 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 419497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/jfs.ko 2023-10-30T13:20:30.447 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/ 2023-10-30T13:20:30.447 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 280313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/lockd.ko 2023-10-30T13:20:30.447 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/ 2023-10-30T13:20:30.447 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/minix.ko 2023-10-30T13:20:30.447 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/ 2023-10-30T13:20:30.447 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 124649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/netfs.ko 2023-10-30T13:20:30.448 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/ 2023-10-30T13:20:30.448 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/ 2023-10-30T13:20:30.448 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 81449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-30T13:20:30.448 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/ 2023-10-30T13:20:30.448 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 59449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-30T13:20:30.448 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/ 2023-10-30T13:20:30.449 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 117513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-30T13:20:30.449 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1055649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfs.ko 2023-10-30T13:20:30.449 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv2.ko 2023-10-30T13:20:30.449 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 120177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv3.ko 2023-10-30T13:20:30.449 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 2041689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv4.ko 2023-10-30T13:20:30.449 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/ 2023-10-30T13:20:30.450 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/grace.ko 2023-10-30T13:20:30.450 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/nfs_acl.ko 2023-10-30T13:20:30.450 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/ 2023-10-30T13:20:30.450 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1516809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/nfsd.ko 2023-10-30T13:20:30.450 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/ 2023-10-30T13:20:30.450 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 592833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/nilfs2.ko 2023-10-30T13:20:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/ 2023-10-30T13:20:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-celtic.ko 2023-10-30T13:20:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-centeuro.ko 2023-10-30T13:20:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-croatian.ko 2023-10-30T13:20:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-cyrillic.ko 2023-10-30T13:20:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-gaelic.ko 2023-10-30T13:20:30.452 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-greek.ko 2023-10-30T13:20:30.452 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-iceland.ko 2023-10-30T13:20:30.452 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-inuit.ko 2023-10-30T13:20:30.452 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-roman.ko 2023-10-30T13:20:30.452 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-romanian.ko 2023-10-30T13:20:30.452 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-turkish.ko 2023-10-30T13:20:30.453 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ascii.ko 2023-10-30T13:20:30.453 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1250.ko 2023-10-30T13:20:30.453 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1251.ko 2023-10-30T13:20:30.453 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1255.ko 2023-10-30T13:20:30.453 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp737.ko 2023-10-30T13:20:30.453 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp775.ko 2023-10-30T13:20:30.454 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp850.ko 2023-10-30T13:20:30.454 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp852.ko 2023-10-30T13:20:30.454 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp855.ko 2023-10-30T13:20:30.454 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp857.ko 2023-10-30T13:20:30.454 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp860.ko 2023-10-30T13:20:30.455 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp861.ko 2023-10-30T13:20:30.455 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp862.ko 2023-10-30T13:20:30.455 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp863.ko 2023-10-30T13:20:30.455 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp864.ko 2023-10-30T13:20:30.455 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp865.ko 2023-10-30T13:20:30.455 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp866.ko 2023-10-30T13:20:30.456 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp869.ko 2023-10-30T13:20:30.456 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp874.ko 2023-10-30T13:20:30.456 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 94137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp932.ko 2023-10-30T13:20:30.456 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 142673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp936.ko 2023-10-30T13:20:30.456 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 165745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp949.ko 2023-10-30T13:20:30.456 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 118121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp950.ko 2023-10-30T13:20:30.457 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_euc-jp.ko 2023-10-30T13:20:30.457 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-1.ko 2023-10-30T13:20:30.457 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-13.ko 2023-10-30T13:20:30.457 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-14.ko 2023-10-30T13:20:30.457 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-15.ko 2023-10-30T13:20:30.457 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-2.ko 2023-10-30T13:20:30.458 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-3.ko 2023-10-30T13:20:30.458 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-4.ko 2023-10-30T13:20:30.458 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-5.ko 2023-10-30T13:20:30.458 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-6.ko 2023-10-30T13:20:30.458 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-7.ko 2023-10-30T13:20:30.459 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-9.ko 2023-10-30T13:20:30.459 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-r.ko 2023-10-30T13:20:30.459 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-ru.ko 2023-10-30T13:20:30.459 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-u.ko 2023-10-30T13:20:30.459 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-30T13:20:30.459 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_utf8.ko 2023-10-30T13:20:30.460 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ 2023-10-30T13:20:30.460 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 219417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ntfs.ko 2023-10-30T13:20:30.460 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ 2023-10-30T13:20:30.460 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ 2023-10-30T13:20:30.460 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 244409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-30T13:20:30.460 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ 2023-10-30T13:20:30.461 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 440737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-30T13:20:30.461 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ 2023-10-30T13:20:30.461 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-30T13:20:30.461 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 2592433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2.ko 2023-10-30T13:20:30.461 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-30T13:20:30.461 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-30T13:20:30.462 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-30T13:20:30.462 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/ 2023-10-30T13:20:30.462 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/omfs.ko 2023-10-30T13:20:30.462 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/ 2023-10-30T13:20:30.462 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 352153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/overlay.ko 2023-10-30T13:20:30.462 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ 2023-10-30T13:20:30.463 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ramoops.ko 2023-10-30T13:20:30.463 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/ 2023-10-30T13:20:30.463 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/qnx4.ko 2023-10-30T13:20:30.463 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/ 2023-10-30T13:20:30.463 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 46689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/qnx6.ko 2023-10-30T13:20:30.464 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/ 2023-10-30T13:20:30.464 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_tree.ko 2023-10-30T13:20:30.464 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v1.ko 2023-10-30T13:20:30.464 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v2.ko 2023-10-30T13:20:30.464 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/ 2023-10-30T13:20:30.464 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 532673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/reiserfs.ko 2023-10-30T13:20:30.465 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/ 2023-10-30T13:20:30.465 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/romfs.ko 2023-10-30T13:20:30.465 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/ 2023-10-30T13:20:30.465 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/ 2023-10-30T13:20:30.465 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 2668121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/cifs.ko 2023-10-30T13:20:30.465 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/ 2023-10-30T13:20:30.466 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_arc4.ko 2023-10-30T13:20:30.466 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_md4.ko 2023-10-30T13:20:30.466 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/ 2023-10-30T13:20:30.466 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 134641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/squashfs.ko 2023-10-30T13:20:30.466 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/ 2023-10-30T13:20:30.466 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 90721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/sysv.ko 2023-10-30T13:20:30.467 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/ 2023-10-30T13:20:30.467 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 247737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/udf.ko 2023-10-30T13:20:30.467 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ 2023-10-30T13:20:30.467 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 162497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ufs.ko 2023-10-30T13:20:30.467 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/ 2023-10-30T13:20:30.468 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 4010729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/xfs.ko 2023-10-30T13:20:30.503 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/ 2023-10-30T13:20:30.504 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/ 2023-10-30T13:20:30.504 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 75489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/locktorture.ko 2023-10-30T13:20:30.504 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/ 2023-10-30T13:20:30.504 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/test_udelay.ko 2023-10-30T13:20:30.504 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/torture.ko 2023-10-30T13:20:30.504 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ 2023-10-30T13:20:30.505 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 6833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc-itu-t.ko 2023-10-30T13:20:30.505 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 6529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc7.ko 2023-10-30T13:20:30.505 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc8.ko 2023-10-30T13:20:30.505 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/ 2023-10-30T13:20:30.505 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libarc4.ko 2023-10-30T13:20:30.506 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libdes.ko 2023-10-30T13:20:30.506 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/interval_tree_test.ko 2023-10-30T13:20:30.506 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/libcrc32c.ko 2023-10-30T13:20:30.506 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lru_cache.ko 2023-10-30T13:20:30.506 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/ 2023-10-30T13:20:30.507 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4_compress.ko 2023-10-30T13:20:30.507 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4hc_compress.ko 2023-10-30T13:20:30.507 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/ 2023-10-30T13:20:30.507 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 6617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/cordic.ko 2023-10-30T13:20:30.507 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/memory-notifier-error-inject.ko 2023-10-30T13:20:30.508 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/notifier-error-inject.ko 2023-10-30T13:20:30.508 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/percpu_test.ko 2023-10-30T13:20:30.508 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/pm-notifier-error-inject.ko 2023-10-30T13:20:30.508 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/ 2023-10-30T13:20:30.508 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 143433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/raid6_pq.ko 2023-10-30T13:20:30.509 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/rbtree_test.ko 2023-10-30T13:20:30.509 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/ 2023-10-30T13:20:30.509 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-30T13:20:30.509 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-kstrtox.ko 2023-10-30T13:20:30.509 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-string_helpers.ko 2023-10-30T13:20:30.509 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 4760425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_bpf.ko 2023-10-30T13:20:30.510 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_firmware.ko 2023-10-30T13:20:30.510 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 6369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_module.ko 2023-10-30T13:20:30.510 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_user_copy.ko 2023-10-30T13:20:30.510 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_bm.ko 2023-10-30T13:20:30.510 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_fsm.ko 2023-10-30T13:20:30.510 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_kmp.ko 2023-10-30T13:20:30.511 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/ 2023-10-30T13:20:30.511 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/xz_dec_test.ko 2023-10-30T13:20:30.511 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/ 2023-10-30T13:20:30.511 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/hwpoison-inject.ko 2023-10-30T13:20:30.511 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ 2023-10-30T13:20:30.512 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/ 2023-10-30T13:20:30.512 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 55737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/6lowpan.ko 2023-10-30T13:20:30.512 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_dest.ko 2023-10-30T13:20:30.512 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_fragment.ko 2023-10-30T13:20:30.512 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_hop.ko 2023-10-30T13:20:30.512 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 6985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-30T13:20:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_mobility.ko 2023-10-30T13:20:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_routing.ko 2023-10-30T13:20:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_udp.ko 2023-10-30T13:20:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/ 2023-10-30T13:20:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/garp.ko 2023-10-30T13:20:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/mrp.ko 2023-10-30T13:20:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/p8022.ko 2023-10-30T13:20:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/psnap.ko 2023-10-30T13:20:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/stp.ko 2023-10-30T13:20:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/ 2023-10-30T13:20:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 81737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/8021q.ko 2023-10-30T13:20:30.515 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/ 2023-10-30T13:20:30.515 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 198337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet.ko 2023-10-30T13:20:30.515 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_fd.ko 2023-10-30T13:20:30.515 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_rdma.ko 2023-10-30T13:20:30.515 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_virtio.ko 2023-10-30T13:20:30.516 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/ 2023-10-30T13:20:30.516 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 649297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/batman-adv.ko 2023-10-30T13:20:30.516 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/ 2023-10-30T13:20:30.516 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 63809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/br_netfilter.ko 2023-10-30T13:20:30.516 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 777689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/bridge.ko 2023-10-30T13:20:30.516 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ 2023-10-30T13:20:30.517 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-30T13:20:30.517 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-30T13:20:30.517 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-30T13:20:30.517 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-30T13:20:30.517 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-30T13:20:30.518 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-30T13:20:30.518 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-30T13:20:30.518 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-30T13:20:30.518 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-30T13:20:30.518 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-30T13:20:30.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-30T13:20:30.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-30T13:20:30.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-30T13:20:30.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-30T13:20:30.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-30T13:20:30.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-30T13:20:30.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-30T13:20:30.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-30T13:20:30.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-30T13:20:30.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-30T13:20:30.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 55929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtables.ko 2023-10-30T13:20:30.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-30T13:20:30.521 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-30T13:20:30.521 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/ 2023-10-30T13:20:30.521 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1047497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/libceph.ko 2023-10-30T13:20:30.521 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/ 2023-10-30T13:20:30.521 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 125697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/pktgen.ko 2023-10-30T13:20:30.522 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/ 2023-10-30T13:20:30.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 194209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp.ko 2023-10-30T13:20:30.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_diag.ko 2023-10-30T13:20:30.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 57881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv4.ko 2023-10-30T13:20:30.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv6.ko 2023-10-30T13:20:30.522 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/ 2023-10-30T13:20:30.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 390473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/dsa_core.ko 2023-10-30T13:20:30.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_brcm.ko 2023-10-30T13:20:30.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_dsa.ko 2023-10-30T13:20:30.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_none.ko 2023-10-30T13:20:30.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_trailer.ko 2023-10-30T13:20:30.523 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/ 2023-10-30T13:20:30.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 99729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/hsr.ko 2023-10-30T13:20:30.524 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ 2023-10-30T13:20:30.524 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ 2023-10-30T13:20:30.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 45233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-30T13:20:30.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 254625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154.ko 2023-10-30T13:20:30.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-30T13:20:30.525 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ 2023-10-30T13:20:30.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ah4.ko 2023-10-30T13:20:30.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/esp4.ko 2023-10-30T13:20:30.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 48401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/fou.ko 2023-10-30T13:20:30.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/gre.ko 2023-10-30T13:20:30.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 48657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/inet_diag.ko 2023-10-30T13:20:30.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_gre.ko 2023-10-30T13:20:30.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 61225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_tunnel.ko 2023-10-30T13:20:30.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_vti.ko 2023-10-30T13:20:30.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipcomp.ko 2023-10-30T13:20:30.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipip.ko 2023-10-30T13:20:30.527 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ 2023-10-30T13:20:30.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-30T13:20:30.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-30T13:20:30.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-30T13:20:30.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-30T13:20:30.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-30T13:20:30.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-30T13:20:30.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-30T13:20:30.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-30T13:20:30.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-30T13:20:30.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-30T13:20:30.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-30T13:20:30.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-30T13:20:30.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-30T13:20:30.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-30T13:20:30.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-30T13:20:30.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-30T13:20:30.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-30T13:20:30.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-30T13:20:30.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-30T13:20:30.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-30T13:20:30.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-30T13:20:30.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-30T13:20:30.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-30T13:20:30.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-30T13:20:30.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-30T13:20:30.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_bic.ko 2023-10-30T13:20:30.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_dctcp.ko 2023-10-30T13:20:30.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_diag.ko 2023-10-30T13:20:30.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_highspeed.ko 2023-10-30T13:20:30.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_htcp.ko 2023-10-30T13:20:30.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_hybla.ko 2023-10-30T13:20:30.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_illinois.ko 2023-10-30T13:20:30.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_lp.ko 2023-10-30T13:20:30.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_scalable.ko 2023-10-30T13:20:30.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_vegas.ko 2023-10-30T13:20:30.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_veno.ko 2023-10-30T13:20:30.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_westwood.ko 2023-10-30T13:20:30.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_yeah.ko 2023-10-30T13:20:30.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tunnel4.ko 2023-10-30T13:20:30.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_diag.ko 2023-10-30T13:20:30.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 41665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_tunnel.ko 2023-10-30T13:20:30.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-30T13:20:30.535 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ 2023-10-30T13:20:30.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ah6.ko 2023-10-30T13:20:30.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/esp6.ko 2023-10-30T13:20:30.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/fou6.ko 2023-10-30T13:20:30.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 80897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_gre.ko 2023-10-30T13:20:30.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 93577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_tunnel.ko 2023-10-30T13:20:30.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-30T13:20:30.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_vti.ko 2023-10-30T13:20:30.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ipcomp6.ko 2023-10-30T13:20:30.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/mip6.ko 2023-10-30T13:20:30.536 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ 2023-10-30T13:20:30.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-30T13:20:30.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-30T13:20:30.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-30T13:20:30.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-30T13:20:30.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-30T13:20:30.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-30T13:20:30.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-30T13:20:30.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-30T13:20:30.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-30T13:20:30.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-30T13:20:30.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-30T13:20:30.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-30T13:20:30.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-30T13:20:30.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-30T13:20:30.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-30T13:20:30.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-30T13:20:30.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-30T13:20:30.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-30T13:20:30.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-30T13:20:30.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-30T13:20:30.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-30T13:20:30.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-30T13:20:30.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-30T13:20:30.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-30T13:20:30.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-30T13:20:30.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 74073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/sit.ko 2023-10-30T13:20:30.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/tunnel6.ko 2023-10-30T13:20:30.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-30T13:20:30.541 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/ 2023-10-30T13:20:30.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 82833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/af_key.ko 2023-10-30T13:20:30.542 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/ 2023-10-30T13:20:30.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 125281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_core.ko 2023-10-30T13:20:30.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-30T13:20:30.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 25953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_eth.ko 2023-10-30T13:20:30.543 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip.ko 2023-10-30T13:20:30.543 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip6.ko 2023-10-30T13:20:30.543 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_netlink.ko 2023-10-30T13:20:30.543 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 53241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ppp.ko 2023-10-30T13:20:30.543 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/ 2023-10-30T13:20:30.543 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 47977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/lapb.ko 2023-10-30T13:20:30.544 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/ 2023-10-30T13:20:30.544 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc.ko 2023-10-30T13:20:30.544 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 267777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc2.ko 2023-10-30T13:20:30.544 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/ 2023-10-30T13:20:30.544 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 290417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/mac802154.ko 2023-10-30T13:20:30.544 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/ 2023-10-30T13:20:30.545 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/mpls_gso.ko 2023-10-30T13:20:30.545 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ 2023-10-30T13:20:30.545 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ 2023-10-30T13:20:30.545 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 119585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set.ko 2023-10-30T13:20:30.545 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-30T13:20:30.545 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-30T13:20:30.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-30T13:20:30.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 100297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-30T13:20:30.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 101441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-30T13:20:30.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 102817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-30T13:20:30.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 103481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-30T13:20:30.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 112849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-30T13:20:30.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 60329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-30T13:20:30.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-30T13:20:30.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 113129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-30T13:20:30.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 112585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-30T13:20:30.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 111585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-30T13:20:30.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 114465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-30T13:20:30.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-30T13:20:30.548 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ 2023-10-30T13:20:30.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 411729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-30T13:20:30.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-30T13:20:30.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-30T13:20:30.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-30T13:20:30.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-30T13:20:30.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-30T13:20:30.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-30T13:20:30.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-30T13:20:30.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-30T13:20:30.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-30T13:20:30.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-30T13:20:30.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-30T13:20:30.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-30T13:20:30.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-30T13:20:30.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conncount.ko 2023-10-30T13:20:30.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 408257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack.ko 2023-10-30T13:20:30.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-30T13:20:30.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-30T13:20:30.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-30T13:20:30.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 118145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-30T13:20:30.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-30T13:20:30.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-30T13:20:30.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 107297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-30T13:20:30.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-30T13:20:30.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-30T13:20:30.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-30T13:20:30.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-30T13:20:30.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-30T13:20:30.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-30T13:20:30.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table.ko 2023-10-30T13:20:30.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-30T13:20:30.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 37129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_log_syslog.ko 2023-10-30T13:20:30.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 95417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat.ko 2023-10-30T13:20:30.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-30T13:20:30.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-30T13:20:30.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_irc.ko 2023-10-30T13:20:30.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_sip.ko 2023-10-30T13:20:30.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-30T13:20:30.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 44065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-30T13:20:30.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 615313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_tables.ko 2023-10-30T13:20:30.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 42561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink.ko 2023-10-30T13:20:30.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-30T13:20:30.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-30T13:20:30.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 39785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_log.ko 2023-10-30T13:20:30.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-30T13:20:30.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-30T13:20:30.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_chain_nat.ko 2023-10-30T13:20:30.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_compat.ko 2023-10-30T13:20:30.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 40721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_ct.ko 2023-10-30T13:20:30.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-30T13:20:30.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fib.ko 2023-10-30T13:20:30.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-30T13:20:30.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_hash.ko 2023-10-30T13:20:30.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_limit.ko 2023-10-30T13:20:30.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_log.ko 2023-10-30T13:20:30.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_masq.ko 2023-10-30T13:20:30.558 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_nat.ko 2023-10-30T13:20:30.558 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_numgen.ko 2023-10-30T13:20:30.558 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_osf.ko 2023-10-30T13:20:30.558 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_queue.ko 2023-10-30T13:20:30.558 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_redir.ko 2023-10-30T13:20:30.558 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject.ko 2023-10-30T13:20:30.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject_inet.ko 2023-10-30T13:20:30.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_socket.ko 2023-10-30T13:20:30.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_synproxy.ko 2023-10-30T13:20:30.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_tproxy.ko 2023-10-30T13:20:30.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 89313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/x_tables.ko 2023-10-30T13:20:30.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_AUDIT.ko 2023-10-30T13:20:30.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-30T13:20:30.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-30T13:20:30.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-30T13:20:30.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 18169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CT.ko 2023-10-30T13:20:30.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_DSCP.ko 2023-10-30T13:20:30.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HL.ko 2023-10-30T13:20:30.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HMARK.ko 2023-10-30T13:20:30.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-30T13:20:30.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LED.ko 2023-10-30T13:20:30.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LOG.ko 2023-10-30T13:20:30.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-30T13:20:30.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NETMAP.ko 2023-10-30T13:20:30.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFLOG.ko 2023-10-30T13:20:30.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-30T13:20:30.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_RATEEST.ko 2023-10-30T13:20:30.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-30T13:20:30.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_SECMARK.ko 2023-10-30T13:20:30.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-30T13:20:30.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-30T13:20:30.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TEE.ko 2023-10-30T13:20:30.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TPROXY.ko 2023-10-30T13:20:30.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TRACE.ko 2023-10-30T13:20:30.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_addrtype.ko 2023-10-30T13:20:30.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_bpf.ko 2023-10-30T13:20:30.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cgroup.ko 2023-10-30T13:20:30.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cluster.ko 2023-10-30T13:20:30.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_comment.ko 2023-10-30T13:20:30.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connbytes.ko 2023-10-30T13:20:30.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlabel.ko 2023-10-30T13:20:30.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlimit.ko 2023-10-30T13:20:30.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connmark.ko 2023-10-30T13:20:30.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_conntrack.ko 2023-10-30T13:20:30.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cpu.ko 2023-10-30T13:20:30.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dccp.ko 2023-10-30T13:20:30.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_devgroup.ko 2023-10-30T13:20:30.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dscp.ko 2023-10-30T13:20:30.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ecn.ko 2023-10-30T13:20:30.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_esp.ko 2023-10-30T13:20:30.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hashlimit.ko 2023-10-30T13:20:30.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_helper.ko 2023-10-30T13:20:30.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hl.ko 2023-10-30T13:20:30.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipcomp.ko 2023-10-30T13:20:30.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_iprange.ko 2023-10-30T13:20:30.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipvs.ko 2023-10-30T13:20:30.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_l2tp.ko 2023-10-30T13:20:30.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_length.ko 2023-10-30T13:20:30.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_limit.ko 2023-10-30T13:20:30.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mac.ko 2023-10-30T13:20:30.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mark.ko 2023-10-30T13:20:30.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_multiport.ko 2023-10-30T13:20:30.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nat.ko 2023-10-30T13:20:30.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nfacct.ko 2023-10-30T13:20:30.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_osf.ko 2023-10-30T13:20:30.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_owner.ko 2023-10-30T13:20:30.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_physdev.ko 2023-10-30T13:20:30.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_pkttype.ko 2023-10-30T13:20:30.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_policy.ko 2023-10-30T13:20:30.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_quota.ko 2023-10-30T13:20:30.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_rateest.ko 2023-10-30T13:20:30.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_realm.ko 2023-10-30T13:20:30.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_recent.ko 2023-10-30T13:20:30.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_sctp.ko 2023-10-30T13:20:30.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_set.ko 2023-10-30T13:20:30.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 16089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_socket.ko 2023-10-30T13:20:30.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_state.ko 2023-10-30T13:20:30.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_statistic.ko 2023-10-30T13:20:30.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_string.ko 2023-10-30T13:20:30.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 8553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpmss.ko 2023-10-30T13:20:30.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpudp.ko 2023-10-30T13:20:30.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_time.ko 2023-10-30T13:20:30.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_u32.ko 2023-10-30T13:20:30.572 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/ 2023-10-30T13:20:30.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/netlink_diag.ko 2023-10-30T13:20:30.572 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/ 2023-10-30T13:20:30.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 12425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/nsh.ko 2023-10-30T13:20:30.573 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/ 2023-10-30T13:20:30.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 380033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/openvswitch.ko 2023-10-30T13:20:30.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-geneve.ko 2023-10-30T13:20:30.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-gre.ko 2023-10-30T13:20:30.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 11793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-vxlan.ko 2023-10-30T13:20:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/ 2023-10-30T13:20:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/af_packet_diag.ko 2023-10-30T13:20:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/ 2023-10-30T13:20:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 110201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/phonet.ko 2023-10-30T13:20:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/pn_pep.ko 2023-10-30T13:20:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/ 2023-10-30T13:20:30.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 251481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds.ko 2023-10-30T13:20:30.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 164937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_rdma.ko 2023-10-30T13:20:30.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 67377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_tcp.ko 2023-10-30T13:20:30.575 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/ 2023-10-30T13:20:30.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 819657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/rxrpc.ko 2023-10-30T13:20:30.576 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/ 2023-10-30T13:20:30.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_csum.ko 2023-10-30T13:20:30.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_gact.ko 2023-10-30T13:20:30.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_ipt.ko 2023-10-30T13:20:30.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_mirred.ko 2023-10-30T13:20:30.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_nat.ko 2023-10-30T13:20:30.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 43449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_pedit.ko 2023-10-30T13:20:30.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_police.ko 2023-10-30T13:20:30.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_simple.ko 2023-10-30T13:20:30.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_skbedit.ko 2023-10-30T13:20:30.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_vlan.ko 2023-10-30T13:20:30.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_basic.ko 2023-10-30T13:20:30.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_bpf.ko 2023-10-30T13:20:30.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_cgroup.ko 2023-10-30T13:20:30.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_flow.ko 2023-10-30T13:20:30.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_fw.ko 2023-10-30T13:20:30.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 38489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_route.ko 2023-10-30T13:20:30.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 61409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_u32.ko 2023-10-30T13:20:30.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_cmp.ko 2023-10-30T13:20:30.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 13409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_ipset.ko 2023-10-30T13:20:30.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_meta.ko 2023-10-30T13:20:30.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_nbyte.ko 2023-10-30T13:20:30.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_text.ko 2023-10-30T13:20:30.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_u32.ko 2023-10-30T13:20:30.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 21649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_choke.ko 2023-10-30T13:20:30.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_codel.ko 2023-10-30T13:20:30.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 34633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_drr.ko 2023-10-30T13:20:30.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 33609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq.ko 2023-10-30T13:20:30.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq_codel.ko 2023-10-30T13:20:30.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 35433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_gred.ko 2023-10-30T13:20:30.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 49601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hfsc.ko 2023-10-30T13:20:30.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hhf.ko 2023-10-30T13:20:30.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 66393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_htb.ko 2023-10-30T13:20:30.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_ingress.ko 2023-10-30T13:20:30.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 30273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio.ko 2023-10-30T13:20:30.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio_lib.ko 2023-10-30T13:20:30.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_multiq.ko 2023-10-30T13:20:30.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 36137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_netem.ko 2023-10-30T13:20:30.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 27497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_pie.ko 2023-10-30T13:20:30.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_plug.ko 2023-10-30T13:20:30.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_prio.ko 2023-10-30T13:20:30.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 48793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_qfq.ko 2023-10-30T13:20:30.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 32905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_red.ko 2023-10-30T13:20:30.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 26881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfb.ko 2023-10-30T13:20:30.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 31353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfq.ko 2023-10-30T13:20:30.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_tbf.ko 2023-10-30T13:20:30.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 28105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_teql.ko 2023-10-30T13:20:30.584 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/ 2023-10-30T13:20:30.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 929745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp.ko 2023-10-30T13:20:30.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 19033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp_diag.ko 2023-10-30T13:20:30.585 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/ 2023-10-30T13:20:30.585 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/ 2023-10-30T13:20:30.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 336529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-30T13:20:30.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 54737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-30T13:20:30.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 1520161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/sunrpc.ko 2023-10-30T13:20:30.585 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/ 2023-10-30T13:20:30.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 707721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-30T13:20:30.586 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/ 2023-10-30T13:20:30.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/diag.ko 2023-10-30T13:20:30.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 862641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/tipc.ko 2023-10-30T13:20:30.586 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/ 2023-10-30T13:20:30.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/unix_diag.ko 2023-10-30T13:20:30.587 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/ 2023-10-30T13:20:30.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 85337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-30T13:20:30.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 68049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-30T13:20:30.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 96193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock.ko 2023-10-30T13:20:30.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 10705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-30T13:20:30.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 17521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-30T13:20:30.588 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/ 2023-10-30T13:20:30.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 125177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/x25.ko 2023-10-30T13:20:30.588 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/ 2023-10-30T13:20:30.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 22657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_algo.ko 2023-10-30T13:20:30.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 20649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-30T13:20:30.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 102257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_user.ko 2023-10-30T13:20:30.589 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/ 2023-10-30T13:20:30.589 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/ 2023-10-30T13:20:30.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 15537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/irqbypass.ko 2023-10-30T13:20:30.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 639863 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias 2023-10-30T13:20:30.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 623919 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias.bin 2023-10-30T13:20:30.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 7663 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin 2023-10-30T13:20:30.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 23428 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.alias.bin 2023-10-30T13:20:30.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 9821 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.bin 2023-10-30T13:20:30.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 70004 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.modinfo 2023-10-30T13:20:30.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 146313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep 2023-10-30T13:20:30.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 232548 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep.bin 2023-10-30T13:20:30.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 210 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.devname 2023-10-30T13:20:30.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 72606 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.order 2023-10-30T13:20:30.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 879 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.softdep 2023-10-30T13:20:30.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 254438 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols 2023-10-30T13:20:30.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 309363 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols.bin 2023-10-30T13:20:30.592 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/ 2023-10-30T13:20:30.592 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/ 2023-10-30T13:20:30.592 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/ 2023-10-30T13:20:30.592 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:20:30.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 173 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/changelog.Debian.gz 2023-10-30T13:20:30.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r-- root/root 691 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/copyright 2023-10-30T13:20:30.593 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:30.593 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:30.594 DEBUG:teuthology.orchestra.run.smithi031:> egrep '(submenu|menuentry.*6.6.0-rc5-geacf00cc51e2).*{' /boot/grub/grub.cfg 2023-10-30T13:20:30.601 INFO:teuthology.orchestra.run.smithi031.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:20:30.601 INFO:teuthology.orchestra.run.smithi031.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:20:30.601 INFO:teuthology.orchestra.run.smithi031.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:20:30.602 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-30T13:20:30.602 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:30.602 DEBUG:teuthology.orchestra.run.smithi031:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-30T13:20:30.645 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi031.front.sepia.ceph.com to complete... 2023-10-30T13:20:31.061 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub' 2023-10-30T13:20:31.062 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T13:20:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-10-30T13:20:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:20:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T13:20:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T13:20:31.643 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T13:20:31.649 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T13:20:31.811 INFO:teuthology.orchestra.run.smithi031.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T13:20:31.811 INFO:teuthology.orchestra.run.smithi031.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T13:20:31.811 INFO:teuthology.orchestra.run.smithi031.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T13:20:31.837 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-10-30T13:21:02.952 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-30T13:21:02.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-30T13:21:02.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:22:02.956 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T13:22:11.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-30T13:22:11.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:22:12.156 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-30T13:22:12.802 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-30T13:22:12.802 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-geacf00cc51e2"... 2023-10-30T13:22:12.802 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-10-30T13:22:12.854 INFO:teuthology.orchestra.run.smithi031.stdout:6.6.0-rc5-geacf00cc51e2 2023-10-30T13:22:12.854 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-geacf00cc51e2 vs 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:22:12.854 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T13:22:12.854 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-30T13:22:13.855 DEBUG:teuthology.parallel:result is None 2023-10-30T13:22:13.856 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T13:22:13.863 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T13:22:13.864 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T13:22:13.867 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T13:22:13.875 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T13:22:13.886 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T13:22:13.897 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T13:22:13.897 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T13:22:13.917 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T13:22:13.935 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T13:22:13.942 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T13:22:13.942 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T13:22:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:13.976 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T13:22:13.980 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T13:22:14.040 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:22:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:22:14.071 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:22:14.092 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:22:14.094 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T13:22:14.101 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T13:22:14.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T13:22:14.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T13:22:14.124 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T13:22:14.133 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T13:22:14.133 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T13:22:14.173 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T13:22:14.179 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T13:22:14.231 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T13:22:14.282 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:22:14.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T13:22:14.360 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T13:22:14.366 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T13:22:14.412 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:22:14.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T13:22:14.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-10-30T13:22:14.481 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2023-10-30T13:22:14.599 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T13:22:14.607 INFO:teuthology.task.internal:Starting timer... 2023-10-30T13:22:14.607 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T13:22:14.618 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T13:22:14.626 DEBUG:teuthology.task.selinux:Excluding smithi031: OS 'ubuntu' does not support SELinux 2023-10-30T13:22:14.626 DEBUG:teuthology.task.selinux:Excluding smithi033: OS 'ubuntu' does not support SELinux 2023-10-30T13:22:14.627 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T13:22:14.627 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-30T13:22:14.627 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T13:22:14.627 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T13:22:14.635 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-30T13:22:14.642 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-30T13:22:14.785 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T13:22:14.801 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T13:22:14.802 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi033.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T13:31:04.095 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi033.front.sepia.ceph.com')] 2023-10-30T13:31:04.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-30T13:31:04.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:31:04.175 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-30T13:31:04.314 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-30T13:31:04.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-30T13:31:04.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:31:04.393 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-10-30T13:31:04.494 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-30T13:31:04.495 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T13:31:04.505 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T13:31:04.506 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T13:31:04.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:31:04.510 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T13:31:04.510 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:31:04.547 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T13:31:04.547 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Command line: ntpd -gq 2023-10-30T13:31:04.548 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: ---------------------------------------------------- 2023-10-30T13:31:04.548 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T13:31:04.548 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T13:31:04.548 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: corporation. Support and training for ntp-4 are 2023-10-30T13:31:04.549 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: available at https://www.nwtime.org/support 2023-10-30T13:31:04.549 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: ---------------------------------------------------- 2023-10-30T13:31:04.549 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: proto: precision = 0.050 usec (-24) 2023-10-30T13:31:04.550 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: basedate set to 2022-02-04 2023-10-30T13:31:04.550 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: gps base set to 2022-02-06 (week 2196) 2023-10-30T13:31:04.550 INFO:teuthology.orchestra.run.smithi031.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:31:04.551 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:31:04.551 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:31:04.552 INFO:teuthology.orchestra.run.smithi031.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:31:04.553 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T13:31:04.553 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T13:31:04.553 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T13:31:04.553 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Listen normally on 3 ens1f0 172.21.15.31:123 2023-10-30T13:31:04.553 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Listen normally on 4 lo [::1]:123 2023-10-30T13:31:04.553 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:caba%4]:123 2023-10-30T13:31:04.554 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:04 ntpd[27862]: Listening on routing socket on fd #22 for interface updates 2023-10-30T13:31:04.563 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T13:31:04.563 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Command line: ntpd -gq 2023-10-30T13:31:04.564 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: ---------------------------------------------------- 2023-10-30T13:31:04.564 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T13:31:04.564 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T13:31:04.564 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: corporation. Support and training for ntp-4 are 2023-10-30T13:31:04.564 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: available at https://www.nwtime.org/support 2023-10-30T13:31:04.565 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: ---------------------------------------------------- 2023-10-30T13:31:04.566 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: proto: precision = 0.069 usec (-24) 2023-10-30T13:31:04.566 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: basedate set to 2022-02-04 2023-10-30T13:31:04.566 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: gps base set to 2022-02-06 (week 2196) 2023-10-30T13:31:04.566 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:31:04.567 INFO:teuthology.orchestra.run.smithi033.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:31:04.567 INFO:teuthology.orchestra.run.smithi033.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:31:04.568 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:31:04.568 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T13:31:04.568 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T13:31:04.569 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T13:31:04.569 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Listen normally on 3 ens1f0 172.21.15.33:123 2023-10-30T13:31:04.569 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Listen normally on 4 lo [::1]:123 2023-10-30T13:31:04.569 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:d294%4]:123 2023-10-30T13:31:04.569 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:04 ntpd[28512]: Listening on routing socket on fd #22 for interface updates 2023-10-30T13:31:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:05.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:05.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:05 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:05.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:05.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:05.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:05.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:05.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:05.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:05.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:05.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:05 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:06.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:06 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:06.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:06 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:06.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:06 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:06.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:06 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:06.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:06 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:06.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:06 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:07.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:07.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:07.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:07 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:07 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:07 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:07 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:07 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:07 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:07 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:08.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:08 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:08.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:08 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:08.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:08 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:08.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:08 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:08.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:08 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:08.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:08 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:09.548 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:09.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:09.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:09.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:09.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:09.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:09 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:09.566 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:09.566 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:09 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:10.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:10 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:10.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:10 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:10.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:10 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:10.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:10 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:10.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:10 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:10.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:10 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:11.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.549 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:11.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:11.551 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:11.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:11.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.552 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.553 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:11.553 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:11.553 INFO:teuthology.orchestra.run.smithi031.stdout:30 Oct 13:31:11 ntpd[27862]: ntpd: time slew -0.003119 s 2023-10-30T13:31:11.554 INFO:teuthology.orchestra.run.smithi031.stdout:ntpd: time slew -0.003119s 2023-10-30T13:31:11.554 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T13:31:11.554 INFO:teuthology.orchestra.run.smithi031.stderr:30 Oct 13:31:11 ntpd[27862]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T13:31:11.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.564 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.565 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:11.566 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:11.566 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:31:11.566 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:31:11.566 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:31:11.566 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:31:11.567 INFO:teuthology.orchestra.run.smithi033.stdout:30 Oct 13:31:11 ntpd[28512]: ntpd: time slew -0.004524 s 2023-10-30T13:31:11.567 INFO:teuthology.orchestra.run.smithi033.stdout:ntpd: time slew -0.004524s 2023-10-30T13:31:11.568 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T13:31:11.568 INFO:teuthology.orchestra.run.smithi033.stderr:30 Oct 13:31:11 ntpd[28512]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T13:31:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:31:11.637 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================== 2023-10-30T13:31:11.638 INFO:teuthology.orchestra.run.smithi031.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.638 INFO:teuthology.orchestra.run.smithi031.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.638 INFO:teuthology.orchestra.run.smithi031.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.638 INFO:teuthology.orchestra.run.smithi031.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.655 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:31:11.656 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-10-30T13:31:11.656 INFO:teuthology.orchestra.run.smithi033.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.656 INFO:teuthology.orchestra.run.smithi033.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.656 INFO:teuthology.orchestra.run.smithi033.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.657 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:31:11.658 INFO:teuthology.run_tasks:Running task install... 2023-10-30T13:31:11.666 DEBUG:teuthology.task.install:project ceph 2023-10-30T13:31:11.667 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}} 2023-10-30T13:31:11.667 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'} 2023-10-30T13:31:11.667 INFO:teuthology.task.install:Using flavor: default 2023-10-30T13:31:11.673 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-30T13:31:11.674 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-30T13:31:11.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-key list | grep Ceph 2023-10-30T13:31:11.675 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-key list | grep Ceph 2023-10-30T13:31:11.756 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T13:31:11.785 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T13:31:11.786 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T13:31:11.787 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T13:31:11.787 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T13:31:11.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:31:11.793 INFO:teuthology.orchestra.run.smithi031.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T13:31:11.848 INFO:teuthology.orchestra.run.smithi031.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T13:31:11.848 INFO:teuthology.orchestra.run.smithi031.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T13:31:11.849 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T13:31:11.849 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T13:31:11.849 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:31:11.944 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T13:31:11.944 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:31:11.997 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T13:31:11.997 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:31:12.114 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:31:12.115 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T13:31:12.129 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-10-30T13:31:12.167 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:31:12.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T13:31:12.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get update 2023-10-30T13:31:12.308 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T13:31:12.319 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T13:31:12.339 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T13:31:12.366 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T13:31:12.383 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T13:31:12.383 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T13:31:12.387 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T13:31:12.394 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T13:31:12.410 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T13:31:12.413 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T13:31:12.429 INFO:teuthology.orchestra.run.smithi031.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T13:31:12.431 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T13:31:12.451 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T13:31:12.459 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T13:31:12.473 INFO:teuthology.orchestra.run.smithi031.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T13:31:12.495 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T13:31:12.528 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T13:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 28.5 kB in 1s (47.9 kB/s) 2023-10-30T13:31:13.955 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2023-10-30T13:31:13.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:31:14.047 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2023-10-30T13:31:14.216 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2023-10-30T13:31:14.217 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2023-10-30T13:31:14.373 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:31:14.373 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:31:14.373 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:31:14.373 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:31:14.374 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T13:31:14.375 INFO:teuthology.orchestra.run.smithi031.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T13:31:14.376 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:31:14.376 INFO:teuthology.orchestra.run.smithi031.stdout:The following additional packages will be installed: 2023-10-30T13:31:14.376 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T13:31:14.376 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T13:31:14.376 INFO:teuthology.orchestra.run.smithi031.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:31:14.377 INFO:teuthology.orchestra.run.smithi031.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T13:31:14.377 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T13:31:14.377 INFO:teuthology.orchestra.run.smithi031.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T13:31:14.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T13:31:14.378 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T13:31:14.378 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T13:31:14.378 INFO:teuthology.orchestra.run.smithi031.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T13:31:14.378 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T13:31:14.378 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T13:31:14.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T13:31:14.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T13:31:14.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T13:31:14.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T13:31:14.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T13:31:14.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T13:31:14.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T13:31:14.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T13:31:14.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T13:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T13:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T13:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:Suggested packages: 2023-10-30T13:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T13:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T13:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T13:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T13:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T13:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T13:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout: python3-watchdog 2023-10-30T13:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:Recommended packages: 2023-10-30T13:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout: btrfs-tools 2023-10-30T13:31:14.523 INFO:teuthology.orchestra.run.smithi031.stdout:The following NEW packages will be installed: 2023-10-30T13:31:14.523 INFO:teuthology.orchestra.run.smithi031.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T13:31:14.524 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T13:31:14.524 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T13:31:14.524 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T13:31:14.525 INFO:teuthology.orchestra.run.smithi031.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:31:14.525 INFO:teuthology.orchestra.run.smithi031.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T13:31:14.525 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T13:31:14.527 INFO:teuthology.orchestra.run.smithi031.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T13:31:14.527 INFO:teuthology.orchestra.run.smithi031.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T13:31:14.527 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T13:31:14.527 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T13:31:14.528 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T13:31:14.528 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T13:31:14.528 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T13:31:14.528 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T13:31:14.529 INFO:teuthology.orchestra.run.smithi031.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T13:31:14.529 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T13:31:14.529 INFO:teuthology.orchestra.run.smithi031.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T13:31:14.529 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T13:31:14.530 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T13:31:14.530 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T13:31:14.530 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T13:31:14.530 INFO:teuthology.orchestra.run.smithi031.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T13:31:14.531 INFO:teuthology.orchestra.run.smithi031.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T13:31:14.531 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T13:31:14.531 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T13:31:14.531 INFO:teuthology.orchestra.run.smithi031.stdout: socat xmlstarlet 2023-10-30T13:31:14.677 INFO:teuthology.orchestra.run.smithi031.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T13:31:14.677 INFO:teuthology.orchestra.run.smithi031.stdout:Need to get 184 MB of archives. 2023-10-30T13:31:14.677 INFO:teuthology.orchestra.run.smithi031.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T13:31:14.677 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T13:31:14.935 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T13:31:15.030 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T13:31:15.052 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T13:31:15.063 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T13:31:15.065 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T13:31:15.070 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T13:31:15.073 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T13:31:15.083 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T13:31:15.223 INFO:teuthology.orchestra.run.smithi031.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T13:31:15.290 INFO:teuthology.orchestra.run.smithi031.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T13:31:15.291 INFO:teuthology.orchestra.run.smithi031.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T13:31:15.305 INFO:teuthology.orchestra.run.smithi031.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T13:31:15.305 INFO:teuthology.orchestra.run.smithi031.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T13:31:15.351 INFO:teuthology.orchestra.run.smithi031.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T13:31:15.556 INFO:teuthology.orchestra.run.smithi031.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T13:31:15.600 INFO:teuthology.orchestra.run.smithi031.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T13:31:15.676 INFO:teuthology.orchestra.run.smithi031.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T13:31:15.687 INFO:teuthology.orchestra.run.smithi031.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T13:31:15.734 INFO:teuthology.orchestra.run.smithi031.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T13:31:15.900 INFO:teuthology.orchestra.run.smithi031.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T13:31:15.907 INFO:teuthology.orchestra.run.smithi031.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T13:31:15.935 INFO:teuthology.orchestra.run.smithi031.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T13:31:16.055 INFO:teuthology.orchestra.run.smithi031.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T13:31:16.107 INFO:teuthology.orchestra.run.smithi031.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T13:31:16.298 INFO:teuthology.orchestra.run.smithi031.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T13:31:16.326 INFO:teuthology.orchestra.run.smithi031.stdout:Get:27 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T13:31:16.328 INFO:teuthology.orchestra.run.smithi031.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T13:31:16.329 INFO:teuthology.orchestra.run.smithi031.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T13:31:16.342 INFO:teuthology.orchestra.run.smithi031.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T13:31:16.662 INFO:teuthology.orchestra.run.smithi031.stdout:Get:31 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T13:31:16.715 INFO:teuthology.orchestra.run.smithi031.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T13:31:16.722 INFO:teuthology.orchestra.run.smithi031.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T13:31:16.723 INFO:teuthology.orchestra.run.smithi031.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T13:31:16.735 INFO:teuthology.orchestra.run.smithi031.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T13:31:16.735 INFO:teuthology.orchestra.run.smithi031.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T13:31:16.737 INFO:teuthology.orchestra.run.smithi031.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T13:31:16.739 INFO:teuthology.orchestra.run.smithi031.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T13:31:16.741 INFO:teuthology.orchestra.run.smithi031.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T13:31:16.742 INFO:teuthology.orchestra.run.smithi031.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T13:31:16.908 INFO:teuthology.orchestra.run.smithi031.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T13:31:17.357 INFO:teuthology.orchestra.run.smithi031.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T13:31:17.367 INFO:teuthology.orchestra.run.smithi031.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T13:31:17.373 INFO:teuthology.orchestra.run.smithi031.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T13:31:17.377 INFO:teuthology.orchestra.run.smithi031.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T13:31:17.396 INFO:teuthology.orchestra.run.smithi031.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T13:31:17.400 INFO:teuthology.orchestra.run.smithi031.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T13:31:17.404 INFO:teuthology.orchestra.run.smithi031.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T13:31:17.442 INFO:teuthology.orchestra.run.smithi031.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T13:31:17.456 INFO:teuthology.orchestra.run.smithi031.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T13:31:17.461 INFO:teuthology.orchestra.run.smithi031.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T13:31:17.712 INFO:teuthology.orchestra.run.smithi031.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T13:31:17.713 INFO:teuthology.orchestra.run.smithi031.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T13:31:17.715 INFO:teuthology.orchestra.run.smithi031.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T13:31:17.730 INFO:teuthology.orchestra.run.smithi031.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T13:31:17.798 INFO:teuthology.orchestra.run.smithi031.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T13:31:17.799 INFO:teuthology.orchestra.run.smithi031.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T13:31:17.808 INFO:teuthology.orchestra.run.smithi031.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T13:31:17.813 INFO:teuthology.orchestra.run.smithi031.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T13:31:17.876 INFO:teuthology.orchestra.run.smithi031.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T13:31:17.890 INFO:teuthology.orchestra.run.smithi031.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T13:31:17.966 INFO:teuthology.orchestra.run.smithi031.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T13:31:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T13:31:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T13:31:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T13:31:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T13:31:18.166 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T13:31:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T13:31:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T13:31:18.216 INFO:teuthology.orchestra.run.smithi031.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T13:31:18.253 INFO:teuthology.orchestra.run.smithi031.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T13:31:18.261 INFO:teuthology.orchestra.run.smithi031.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T13:31:18.349 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 28.5 kB in 6s (4,739 B/s) 2023-10-30T13:31:18.452 INFO:teuthology.orchestra.run.smithi031.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T13:31:19.260 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-30T13:31:19.275 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:31:19.283 INFO:teuthology.orchestra.run.smithi031.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T13:31:19.284 INFO:teuthology.orchestra.run.smithi031.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T13:31:19.286 INFO:teuthology.orchestra.run.smithi031.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T13:31:19.286 INFO:teuthology.orchestra.run.smithi031.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T13:31:19.287 INFO:teuthology.orchestra.run.smithi031.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T13:31:19.288 INFO:teuthology.orchestra.run.smithi031.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T13:31:19.314 INFO:teuthology.orchestra.run.smithi031.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T13:31:19.316 INFO:teuthology.orchestra.run.smithi031.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T13:31:19.317 INFO:teuthology.orchestra.run.smithi031.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T13:31:19.338 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-30T13:31:19.385 INFO:teuthology.orchestra.run.smithi031.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T13:31:19.528 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-10-30T13:31:19.528 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-10-30T13:31:19.548 INFO:teuthology.orchestra.run.smithi031.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T13:31:19.700 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:31:19.701 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:31:19.701 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:31:19.702 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:31:19.703 INFO:teuthology.orchestra.run.smithi033.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T13:31:19.703 INFO:teuthology.orchestra.run.smithi033.stdout: python2.7-minimal 2023-10-30T13:31:19.703 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:31:19.703 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-10-30T13:31:19.704 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T13:31:19.704 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T13:31:19.704 INFO:teuthology.orchestra.run.smithi033.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:31:19.704 INFO:teuthology.orchestra.run.smithi033.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T13:31:19.706 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T13:31:19.707 INFO:teuthology.orchestra.run.smithi033.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T13:31:19.707 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T13:31:19.707 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T13:31:19.708 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T13:31:19.708 INFO:teuthology.orchestra.run.smithi033.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T13:31:19.708 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T13:31:19.708 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T13:31:19.709 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T13:31:19.709 INFO:teuthology.orchestra.run.smithi033.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T13:31:19.709 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T13:31:19.709 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T13:31:19.710 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T13:31:19.710 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T13:31:19.710 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T13:31:19.710 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T13:31:19.711 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T13:31:19.711 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T13:31:19.711 INFO:teuthology.orchestra.run.smithi033.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T13:31:19.711 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-10-30T13:31:19.712 INFO:teuthology.orchestra.run.smithi033.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T13:31:19.712 INFO:teuthology.orchestra.run.smithi033.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T13:31:19.712 INFO:teuthology.orchestra.run.smithi033.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T13:31:19.712 INFO:teuthology.orchestra.run.smithi033.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T13:31:19.713 INFO:teuthology.orchestra.run.smithi033.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T13:31:19.713 INFO:teuthology.orchestra.run.smithi033.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T13:31:19.713 INFO:teuthology.orchestra.run.smithi033.stdout: python3-watchdog 2023-10-30T13:31:19.713 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2023-10-30T13:31:19.714 INFO:teuthology.orchestra.run.smithi033.stdout: btrfs-tools 2023-10-30T13:31:19.750 INFO:teuthology.orchestra.run.smithi031.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T13:31:19.751 INFO:teuthology.orchestra.run.smithi031.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T13:31:19.753 INFO:teuthology.orchestra.run.smithi031.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T13:31:19.753 INFO:teuthology.orchestra.run.smithi031.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T13:31:19.754 INFO:teuthology.orchestra.run.smithi031.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T13:31:19.754 INFO:teuthology.orchestra.run.smithi031.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T13:31:19.806 INFO:teuthology.orchestra.run.smithi031.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T13:31:19.814 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-10-30T13:31:19.814 INFO:teuthology.orchestra.run.smithi033.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T13:31:19.814 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T13:31:19.815 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T13:31:19.815 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T13:31:19.817 INFO:teuthology.orchestra.run.smithi033.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:31:19.817 INFO:teuthology.orchestra.run.smithi033.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T13:31:19.818 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T13:31:19.818 INFO:teuthology.orchestra.run.smithi033.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T13:31:19.818 INFO:teuthology.orchestra.run.smithi033.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T13:31:19.818 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T13:31:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T13:31:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T13:31:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T13:31:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T13:31:19.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T13:31:19.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T13:31:19.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T13:31:19.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T13:31:19.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T13:31:19.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T13:31:19.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T13:31:19.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T13:31:19.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T13:31:19.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T13:31:19.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T13:31:19.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T13:31:19.823 INFO:teuthology.orchestra.run.smithi033.stdout: socat xmlstarlet 2023-10-30T13:31:19.838 INFO:teuthology.orchestra.run.smithi031.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T13:31:19.900 INFO:teuthology.orchestra.run.smithi031.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T13:31:19.935 INFO:teuthology.orchestra.run.smithi031.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T13:31:19.953 INFO:teuthology.orchestra.run.smithi031.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T13:31:19.979 INFO:teuthology.orchestra.run.smithi031.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T13:31:20.005 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T13:31:20.006 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 184 MB of archives. 2023-10-30T13:31:20.006 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T13:31:20.006 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T13:31:20.088 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T13:31:20.113 INFO:teuthology.orchestra.run.smithi031.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T13:31:20.133 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T13:31:20.148 INFO:teuthology.orchestra.run.smithi031.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T13:31:20.158 INFO:teuthology.orchestra.run.smithi031.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T13:31:20.168 INFO:teuthology.orchestra.run.smithi031.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T13:31:20.170 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T13:31:20.176 INFO:teuthology.orchestra.run.smithi031.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T13:31:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T13:31:20.371 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T13:31:20.379 INFO:teuthology.orchestra.run.smithi031.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T13:31:20.446 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T13:31:20.457 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T13:31:20.503 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T13:31:20.668 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T13:31:20.677 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T13:31:20.703 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T13:31:20.856 INFO:teuthology.orchestra.run.smithi031.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T13:31:20.861 INFO:teuthology.orchestra.run.smithi031.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T13:31:20.875 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T13:31:20.989 INFO:teuthology.orchestra.run.smithi031.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T13:31:21.037 INFO:teuthology.orchestra.run.smithi031.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T13:31:21.037 INFO:teuthology.orchestra.run.smithi031.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T13:31:21.042 INFO:teuthology.orchestra.run.smithi031.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T13:31:21.044 INFO:teuthology.orchestra.run.smithi031.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T13:31:21.060 INFO:teuthology.orchestra.run.smithi031.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T13:31:21.068 INFO:teuthology.orchestra.run.smithi031.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T13:31:21.070 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T13:31:21.128 INFO:teuthology.orchestra.run.smithi031.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T13:31:21.293 INFO:teuthology.orchestra.run.smithi031.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T13:31:21.307 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T13:31:21.437 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T13:31:21.445 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T13:31:21.446 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T13:31:21.458 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T13:31:21.459 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T13:31:21.459 INFO:teuthology.orchestra.run.smithi031.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T13:31:21.461 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T13:31:21.462 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T13:31:21.464 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T13:31:21.466 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T13:31:21.631 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T13:31:21.748 INFO:teuthology.orchestra.run.smithi031.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T13:31:21.774 INFO:teuthology.orchestra.run.smithi031.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T13:31:21.817 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T13:31:21.894 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T13:31:21.965 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T13:31:21.967 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T13:31:21.971 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T13:31:21.972 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T13:31:22.042 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T13:31:22.062 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T13:31:22.072 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T13:31:22.078 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T13:31:22.082 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T13:31:22.100 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T13:31:22.103 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T13:31:22.105 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T13:31:22.147 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T13:31:22.161 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T13:31:22.165 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T13:31:22.419 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T13:31:22.502 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T13:31:22.592 INFO:teuthology.orchestra.run.smithi033.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T13:31:22.634 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 184 MB in 7s (24.6 MB/s) 2023-10-30T13:31:22.649 INFO:teuthology.orchestra.run.smithi033.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T13:31:22.736 INFO:teuthology.orchestra.run.smithi033.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T13:31:22.739 INFO:teuthology.orchestra.run.smithi033.stdout:Get:48 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T13:31:22.750 INFO:teuthology.orchestra.run.smithi033.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T13:31:22.771 INFO:teuthology.orchestra.run.smithi033.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T13:31:22.816 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T13:31:22.858 INFO:teuthology.orchestra.run.smithi033.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T13:31:22.876 INFO:teuthology.orchestra.run.smithi033.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T13:31:22.886 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-30T13:31:22.889 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T13:31:22.905 INFO:teuthology.orchestra.run.smithi033.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T13:31:22.918 INFO:teuthology.orchestra.run.smithi033.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T13:31:22.940 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T13:31:22.955 INFO:teuthology.orchestra.run.smithi033.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T13:31:22.963 INFO:teuthology.orchestra.run.smithi033.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T13:31:23.155 INFO:teuthology.orchestra.run.smithi033.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T13:31:23.259 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T13:31:23.273 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T13:31:23.317 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:31:23.687 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T13:31:23.702 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T13:31:23.754 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T13:31:23.964 INFO:teuthology.orchestra.run.smithi033.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T13:31:23.967 INFO:teuthology.orchestra.run.smithi033.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T13:31:23.970 INFO:teuthology.orchestra.run.smithi033.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T13:31:23.971 INFO:teuthology.orchestra.run.smithi033.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T13:31:23.972 INFO:teuthology.orchestra.run.smithi033.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T13:31:23.975 INFO:teuthology.orchestra.run.smithi033.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T13:31:24.021 INFO:teuthology.orchestra.run.smithi033.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T13:31:24.024 INFO:teuthology.orchestra.run.smithi033.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T13:31:24.024 INFO:teuthology.orchestra.run.smithi033.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T13:31:24.078 INFO:teuthology.orchestra.run.smithi033.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T13:31:24.081 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T13:31:24.097 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T13:31:24.182 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:31:24.229 INFO:teuthology.orchestra.run.smithi033.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T13:31:24.241 INFO:teuthology.orchestra.run.smithi033.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T13:31:24.440 INFO:teuthology.orchestra.run.smithi033.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T13:31:24.446 INFO:teuthology.orchestra.run.smithi033.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T13:31:24.449 INFO:teuthology.orchestra.run.smithi033.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T13:31:24.499 INFO:teuthology.orchestra.run.smithi033.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T13:31:24.509 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T13:31:24.524 INFO:teuthology.orchestra.run.smithi033.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T13:31:24.524 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T13:31:24.531 INFO:teuthology.orchestra.run.smithi033.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T13:31:24.556 INFO:teuthology.orchestra.run.smithi033.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T13:31:24.567 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:31:24.608 INFO:teuthology.orchestra.run.smithi033.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T13:31:24.628 INFO:teuthology.orchestra.run.smithi033.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T13:31:24.655 INFO:teuthology.orchestra.run.smithi033.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T13:31:24.784 INFO:teuthology.orchestra.run.smithi033.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T13:31:24.815 INFO:teuthology.orchestra.run.smithi033.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T13:31:24.835 INFO:teuthology.orchestra.run.smithi033.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T13:31:24.842 INFO:teuthology.orchestra.run.smithi033.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T13:31:24.904 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T13:31:24.920 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T13:31:24.922 INFO:teuthology.orchestra.run.smithi033.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T13:31:24.962 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:31:25.086 INFO:teuthology.orchestra.run.smithi033.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T13:31:25.257 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T13:31:25.272 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T13:31:25.332 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:31:25.548 INFO:teuthology.orchestra.run.smithi033.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T13:31:25.553 INFO:teuthology.orchestra.run.smithi033.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T13:31:25.679 INFO:teuthology.orchestra.run.smithi033.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T13:31:25.726 INFO:teuthology.orchestra.run.smithi033.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T13:31:25.727 INFO:teuthology.orchestra.run.smithi033.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T13:31:25.733 INFO:teuthology.orchestra.run.smithi033.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T13:31:25.735 INFO:teuthology.orchestra.run.smithi033.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T13:31:25.750 INFO:teuthology.orchestra.run.smithi033.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T13:31:25.758 INFO:teuthology.orchestra.run.smithi033.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T13:31:25.785 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T13:31:25.801 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:31:25.818 INFO:teuthology.orchestra.run.smithi033.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T13:31:25.852 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:31:25.983 INFO:teuthology.orchestra.run.smithi033.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T13:31:26.149 INFO:teuthology.orchestra.run.smithi033.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T13:31:26.230 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T13:31:26.233 INFO:teuthology.orchestra.run.smithi033.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T13:31:26.234 INFO:teuthology.orchestra.run.smithi033.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T13:31:26.245 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:31:26.251 INFO:teuthology.orchestra.run.smithi033.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T13:31:26.288 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:31:26.292 INFO:teuthology.orchestra.run.smithi033.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T13:31:26.297 INFO:teuthology.orchestra.run.smithi033.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T13:31:26.299 INFO:teuthology.orchestra.run.smithi033.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T13:31:26.330 INFO:teuthology.orchestra.run.smithi033.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T13:31:26.434 INFO:teuthology.orchestra.run.smithi033.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T13:31:26.460 INFO:teuthology.orchestra.run.smithi033.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T13:31:26.607 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T13:31:26.623 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:31:26.691 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:31:26.750 INFO:teuthology.orchestra.run.smithi033.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T13:31:26.750 INFO:teuthology.orchestra.run.smithi033.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T13:31:27.027 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T13:31:27.042 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T13:31:27.085 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:31:27.688 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librados2. 2023-10-30T13:31:27.704 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:27.747 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:28.191 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librbd1. 2023-10-30T13:31:28.206 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:28.249 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:28.610 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T13:31:28.625 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:28.668 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:28.937 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rados. 2023-10-30T13:31:28.953 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:28.996 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:29.356 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T13:31:29.372 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:29.415 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:29.502 INFO:teuthology.orchestra.run.smithi033.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T13:31:29.505 INFO:teuthology.orchestra.run.smithi033.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T13:31:29.510 INFO:teuthology.orchestra.run.smithi033.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T13:31:29.511 INFO:teuthology.orchestra.run.smithi033.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T13:31:29.512 INFO:teuthology.orchestra.run.smithi033.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T13:31:29.513 INFO:teuthology.orchestra.run.smithi033.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T13:31:29.667 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T13:31:29.682 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:29.725 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:30.003 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T13:31:30.019 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:30.061 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:30.123 INFO:teuthology.orchestra.run.smithi033.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T13:31:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T13:31:30.380 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T13:31:30.422 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T13:31:30.700 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T13:31:30.703 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 184 MB in 10s (17.9 MB/s) 2023-10-30T13:31:30.715 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:30.758 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:30.843 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T13:31:30.926 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130506 files and directories currently installed.) 2023-10-30T13:31:30.931 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T13:31:30.975 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T13:31:31.102 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T13:31:31.117 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T13:31:31.160 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:31:31.303 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T13:31:31.323 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T13:31:31.361 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:31:31.455 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librgw2. 2023-10-30T13:31:31.470 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:31.504 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:31.656 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T13:31:31.676 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T13:31:31.705 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T13:31:31.932 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T13:31:31.947 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:31.990 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:32.017 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T13:31:32.038 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T13:31:32.101 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:31:32.334 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T13:31:32.350 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T13:31:32.393 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:31:32.437 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T13:31:32.458 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T13:31:32.529 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:31:32.729 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T13:31:32.745 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:32.787 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:32.874 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T13:31:32.895 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T13:31:32.932 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:31:33.140 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-common. 2023-10-30T13:31:33.156 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:33.190 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:33.285 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T13:31:33.306 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T13:31:33.335 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:31:33.906 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T13:31:33.926 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:31:34.005 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:31:34.279 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-base. 2023-10-30T13:31:34.294 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:34.347 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:34.442 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T13:31:34.463 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:31:34.501 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:31:34.829 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T13:31:34.850 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:31:34.879 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:31:35.024 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T13:31:35.038 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T13:31:35.082 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:31:35.207 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T13:31:35.228 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T13:31:35.257 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:31:35.402 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T13:31:35.417 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T13:31:35.460 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:31:35.585 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librados2. 2023-10-30T13:31:35.606 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:35.643 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:35.772 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T13:31:35.787 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T13:31:35.821 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:31:36.132 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T13:31:36.148 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T13:31:36.190 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:31:36.371 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librbd1. 2023-10-30T13:31:36.392 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:36.429 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:36.493 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T13:31:36.509 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T13:31:36.551 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:31:36.849 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T13:31:36.862 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T13:31:36.869 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:36.878 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T13:31:36.920 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T13:31:36.932 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:37.168 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rados. 2023-10-30T13:31:37.181 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:37.190 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-portend. 2023-10-30T13:31:37.206 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T13:31:37.218 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:37.240 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T13:31:37.521 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T13:31:37.542 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:37.543 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T13:31:37.563 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T13:31:37.579 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:37.751 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T13:31:37.856 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T13:31:37.869 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:37.907 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:38.054 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T13:31:38.069 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T13:31:38.112 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:31:38.210 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T13:31:38.231 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:38.268 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T13:31:38.481 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T13:31:38.523 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T13:31:38.588 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T13:31:38.608 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T13:31:38.638 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T13:31:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T13:31:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T13:31:38.890 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T13:31:38.902 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:38.940 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T13:31:39.245 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T13:31:39.261 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T13:31:39.304 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:31:39.327 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T13:31:39.348 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T13:31:39.385 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:31:39.565 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-mako. 2023-10-30T13:31:39.580 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T13:31:39.614 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:31:39.705 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librgw2. 2023-10-30T13:31:39.725 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:39.763 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:39.926 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T13:31:39.941 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T13:31:39.975 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:31:40.287 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T13:31:40.302 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T13:31:40.307 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T13:31:40.328 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:40.345 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:31:40.365 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-webob. 2023-10-30T13:31:40.663 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T13:31:40.706 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:31:40.710 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T13:31:40.731 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T13:31:40.760 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:31:41.050 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T13:31:41.065 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T13:31:41.079 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T13:31:41.100 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:41.111 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:31:41.138 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:41.428 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T13:31:41.443 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T13:31:41.494 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:31:41.717 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-common. 2023-10-30T13:31:41.737 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:41.772 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-paste. 2023-10-30T13:31:41.774 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:41.788 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T13:31:41.830 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:31:42.141 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T13:31:42.157 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T13:31:42.200 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:31:42.453 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T13:31:42.468 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T13:31:42.510 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:31:42.798 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T13:31:42.813 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T13:31:42.855 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T13:31:43.183 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T13:31:43.189 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-base. 2023-10-30T13:31:43.198 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T13:31:43.210 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:43.241 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:31:43.257 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:43.604 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T13:31:43.619 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T13:31:43.662 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T13:31:43.973 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-idna. 2023-10-30T13:31:43.988 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T13:31:44.031 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T13:31:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T13:31:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T13:31:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:31:44.300 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T13:31:44.316 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T13:31:44.350 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:31:44.530 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T13:31:44.550 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T13:31:44.588 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:31:44.637 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-requests. 2023-10-30T13:31:44.652 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T13:31:44.695 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:31:44.916 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T13:31:44.937 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T13:31:44.974 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:31:44.981 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T13:31:44.996 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T13:31:45.039 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:31:45.300 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T13:31:45.311 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T13:31:45.315 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:45.331 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T13:31:45.350 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:45.369 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:31:45.647 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T13:31:45.659 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T13:31:45.688 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:31:45.771 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T13:31:45.787 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:45.829 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:46.000 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T13:31:46.020 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T13:31:46.058 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T13:31:46.116 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T13:31:46.131 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T13:31:46.165 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T13:31:46.361 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-portend. 2023-10-30T13:31:46.382 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T13:31:46.419 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T13:31:46.485 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T13:31:46.500 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:46.543 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:46.747 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T13:31:46.768 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T13:31:46.805 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T13:31:46.971 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T13:31:46.987 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:47.029 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:47.075 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T13:31:47.096 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T13:31:47.135 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:31:47.481 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T13:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T13:31:47.539 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T13:31:47.853 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T13:31:47.871 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T13:31:47.908 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T13:31:48.228 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T13:31:48.249 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T13:31:48.286 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:31:48.533 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T13:31:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:48.589 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-mako. 2023-10-30T13:31:48.591 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:48.610 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T13:31:48.647 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:31:49.017 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T13:31:49.038 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T13:31:49.075 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:31:49.353 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T13:31:49.374 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T13:31:49.411 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:31:49.739 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webob. 2023-10-30T13:31:49.760 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T13:31:49.797 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:31:49.803 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph. 2023-10-30T13:31:49.818 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:49.861 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:50.147 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T13:31:50.150 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T13:31:50.162 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:50.171 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T13:31:50.196 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:50.236 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:31:50.483 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T13:31:50.498 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:50.533 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:50.545 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T13:31:50.566 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T13:31:50.603 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:31:50.877 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package cephadm. 2023-10-30T13:31:50.893 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:50.906 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-paste. 2023-10-30T13:31:50.927 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T13:31:50.935 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:50.980 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:31:51.280 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T13:31:51.296 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T13:31:51.317 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T13:31:51.329 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T13:31:51.346 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T13:31:51.367 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:31:51.657 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T13:31:51.673 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:31:51.716 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:31:51.912 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T13:31:51.932 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T13:31:51.969 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:31:52.223 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T13:31:52.243 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T13:31:52.272 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T13:31:52.592 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T13:31:52.613 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T13:31:52.650 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:31:52.737 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-babel. 2023-10-30T13:31:52.752 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:31:52.795 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:31:53.013 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T13:31:53.034 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T13:31:53.072 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T13:31:53.106 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T13:31:53.121 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T13:31:53.173 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T13:31:53.416 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-idna. 2023-10-30T13:31:53.437 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T13:31:53.474 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T13:31:53.518 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T13:31:53.533 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:53.576 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:53.760 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T13:31:53.781 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T13:31:53.810 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:31:53.946 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T13:31:53.961 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T13:31:54.004 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:31:54.307 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T13:31:54.322 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T13:31:54.365 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T13:31:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-routes. 2023-10-30T13:31:54.658 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T13:31:54.692 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:31:54.923 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests. 2023-10-30T13:31:54.944 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T13:31:54.972 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:31:55.042 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T13:31:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:55.095 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:55.267 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T13:31:55.288 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T13:31:55.325 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:31:55.498 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T13:31:55.513 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T13:31:55.557 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:31:55.636 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T13:31:55.649 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:55.686 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:56.035 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T13:31:56.050 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T13:31:56.074 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T13:31:56.093 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:31:56.096 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:56.157 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:56.469 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T13:31:56.488 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T13:31:56.490 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T13:31:56.504 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T13:31:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T13:31:56.546 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:31:56.850 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T13:31:56.865 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T13:31:56.888 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T13:31:56.908 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:31:56.909 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:56.946 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:57.333 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T13:31:57.354 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:57.382 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:57.541 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T13:31:57.555 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:31:57.599 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:58.278 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T13:31:58.299 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:31:58.357 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:31:58.616 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T13:31:58.632 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T13:31:58.752 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T13:31:59.048 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T13:31:59.064 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T13:31:59.115 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:31:59.435 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T13:31:59.450 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T13:31:59.493 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:31:59.804 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T13:31:59.819 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T13:31:59.862 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T13:32:00.251 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T13:32:00.266 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T13:32:00.300 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T13:32:00.472 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph. 2023-10-30T13:32:00.485 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:00.522 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:00.612 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T13:32:00.628 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T13:32:00.670 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:32:00.833 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T13:32:00.854 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:00.891 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:00.973 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T13:32:00.989 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T13:32:01.032 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T13:32:01.254 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T13:32:01.275 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:01.327 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:01.360 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T13:32:01.376 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T13:32:01.418 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:32:01.723 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephadm. 2023-10-30T13:32:01.744 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:01.781 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:02.090 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T13:32:02.105 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:02.148 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:02.367 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T13:32:02.389 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T13:32:02.425 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T13:32:02.468 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T13:32:02.483 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T13:32:02.527 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:32:02.762 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T13:32:02.774 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:32:02.811 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:32:02.863 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T13:32:02.879 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:32:02.921 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:32:03.190 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package jq. 2023-10-30T13:32:03.206 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:32:03.249 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T13:32:03.501 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package socat. 2023-10-30T13:32:03.517 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T13:32:03.559 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:32:03.892 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-babel. 2023-10-30T13:32:03.913 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T13:32:03.913 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:32:03.929 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T13:32:03.958 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:32:03.971 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:32:04.261 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T13:32:04.282 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T13:32:04.308 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-test. 2023-10-30T13:32:04.311 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T13:32:04.323 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:04.367 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:04.656 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T13:32:04.677 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:04.714 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:05.051 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T13:32:05.072 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T13:32:05.109 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:32:05.454 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T13:32:05.475 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T13:32:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T13:32:05.815 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-routes. 2023-10-30T13:32:05.836 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T13:32:05.898 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:32:06.213 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T13:32:06.229 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:06.243 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T13:32:06.264 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:06.270 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:06.301 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:06.642 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T13:32:06.658 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:06.691 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:06.738 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T13:32:06.750 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T13:32:06.788 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:32:06.970 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T13:32:06.986 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T13:32:07.028 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T13:32:07.291 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T13:32:07.313 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T13:32:07.314 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T13:32:07.330 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:07.381 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:07.383 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:32:07.633 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T13:32:07.650 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:07.691 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:07.728 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T13:32:07.749 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T13:32:07.778 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:32:07.927 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T13:32:07.943 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:07.986 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:08.106 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T13:32:08.130 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T13:32:08.164 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:32:08.489 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T13:32:08.505 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T13:32:08.548 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:32:09.094 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T13:32:09.111 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T13:32:09.152 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:32:09.190 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T13:32:09.212 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:09.256 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:09.572 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T13:32:09.588 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T13:32:09.622 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T13:32:09.917 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T13:32:09.933 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T13:32:09.974 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T13:32:10.234 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T13:32:10.256 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T13:32:10.320 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T13:32:10.333 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T13:32:10.336 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T13:32:10.378 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:32:10.645 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T13:32:10.667 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T13:32:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T13:32:10.705 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T13:32:10.711 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:32:10.747 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T13:32:11.073 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T13:32:11.095 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T13:32:11.100 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-py. 2023-10-30T13:32:11.117 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T13:32:11.139 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:32:11.158 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T13:32:11.451 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T13:32:11.472 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T13:32:11.509 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T13:32:11.512 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T13:32:11.528 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T13:32:11.570 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:32:11.906 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T13:32:11.928 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T13:32:11.964 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T13:32:12.024 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T13:32:12.040 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T13:32:12.082 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:32:12.276 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T13:32:12.297 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T13:32:12.342 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:32:12.376 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-toml. 2023-10-30T13:32:12.392 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T13:32:12.434 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T13:32:12.670 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T13:32:12.692 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T13:32:12.737 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T13:32:12.746 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T13:32:12.762 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T13:32:12.804 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:32:13.073 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T13:32:13.095 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T13:32:13.140 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:32:13.149 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T13:32:13.166 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T13:32:13.207 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:32:13.477 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T13:32:13.493 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T13:32:13.526 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:32:13.845 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T13:32:13.860 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:13.903 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:14.247 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package radosgw. 2023-10-30T13:32:14.262 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:14.289 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T13:32:14.305 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:14.311 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T13:32:14.356 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:32:14.742 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T13:32:14.764 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:32:14.808 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T13:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:15.078 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package jq. 2023-10-30T13:32:15.099 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:32:15.153 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T13:32:15.195 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T13:32:15.406 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package socat. 2023-10-30T13:32:15.427 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T13:32:15.471 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:32:15.497 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:32:15.606 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:32:15.792 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T13:32:15.813 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T13:32:15.858 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:32:15.870 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T13:32:16.255 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-test. 2023-10-30T13:32:16.269 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T13:32:16.276 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:16.320 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:16.901 INFO:teuthology.orchestra.run.smithi031.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T13:32:16.959 INFO:teuthology.orchestra.run.smithi031.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-30T13:32:17.008 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:17.646 INFO:teuthology.orchestra.run.smithi031.stdout:Adding system user cephadm....done 2023-10-30T13:32:18.040 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:32:18.442 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:32:18.693 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:32:18.791 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T13:32:18.813 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:18.819 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:32:18.858 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:19.087 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T13:32:19.279 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T13:32:19.301 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:19.345 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:19.363 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:32:19.489 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T13:32:19.632 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T13:32:19.653 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T13:32:19.690 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T13:32:19.841 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:32:19.959 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T13:32:19.981 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:20.025 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:20.209 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T13:32:20.287 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T13:32:20.309 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:32:20.353 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:20.494 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T13:32:20.631 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T13:32:20.644 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:20.681 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T13:32:21.001 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T13:32:21.023 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T13:32:21.029 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:32:21.059 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:32:21.397 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T13:32:21.419 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T13:32:21.422 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:32:21.463 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:32:21.707 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:32:21.850 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T13:32:21.872 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T13:32:21.908 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T13:32:21.958 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:22.236 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T13:32:22.243 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:32:22.258 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T13:32:22.302 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T13:32:22.343 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T13:32:22.594 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:32:22.606 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T13:32:22.628 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T13:32:22.672 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:32:22.862 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:32:22.988 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T13:32:23.017 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T13:32:23.039 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T13:32:23.083 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T13:32:23.373 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:32:23.420 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-py. 2023-10-30T13:32:23.441 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T13:32:23.486 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T13:32:23.499 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T13:32:23.792 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:32:23.831 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T13:32:23.853 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T13:32:23.898 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:32:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:32:24.369 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T13:32:24.391 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T13:32:24.426 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:32:24.452 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:32:24.746 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-toml. 2023-10-30T13:32:24.768 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T13:32:24.812 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T13:32:25.141 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T13:32:25.154 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:32:25.163 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T13:32:25.207 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:32:25.431 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:32:25.557 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:32:25.561 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T13:32:25.583 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T13:32:25.627 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:32:25.674 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:32:25.922 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T13:32:25.944 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T13:32:25.983 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T13:32:25.988 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:32:26.393 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T13:32:26.652 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:32:26.767 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package radosgw. 2023-10-30T13:32:26.789 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:26.834 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:27.046 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:32:27.356 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T13:32:27.628 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T13:32:27.641 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:32:27.678 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:27.690 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:32:27.993 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T13:32:28.058 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:32:28.250 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:32:28.309 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:32:28.376 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:32:28.435 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:28.650 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T13:32:28.745 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T13:32:29.019 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T13:32:29.104 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:32:29.465 INFO:teuthology.orchestra.run.smithi033.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T13:32:29.547 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:30.135 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user cephadm....done 2023-10-30T13:32:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:32:30.413 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:32:30.528 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T13:32:30.797 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:32:30.846 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:32:30.972 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:32:31.098 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:32:31.211 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:32:31.399 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:32:31.555 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:32:31.650 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T13:32:31.891 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T13:32:31.901 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:32:32.027 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:32:32.153 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:32:32.167 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:32:32.318 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T13:32:32.480 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:32:32.678 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:32:32.748 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T13:32:33.016 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T13:32:33.063 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T13:32:33.275 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:32:33.373 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T13:32:33.535 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T13:32:33.624 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T13:32:33.810 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:32:33.936 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T13:32:34.075 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:32:34.230 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:32:34.360 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:32:34.407 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:32:34.661 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:32:34.715 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:32:34.937 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:35.024 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:32:35.222 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:32:35.284 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:32:35.381 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T13:32:35.641 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:32:35.721 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T13:32:35.883 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:32:35.995 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:32:36.054 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T13:32:36.121 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:32:36.444 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:32:36.531 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:32:36.570 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T13:32:36.799 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T13:32:36.846 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:32:37.059 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:32:37.130 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:32:37.184 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T13:32:37.432 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:32:37.511 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:32:37.837 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:32:38.125 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:32:38.138 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T13:32:38.264 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T13:32:38.393 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:32:38.544 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:32:38.565 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T13:32:38.695 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:32:38.833 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:32:39.055 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T13:32:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:32:39.465 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T13:32:39.486 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T13:32:39.699 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:32:39.803 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:32:40.164 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:32:40.167 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:32:40.290 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:40.399 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:40.502 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T13:32:40.594 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:40.720 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:40.828 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:32:40.846 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:40.972 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T13:32:41.229 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:32:41.299 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:41.400 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:41.481 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:32:41.501 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:41.582 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:41.711 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:32:41.712 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:32:41.891 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T13:32:42.251 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:32:42.361 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:42.520 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:42.646 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:42.772 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:32:43.431 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:32:43.565 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T13:32:43.833 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:32:43.959 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:32:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:32:44.110 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:44.236 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:44.362 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:44.428 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:32:44.488 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:44.614 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:44.696 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T13:32:44.740 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:44.939 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:32:45.065 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:32:45.191 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:32:45.203 INFO:teuthology.orchestra.run.smithi031.stdout:Adding group ceph....done 2023-10-30T13:32:45.501 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:32:45.570 INFO:teuthology.orchestra.run.smithi031.stdout:Adding system user ceph....done 2023-10-30T13:32:45.723 INFO:teuthology.orchestra.run.smithi031.stdout:Setting system user ceph properties....done 2023-10-30T13:32:45.745 INFO:teuthology.orchestra.run.smithi031.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T13:32:45.752 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T13:32:45.893 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T13:32:46.028 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T13:32:46.287 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:32:46.293 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T13:32:46.572 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T13:32:46.831 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:32:46.957 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T13:32:47.067 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:47.250 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:32:47.267 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:32:47.268 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:32:47.351 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:32:47.661 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:32:47.885 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:47.987 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:32:48.011 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:48.255 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:32:48.563 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:32:48.563 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:32:48.629 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T13:32:48.933 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:32:49.059 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:32:49.171 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:49.468 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:32:49.536 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T13:32:49.744 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T13:32:50.012 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:32:50.163 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T13:32:50.198 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:50.444 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:32:50.445 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:32:50.481 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:32:50.791 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:32:51.075 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:51.108 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T13:32:51.234 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T13:32:51.278 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:32:51.279 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T13:32:51.812 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:32:51.909 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:52.088 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:32:52.227 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:32:52.227 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:32:52.473 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T13:32:52.749 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:32:52.936 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.062 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.093 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:32:53.219 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.224 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.345 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.440 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:32:53.441 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:32:53.547 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.648 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.799 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:53.925 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T13:32:54.100 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:54.202 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:54.243 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:54.328 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:54.369 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:54.479 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:54.545 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:54.721 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:32:54.721 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:32:54.948 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:32:55.247 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:55.348 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:55.483 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:55.609 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:32:56.007 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:32:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:32:56.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T13:32:56.557 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2023-10-30T13:32:56.731 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2023-10-30T13:32:56.732 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2023-10-30T13:32:56.887 INFO:teuthology.orchestra.run.smithi031.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T13:32:56.887 INFO:teuthology.orchestra.run.smithi031.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T13:32:56.887 INFO:teuthology.orchestra.run.smithi031.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:32:56.887 INFO:teuthology.orchestra.run.smithi031.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:32:56.888 INFO:teuthology.orchestra.run.smithi031.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T13:32:56.888 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:32:56.888 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:32:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:32:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:32:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T13:32:56.890 INFO:teuthology.orchestra.run.smithi031.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T13:32:56.891 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:32:56.891 INFO:teuthology.orchestra.run.smithi031.stdout:The following additional packages will be installed: 2023-10-30T13:32:56.891 INFO:teuthology.orchestra.run.smithi031.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T13:32:56.892 INFO:teuthology.orchestra.run.smithi031.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T13:32:56.892 INFO:teuthology.orchestra.run.smithi031.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T13:32:56.892 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T13:32:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:32:56.893 INFO:teuthology.orchestra.run.smithi031.stdout:Suggested packages: 2023-10-30T13:32:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T13:32:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T13:32:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: libengine-pkcs11-openssl 2023-10-30T13:32:56.904 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:56.939 INFO:teuthology.orchestra.run.smithi031.stdout:The following NEW packages will be installed: 2023-10-30T13:32:56.940 INFO:teuthology.orchestra.run.smithi031.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T13:32:56.941 INFO:teuthology.orchestra.run.smithi031.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T13:32:56.941 INFO:teuthology.orchestra.run.smithi031.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T13:32:56.941 INFO:teuthology.orchestra.run.smithi031.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T13:32:56.942 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T13:32:56.942 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T13:32:56.942 INFO:teuthology.orchestra.run.smithi031.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:32:56.943 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages will be upgraded: 2023-10-30T13:32:56.943 INFO:teuthology.orchestra.run.smithi031.stdout: libssl3 2023-10-30T13:32:57.000 INFO:teuthology.orchestra.run.smithi031.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T13:32:57.001 INFO:teuthology.orchestra.run.smithi031.stdout:Need to get 54.6 MB of archives. 2023-10-30T13:32:57.001 INFO:teuthology.orchestra.run.smithi031.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T13:32:57.001 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T13:32:57.013 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:57.102 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T13:32:57.114 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:57.157 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T13:32:57.164 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T13:32:57.165 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T13:32:57.167 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T13:32:57.168 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T13:32:57.168 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T13:32:57.169 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T13:32:57.170 INFO:teuthology.orchestra.run.smithi031.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T13:32:57.171 INFO:teuthology.orchestra.run.smithi031.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T13:32:57.176 INFO:teuthology.orchestra.run.smithi031.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T13:32:57.188 INFO:teuthology.orchestra.run.smithi031.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T13:32:57.224 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:57.350 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:57.404 INFO:teuthology.orchestra.run.smithi031.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T13:32:57.406 INFO:teuthology.orchestra.run.smithi031.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T13:32:57.407 INFO:teuthology.orchestra.run.smithi031.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T13:32:57.409 INFO:teuthology.orchestra.run.smithi031.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T13:32:57.410 INFO:teuthology.orchestra.run.smithi031.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T13:32:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T13:32:57.415 INFO:teuthology.orchestra.run.smithi031.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T13:32:57.423 INFO:teuthology.orchestra.run.smithi031.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T13:32:57.423 INFO:teuthology.orchestra.run.smithi031.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T13:32:57.447 INFO:teuthology.orchestra.run.smithi031.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T13:32:57.447 INFO:teuthology.orchestra.run.smithi031.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T13:32:57.457 INFO:teuthology.orchestra.run.smithi031.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T13:32:57.480 INFO:teuthology.orchestra.run.smithi031.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T13:32:57.484 INFO:teuthology.orchestra.run.smithi031.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T13:32:57.484 INFO:teuthology.orchestra.run.smithi031.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T13:32:57.485 INFO:teuthology.orchestra.run.smithi031.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T13:32:57.485 INFO:teuthology.orchestra.run.smithi031.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T13:32:57.497 INFO:teuthology.orchestra.run.smithi031.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T13:32:57.499 INFO:teuthology.orchestra.run.smithi031.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T13:32:57.509 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:57.650 INFO:teuthology.orchestra.run.smithi031.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T13:32:57.651 INFO:teuthology.orchestra.run.smithi031.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T13:32:57.651 INFO:teuthology.orchestra.run.smithi031.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T13:32:57.652 INFO:teuthology.orchestra.run.smithi031.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T13:32:57.652 INFO:teuthology.orchestra.run.smithi031.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T13:32:57.652 INFO:teuthology.orchestra.run.smithi031.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T13:32:57.665 INFO:teuthology.orchestra.run.smithi031.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T13:32:57.665 INFO:teuthology.orchestra.run.smithi031.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T13:32:58.018 INFO:teuthology.orchestra.run.smithi033.stdout:Adding group ceph....done 2023-10-30T13:32:58.064 INFO:teuthology.orchestra.run.smithi031.stdout:Preconfiguring packages ... 2023-10-30T13:32:58.205 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 54.6 MB in 1s (76.3 MB/s) 2023-10-30T13:32:58.346 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package flex. 2023-10-30T13:32:58.420 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-30T13:32:58.424 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T13:32:58.429 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user ceph....done 2023-10-30T13:32:58.470 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T13:32:58.605 INFO:teuthology.orchestra.run.smithi033.stdout:Setting system user ceph properties....done 2023-10-30T13:32:58.626 INFO:teuthology.orchestra.run.smithi033.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T13:32:58.768 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T13:32:59.057 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:32:59.162 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T13:32:59.234 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T13:32:59.624 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:32:59.711 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:32:59.990 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:32:59.991 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:33:00.294 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package bison. 2023-10-30T13:33:00.367 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-30T13:33:00.370 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T13:33:00.422 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:33:00.512 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:00.613 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:00.830 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T13:33:00.845 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T13:33:00.889 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:33:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:33:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:33:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package dump. 2023-10-30T13:33:01.281 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T13:33:01.325 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T13:33:01.578 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package indent. 2023-10-30T13:33:01.593 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:01.606 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:01.628 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:33:01.839 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T13:33:01.854 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T13:33:01.897 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:33:01.936 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T13:33:02.226 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T13:33:02.241 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T13:33:02.284 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:33:02.425 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:02.545 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T13:33:02.561 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T13:33:02.628 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:33:02.672 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:33:02.673 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:33:02.890 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T13:33:02.906 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T13:33:02.940 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:03.167 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T13:33:03.183 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T13:33:03.210 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:03.225 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:03.461 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:33:03.461 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:33:03.478 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T13:33:03.494 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T13:33:03.536 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T13:33:03.856 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T13:33:03.872 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T13:33:03.950 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:33:03.995 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:04.333 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:33:04.334 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:33:05.039 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:05.140 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:05.217 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T13:33:05.233 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:33:05.292 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:05.433 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:05.561 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T13:33:05.577 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:33:05.683 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:33:05.684 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:33:05.699 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:06.201 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:06.368 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:06.406 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T13:33:06.422 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:33:06.456 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:06.469 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:06.603 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:33:06.825 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T13:33:06.853 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T13:33:06.883 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:33:06.990 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:33:07.211 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T13:33:07.227 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:33:07.269 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:33:07.555 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T13:33:07.571 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:07.605 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:07.774 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:33:07.941 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T13:33:07.957 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:08.008 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:08.269 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:33:08.272 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T13:33:08.293 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T13:33:08.309 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:08.328 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-30T13:33:08.351 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:08.519 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-10-30T13:33:08.520 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-10-30T13:33:08.579 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T13:33:08.594 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:33:08.637 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:33:08.690 INFO:teuthology.orchestra.run.smithi033.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T13:33:08.690 INFO:teuthology.orchestra.run.smithi033.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T13:33:08.691 INFO:teuthology.orchestra.run.smithi033.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:33:08.691 INFO:teuthology.orchestra.run.smithi033.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:33:08.691 INFO:teuthology.orchestra.run.smithi033.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T13:33:08.691 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:33:08.691 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:33:08.692 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:33:08.693 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:33:08.693 INFO:teuthology.orchestra.run.smithi033.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T13:33:08.693 INFO:teuthology.orchestra.run.smithi033.stdout: python2.7-minimal 2023-10-30T13:33:08.693 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:33:08.694 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-10-30T13:33:08.694 INFO:teuthology.orchestra.run.smithi033.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T13:33:08.694 INFO:teuthology.orchestra.run.smithi033.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T13:33:08.694 INFO:teuthology.orchestra.run.smithi033.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T13:33:08.694 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T13:33:08.695 INFO:teuthology.orchestra.run.smithi033.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:33:08.695 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-10-30T13:33:08.695 INFO:teuthology.orchestra.run.smithi033.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T13:33:08.695 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T13:33:08.696 INFO:teuthology.orchestra.run.smithi033.stdout: libengine-pkcs11-openssl 2023-10-30T13:33:08.733 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-10-30T13:33:08.734 INFO:teuthology.orchestra.run.smithi033.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T13:33:08.734 INFO:teuthology.orchestra.run.smithi033.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T13:33:08.734 INFO:teuthology.orchestra.run.smithi033.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T13:33:08.735 INFO:teuthology.orchestra.run.smithi033.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T13:33:08.735 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T13:33:08.735 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T13:33:08.736 INFO:teuthology.orchestra.run.smithi033.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:33:08.736 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-10-30T13:33:08.736 INFO:teuthology.orchestra.run.smithi033.stdout: libssl3 2023-10-30T13:33:08.797 INFO:teuthology.orchestra.run.smithi033.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T13:33:08.797 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 54.6 MB of archives. 2023-10-30T13:33:08.797 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T13:33:08.797 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T13:33:08.921 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T13:33:09.004 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T13:33:09.022 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T13:33:09.023 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T13:33:09.026 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T13:33:09.029 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T13:33:09.029 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T13:33:09.031 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T13:33:09.033 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T13:33:09.033 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T13:33:09.042 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T13:33:09.062 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T13:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T13:33:09.164 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T13:33:09.207 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:33:09.584 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package modemmanager. 2023-10-30T13:33:09.594 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T13:33:09.597 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T13:33:09.598 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T13:33:09.601 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:33:09.602 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T13:33:09.602 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T13:33:09.610 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T13:33:09.612 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T13:33:09.625 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T13:33:09.625 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T13:33:09.643 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:09.660 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T13:33:09.660 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T13:33:09.673 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T13:33:09.712 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T13:33:09.719 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T13:33:09.720 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T13:33:09.721 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T13:33:09.721 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T13:33:09.744 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T13:33:09.747 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T13:33:10.070 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T13:33:10.070 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T13:33:10.071 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T13:33:10.071 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T13:33:10.138 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T13:33:10.139 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T13:33:10.155 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T13:33:10.156 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T13:33:10.232 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package network-manager. 2023-10-30T13:33:10.248 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:33:10.299 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:33:10.571 INFO:teuthology.orchestra.run.smithi033.stdout:Preconfiguring packages ... 2023-10-30T13:33:10.818 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 54.6 MB in 1s (38.6 MB/s) 2023-10-30T13:33:10.897 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ppp. 2023-10-30T13:33:10.912 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T13:33:10.965 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package flex. 2023-10-30T13:33:11.054 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137425 files and directories currently installed.) 2023-10-30T13:33:11.059 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T13:33:11.104 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T13:33:11.287 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:33:11.619 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T13:33:11.633 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T13:33:11.677 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T13:33:11.707 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:33:11.902 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T13:33:11.929 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T13:33:11.944 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T13:33:12.013 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T13:33:12.315 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T13:33:12.331 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T13:33:12.351 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:33:12.382 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T13:33:12.661 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T13:33:12.676 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:12.719 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:12.861 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package bison. 2023-10-30T13:33:12.952 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137498 files and directories currently installed.) 2023-10-30T13:33:12.957 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T13:33:13.013 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:33:13.307 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T13:33:13.322 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T13:33:13.400 INFO:teuthology.orchestra.run.smithi031.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T13:33:13.431 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T13:33:13.431 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking postgresql-common (238) ... 2023-10-30T13:33:13.453 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T13:33:13.497 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:33:13.859 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package dump. 2023-10-30T13:33:13.880 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T13:33:13.897 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T13:33:13.914 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:13.925 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T13:33:13.958 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:14.254 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package indent. 2023-10-30T13:33:14.276 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:14.320 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:33:14.606 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T13:33:14.619 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T13:33:14.656 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:33:14.943 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T13:33:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T13:33:14.983 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package postgresql. 2023-10-30T13:33:14.993 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:33:14.998 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T13:33:15.041 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking postgresql (14+238) ... 2023-10-30T13:33:15.268 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T13:33:15.283 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T13:33:15.296 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T13:33:15.309 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T13:33:15.326 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T13:33:15.346 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:33:15.570 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T13:33:15.585 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T13:33:15.628 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T13:33:15.901 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T13:33:15.923 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T13:33:15.939 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T13:33:15.954 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T13:33:15.967 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:15.997 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:33:16.187 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T13:33:16.209 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T13:33:16.253 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:16.284 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package tcl. 2023-10-30T13:33:16.299 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T13:33:16.338 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T13:33:16.506 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T13:33:16.519 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T13:33:16.590 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T13:33:16.855 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T13:33:16.871 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T13:33:16.892 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T13:33:16.913 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:33:16.914 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T13:33:16.959 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:33:17.292 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T13:33:17.307 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T13:33:17.342 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T13:33:17.703 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T13:33:17.718 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T13:33:17.770 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:33:18.045 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T13:33:18.267 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T13:33:18.393 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:33:18.519 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:18.629 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:33:18.653 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T13:33:18.666 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:33:18.695 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:18.779 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:33:18.905 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:18.998 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T13:33:19.011 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:33:19.048 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:19.375 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T13:33:19.397 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:33:19.442 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:19.820 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T13:33:19.842 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T13:33:19.886 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:33:19.933 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T13:33:20.019 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:33:20.190 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T13:33:20.212 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:33:20.256 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:33:20.429 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:33:20.429 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:33:20.651 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T13:33:20.673 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:20.717 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:21.130 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:33:21.179 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T13:33:21.201 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:21.245 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:21.256 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:33:21.382 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T13:33:21.550 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:33:21.565 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T13:33:21.578 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:21.623 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:21.676 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:21.802 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:33:21.911 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:33:21.943 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T13:33:21.965 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:33:22.017 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:33:22.550 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:33:22.604 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T13:33:22.627 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T13:33:22.671 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:33:22.676 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T13:33:22.771 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T13:33:22.960 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:23.024 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package modemmanager. 2023-10-30T13:33:23.046 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:33:23.070 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:33:23.090 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:23.179 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T13:33:23.305 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T13:33:23.431 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:33:23.520 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T13:33:23.582 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:33:23.717 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:33:23.749 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package network-manager. 2023-10-30T13:33:23.772 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:33:23.824 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:33:23.893 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:33:24.011 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:24.153 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T13:33:24.296 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T13:33:24.422 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:33:24.481 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ppp. 2023-10-30T13:33:24.504 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T13:33:24.797 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:33:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:24.985 INFO:teuthology.orchestra.run.smithi031.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T13:33:24.986 INFO:teuthology.orchestra.run.smithi031.stdout:which means they are currently configured by ifupdown: 2023-10-30T13:33:24.986 INFO:teuthology.orchestra.run.smithi031.stdout:- ens1f0 2023-10-30T13:33:24.986 INFO:teuthology.orchestra.run.smithi031.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T13:33:24.986 INFO:teuthology.orchestra.run.smithi031.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T13:33:24.987 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:25.124 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T13:33:25.145 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T13:33:25.167 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T13:33:25.211 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T13:33:25.495 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T13:33:25.518 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T13:33:25.552 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T13:33:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T13:33:25.877 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T13:33:25.899 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T13:33:25.968 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T13:33:26.005 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T13:33:26.281 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T13:33:26.303 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:26.347 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:26.716 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:26.986 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T13:33:27.009 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T13:33:27.074 INFO:teuthology.orchestra.run.smithi033.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T13:33:27.101 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking postgresql-common (238) ... 2023-10-30T13:33:27.101 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:27.294 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T13:33:27.411 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up postgresql-common (238) ... 2023-10-30T13:33:27.560 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T13:33:27.583 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:33:27.630 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:28.362 INFO:teuthology.orchestra.run.smithi031.stdout:Adding user postgres to group ssl-cert 2023-10-30T13:33:28.612 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:28.612 INFO:teuthology.orchestra.run.smithi031.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T13:33:28.887 INFO:teuthology.orchestra.run.smithi031.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T13:33:28.888 INFO:teuthology.orchestra.run.smithi031.stdout:Removing obsolete dictionary files: 2023-10-30T13:33:28.997 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package postgresql. 2023-10-30T13:33:29.011 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T13:33:29.047 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking postgresql (14+238) ... 2023-10-30T13:33:29.266 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T13:33:29.280 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T13:33:29.358 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T13:33:29.602 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T13:33:29.609 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T13:33:29.625 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T13:33:29.669 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T13:33:29.996 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T13:33:30.019 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T13:33:30.088 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:30.088 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:33:30.357 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package tcl. 2023-10-30T13:33:30.371 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T13:33:30.421 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T13:33:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T13:33:30.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T13:33:30.512 INFO:teuthology.orchestra.run.smithi031.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T13:33:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:This user must also own the server process. 2023-10-30T13:33:30.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T13:33:30.513 INFO:teuthology.orchestra.run.smithi031.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T13:33:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:The default text search configuration will be set to "english". 2023-10-30T13:33:30.514 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:Data page checksums are disabled. 2023-10-30T13:33:30.514 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:30.514 INFO:teuthology.orchestra.run.smithi031.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T13:33:30.515 INFO:teuthology.orchestra.run.smithi031.stdout:creating subdirectories ... ok 2023-10-30T13:33:30.515 INFO:teuthology.orchestra.run.smithi031.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T13:33:30.539 INFO:teuthology.orchestra.run.smithi031.stdout:selecting default max_connections ... 100 2023-10-30T13:33:30.576 INFO:teuthology.orchestra.run.smithi031.stdout:selecting default shared_buffers ... 128MB 2023-10-30T13:33:30.589 INFO:teuthology.orchestra.run.smithi031.stdout:selecting default time zone ... Etc/UTC 2023-10-30T13:33:30.590 INFO:teuthology.orchestra.run.smithi031.stdout:creating configuration files ... ok 2023-10-30T13:33:30.768 INFO:teuthology.orchestra.run.smithi031.stdout:running bootstrap script ... ok 2023-10-30T13:33:30.809 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T13:33:30.832 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T13:33:30.877 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:33:31.407 INFO:teuthology.orchestra.run.smithi031.stdout:performing post-bootstrap initialization ... ok 2023-10-30T13:33:31.625 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T13:33:31.647 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T13:33:31.691 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T13:33:31.911 INFO:teuthology.orchestra.run.smithi031.stdout:syncing data to disk ... ok 2023-10-30T13:33:32.070 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T13:33:32.092 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T13:33:32.137 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:33:32.437 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T13:33:32.480 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T13:33:32.642 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T13:33:32.768 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:33:32.894 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:33.045 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:33:33.171 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:33:33.297 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:33.888 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T13:33:33.978 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:33:34.347 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:33:34.347 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:33:34.856 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:33:34.982 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:33:35.108 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T13:33:35.259 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:33:35.385 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:35.511 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:33:35.528 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:35.637 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:33:35.638 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T13:33:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up postgresql (14+238) ... 2023-10-30T13:33:35.889 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:36.014 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:36.292 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:33:36.292 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:33:36.554 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:33:37.052 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T13:33:37.092 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T13:33:37.146 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T13:33:37.212 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:37.338 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:33:37.402 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:33:37.464 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T13:33:37.545 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:33:37.607 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T13:33:37.716 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:33:37.808 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T13:33:37.885 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:33:38.011 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:33:38.187 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:33:38.338 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:38.439 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T13:33:38.608 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T13:33:38.734 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:33:39.252 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:39.253 INFO:teuthology.orchestra.run.smithi033.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T13:33:39.253 INFO:teuthology.orchestra.run.smithi033.stdout:which means they are currently configured by ifupdown: 2023-10-30T13:33:39.253 INFO:teuthology.orchestra.run.smithi033.stdout:- ens1f0 2023-10-30T13:33:39.253 INFO:teuthology.orchestra.run.smithi033.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T13:33:39.254 INFO:teuthology.orchestra.run.smithi033.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T13:33:39.254 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:39.339 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T13:33:39.377 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T13:33:39.439 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T13:33:39.735 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T13:33:40.107 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T13:33:40.126 INFO:teuthology.orchestra.run.smithi031.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:33:40.131 DEBUG:teuthology.parallel:result is None 2023-10-30T13:33:40.670 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:33:40.813 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:33:40.939 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T13:33:41.065 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up postgresql-common (238) ... 2023-10-30T13:33:42.505 INFO:teuthology.orchestra.run.smithi033.stdout:Adding user postgres to group ssl-cert 2023-10-30T13:33:42.684 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:42.684 INFO:teuthology.orchestra.run.smithi033.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T13:33:42.885 INFO:teuthology.orchestra.run.smithi033.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T13:33:42.885 INFO:teuthology.orchestra.run.smithi033.stdout:Removing obsolete dictionary files: 2023-10-30T13:33:43.517 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T13:33:43.960 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:33:44.299 INFO:teuthology.orchestra.run.smithi033.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T13:33:44.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T13:33:44.355 INFO:teuthology.orchestra.run.smithi033.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T13:33:44.355 INFO:teuthology.orchestra.run.smithi033.stdout:This user must also own the server process. 2023-10-30T13:33:44.355 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:44.356 INFO:teuthology.orchestra.run.smithi033.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T13:33:44.356 INFO:teuthology.orchestra.run.smithi033.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T13:33:44.356 INFO:teuthology.orchestra.run.smithi033.stdout:The default text search configuration will be set to "english". 2023-10-30T13:33:44.356 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:44.357 INFO:teuthology.orchestra.run.smithi033.stdout:Data page checksums are disabled. 2023-10-30T13:33:44.357 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:44.357 INFO:teuthology.orchestra.run.smithi033.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T13:33:44.357 INFO:teuthology.orchestra.run.smithi033.stdout:creating subdirectories ... ok 2023-10-30T13:33:44.358 INFO:teuthology.orchestra.run.smithi033.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T13:33:44.371 INFO:teuthology.orchestra.run.smithi033.stdout:selecting default max_connections ... 100 2023-10-30T13:33:44.393 INFO:teuthology.orchestra.run.smithi033.stdout:selecting default shared_buffers ... 128MB 2023-10-30T13:33:44.400 INFO:teuthology.orchestra.run.smithi033.stdout:selecting default time zone ... Etc/UTC 2023-10-30T13:33:44.401 INFO:teuthology.orchestra.run.smithi033.stdout:creating configuration files ... ok 2023-10-30T13:33:44.518 INFO:teuthology.orchestra.run.smithi033.stdout:running bootstrap script ... ok 2023-10-30T13:33:45.164 INFO:teuthology.orchestra.run.smithi033.stdout:performing post-bootstrap initialization ... ok 2023-10-30T13:33:45.920 INFO:teuthology.orchestra.run.smithi033.stdout:syncing data to disk ... ok 2023-10-30T13:33:46.337 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T13:33:49.142 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:49.268 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T13:33:49.419 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up postgresql (14+238) ... 2023-10-30T13:33:49.545 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:33:49.663 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:33:49.907 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:33:49.908 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:33:50.491 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T13:33:50.818 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:33:50.995 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:33:52.473 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T13:33:52.556 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T13:33:53.232 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:33:53.235 DEBUG:teuthology.parallel:result is None 2023-10-30T13:33:53.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:33:53.523 DEBUG:teuthology.orchestra.run.smithi031:> dpkg-query -W -f '${Version}' ceph 2023-10-30T13:33:53.548 INFO:teuthology.orchestra.run.smithi031.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:33:53.548 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:33:53.549 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T13:33:53.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:33:53.780 DEBUG:teuthology.orchestra.run.smithi033:> dpkg-query -W -f '${Version}' ceph 2023-10-30T13:33:53.800 INFO:teuthology.orchestra.run.smithi033.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:33:53.801 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:33:53.801 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T13:33:53.801 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T13:33:53.802 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:53.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T13:33:53.826 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:53.826 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T13:33:53.855 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T13:33:53.855 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:53.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T13:33:53.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T13:33:53.952 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:53.953 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T13:33:53.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T13:33:54.026 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T13:33:54.026 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:54.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T13:33:54.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T13:33:54.112 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:54.113 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T13:33:54.126 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T13:33:54.182 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T13:33:54.182 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:54.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T13:33:54.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T13:33:54.268 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:54.269 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T13:33:54.282 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T13:33:54.338 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T13:33:54.401 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T13:33:54.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2023-10-30T13:33:54.405 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2023-10-30T13:33:54.423 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T13:33:54.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T13:33:54.471 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T13:33:54.489 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T13:33:54.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T13:33:54.539 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T13:33:54.563 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-30T13:33:54.564 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-30T13:33:54.564 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-30T13:33:54.564 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:54.565 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T13:33:54.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:54.707 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:54.707 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T13:33:54.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:54.792 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T13:33:54.792 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T13:33:54.793 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046', 'branch': 'wip-lusov-balance-after-stop', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7441046', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-balance-after-stop'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-balance-after-stop', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 8397, 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'sleep_before_teardown': 0, 'subset': '464/1024', 'suite': 'fs', 'suite_branch': 'wip-lusov-balance-after-stop', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVLU+jSgOeJeW7UG/jbVLLJdv0luZt06WSrUBK/zLu/CZnGWK3S1K+PNeK43Y3u0/bIANzRVtLcV7EpzeK5vxA=', 'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGasMAKSbdhk/Ddh2VZPS5CPuVlwKIWRX7nVm+pw5RdOWvQBxabSh4+BgLboPQkMclKOfNZsf1BkjclIvpQzn0M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_06:34:16', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T13:33:54.794 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T13:33:54.798 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T13:33:54.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T13:33:54.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T13:33:54.895 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:54.896 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T13:33:54.919 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T13:33:54.920 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-10-30T13:33:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T13:33:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:54.973 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 903 Links: 1 2023-10-30T13:33:54.973 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:54.973 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-30 13:27:27.294997622 +0000 2023-10-30T13:33:54.973 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-30 13:27:26.895001195 +0000 2023-10-30T13:33:54.973 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-30 13:27:26.895001195 +0000 2023-10-30T13:33:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-30 13:27:26.895001195 +0000 2023-10-30T13:33:54.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T13:33:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-30T13:33:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-30T13:33:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000384676 s, 1.3 MB/s 2023-10-30T13:33:55.037 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T13:33:55.090 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-10-30T13:33:55.144 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T13:33:55.145 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:55.145 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 920 Links: 1 2023-10-30T13:33:55.145 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:55.145 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-30 13:27:27.562995227 +0000 2023-10-30T13:33:55.145 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-30 13:27:27.558995263 +0000 2023-10-30T13:33:55.146 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-30 13:27:27.558995263 +0000 2023-10-30T13:33:55.146 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-30 13:27:27.558995263 +0000 2023-10-30T13:33:55.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T13:33:55.207 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-30T13:33:55.208 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-30T13:33:55.208 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000395561 s, 1.3 MB/s 2023-10-30T13:33:55.210 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T13:33:55.262 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-10-30T13:33:55.312 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T13:33:55.312 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:55.313 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 935 Links: 1 2023-10-30T13:33:55.313 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:55.313 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-30 13:27:28.626985724 +0000 2023-10-30T13:33:55.313 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-30 13:27:28.218989368 +0000 2023-10-30T13:33:55.313 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-30 13:27:28.218989368 +0000 2023-10-30T13:33:55.313 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-30 13:27:28.218989368 +0000 2023-10-30T13:33:55.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T13:33:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-30T13:33:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-30T13:33:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000381492 s, 1.3 MB/s 2023-10-30T13:33:55.381 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T13:33:55.434 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-10-30T13:33:55.484 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T13:33:55.484 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:55.484 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 952 Links: 1 2023-10-30T13:33:55.485 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:55.485 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-30 13:27:28.906983223 +0000 2023-10-30T13:33:55.485 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-30 13:27:28.902983259 +0000 2023-10-30T13:33:55.485 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-30 13:27:28.902983259 +0000 2023-10-30T13:33:55.485 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-30 13:27:28.902983259 +0000 2023-10-30T13:33:55.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T13:33:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-30T13:33:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-30T13:33:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000391012 s, 1.3 MB/s 2023-10-30T13:33:55.549 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T13:33:55.606 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:33:55.607 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:55.607 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T13:33:55.613 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T13:33:55.613 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2023-10-30T13:33:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T13:33:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:55.661 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 890 Links: 1 2023-10-30T13:33:55.661 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:55.661 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-30 13:26:44.684690039 +0000 2023-10-30T13:33:55.661 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-30 13:26:44.320701661 +0000 2023-10-30T13:33:55.661 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-30 13:26:44.320701661 +0000 2023-10-30T13:33:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-30T13:33:55.662 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T13:33:55.713 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-30T13:33:55.714 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-30T13:33:55.714 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000529749 s, 966 kB/s 2023-10-30T13:33:55.715 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T13:33:55.764 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2023-10-30T13:33:55.811 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T13:33:55.811 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:55.811 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 910 Links: 1 2023-10-30T13:33:55.811 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:55.812 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-30 13:26:45.296670497 +0000 2023-10-30T13:33:55.812 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-30 13:26:44.948681609 +0000 2023-10-30T13:33:55.812 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-30 13:26:44.948681609 +0000 2023-10-30T13:33:55.812 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-30T13:33:55.812 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T13:33:55.869 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-30T13:33:55.870 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-30T13:33:55.870 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000472852 s, 1.1 MB/s 2023-10-30T13:33:55.872 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T13:33:55.920 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2023-10-30T13:33:55.967 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T13:33:55.967 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 926 Links: 1 2023-10-30T13:33:55.968 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:55.968 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-30 13:26:45.860652487 +0000 2023-10-30T13:33:55.968 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-30 13:26:45.516663471 +0000 2023-10-30T13:33:55.968 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-30 13:26:45.516663471 +0000 2023-10-30T13:33:55.968 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-30T13:33:55.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T13:33:56.021 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-30T13:33:56.021 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-30T13:33:56.022 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000550049 s, 931 kB/s 2023-10-30T13:33:56.023 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T13:33:56.074 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2023-10-30T13:33:56.123 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T13:33:56.123 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:33:56.123 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 944 Links: 1 2023-10-30T13:33:56.123 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:33:56.123 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-30 13:26:46.492632794 +0000 2023-10-30T13:33:56.124 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-30 13:26:46.108644672 +0000 2023-10-30T13:33:56.124 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-30 13:26:46.108644672 +0000 2023-10-30T13:33:56.124 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-30T13:33:56.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T13:33:56.177 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-30T13:33:56.178 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-30T13:33:56.178 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00046229 s, 1.1 MB/s 2023-10-30T13:33:56.179 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T13:33:56.228 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:33:56.228 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T13:33:56.229 INFO:tasks.ceph:Generating config... 2023-10-30T13:33:56.231 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-30T13:33:56.231 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-30T13:33:56.232 INFO:tasks.ceph:[client] debug client = 20 2023-10-30T13:33:56.232 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T13:33:56.233 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-30T13:33:56.233 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-30T13:33:56.233 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-30T13:33:56.234 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-30T13:33:56.234 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-30T13:33:56.234 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-30T13:33:56.234 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-30T13:33:56.235 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-30T13:33:56.235 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-30T13:33:56.235 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-30T13:33:56.235 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-30T13:33:56.236 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-30T13:33:56.236 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-10-30T13:33:56.236 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-30T13:33:56.237 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-30T13:33:56.237 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-10-30T13:33:56.237 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-30T13:33:56.237 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-30T13:33:56.238 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-30T13:33:56.238 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-30T13:33:56.238 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-30T13:33:56.238 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-30T13:33:56.239 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-30T13:33:56.239 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-30T13:33:56.239 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-30T13:33:56.239 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T13:33:56.240 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T13:33:56.240 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T13:33:56.240 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T13:33:56.241 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T13:33:56.241 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-30T13:33:56.241 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T13:33:56.241 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T13:33:56.242 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-30T13:33:56.242 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T13:33:56.242 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T13:33:56.242 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-30T13:33:56.243 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-30T13:33:56.243 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T13:33:56.243 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T13:33:56.243 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-30T13:33:56.244 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-30T13:33:56.244 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T13:33:56.244 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T13:33:56.244 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T13:33:56.245 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-30T13:33:56.245 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-30T13:33:56.245 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-30T13:33:56.245 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-30T13:33:56.246 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-30T13:33:56.246 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T13:33:56.246 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-30T13:33:56.247 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T13:33:56.247 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-30T13:33:56.247 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T13:33:56.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T13:33:56.295 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T13:33:56.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T13:33:56.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:33:56.419 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31'), ('mon.b', '172.21.15.33'), ('mon.c', '[v2:172.21.15.33:3301,v1:172.21.15.33:6790]')] 2023-10-30T13:33:56.419 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31,172.21.15.33,[v2:172.21.15.33:3301,v1:172.21.15.33:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-30T13:33:56.420 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:56.420 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T13:33:56.472 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --add b 172.21.15.33 --addv c '[v2:172.21.15.33:3301,v1:172.21.15.33:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:33:56.563 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:33:56.563 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 62b651f4-6cfc-4dcb-91a0-e9335b7700dd 2023-10-30T13:33:56.563 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-10-30T13:33:56.564 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2023-10-30T13:33:56.564 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 62b651f4-6cfc-4dcb-91a0-e9335b7700dd 2023-10-30T13:33:56.564 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2023-10-30T13:33:56.561271+0000 2023-10-30T13:33:56.564 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-10-30T13:33:56.561271+0000 2023-10-30T13:33:56.564 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 16 (pacific) 2023-10-30T13:33:56.565 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2023-10-30T13:33:56.565 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2023-10-30T13:33:56.565 INFO:teuthology.orchestra.run.smithi031.stdout:1: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2023-10-30T13:33:56.565 INFO:teuthology.orchestra.run.smithi031.stdout:2: [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] mon.c 2023-10-30T13:33:56.565 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T13:33:56.567 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T13:33:56.575 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 62b651f4-6cfc-4dcb-91a0-e9335b7700dd... 2023-10-30T13:33:56.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T13:33:56.626 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T13:33:56.658 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2023-10-30T13:33:56.659 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2023-10-30T13:33:56.659 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T13:33:56.659 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2023-10-30T13:33:56.659 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.660 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2023-10-30T13:33:56.660 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.660 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2023-10-30T13:33:56.660 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.660 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2023-10-30T13:33:56.660 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2023-10-30T13:33:56.661 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.661 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2023-10-30T13:33:56.661 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2023-10-30T13:33:56.661 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.661 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T13:33:56.662 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2023-10-30T13:33:56.662 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.662 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T13:33:56.662 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T13:33:56.662 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.663 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2023-10-30T13:33:56.663 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.663 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2023-10-30T13:33:56.663 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2023-10-30T13:33:56.663 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2023-10-30T13:33:56.664 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2023-10-30T13:33:56.664 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2023-10-30T13:33:56.664 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2023-10-30T13:33:56.664 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2023-10-30T13:33:56.664 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T13:33:56.665 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T13:33:56.665 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2023-10-30T13:33:56.665 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.665 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T13:33:56.665 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.665 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2023-10-30T13:33:56.666 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.666 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2023-10-30T13:33:56.666 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.666 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2023-10-30T13:33:56.667 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2023-10-30T13:33:56.667 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2023-10-30T13:33:56.667 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = "172.21.15.31,172.21.15.33,[v2:172.21.15.33:3301,v1:172.21.15.33:6790]" 2023-10-30T13:33:56.667 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on skipped message = False 2023-10-30T13:33:56.667 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = 62b651f4-6cfc-4dcb-91a0-e9335b7700dd 2023-10-30T13:33:56.667 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.668 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2023-10-30T13:33:56.668 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2023-10-30T13:33:56.668 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.668 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2023-10-30T13:33:56.668 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2023-10-30T13:33:56.669 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2023-10-30T13:33:56.669 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.669 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2023-10-30T13:33:56.669 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2023-10-30T13:33:56.669 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.670 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2023-10-30T13:33:56.670 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2023-10-30T13:33:56.670 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2023-10-30T13:33:56.670 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.670 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2023-10-30T13:33:56.671 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2023-10-30T13:33:56.671 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.671 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2023-10-30T13:33:56.671 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.671 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2023-10-30T13:33:56.671 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.672 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2023-10-30T13:33:56.672 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.672 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2023-10-30T13:33:56.672 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2023-10-30T13:33:56.672 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.673 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2023-10-30T13:33:56.673 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2023-10-30T13:33:56.673 INFO:teuthology.orchestra.run.smithi033.stdout: bdev async discard = True 2023-10-30T13:33:56.673 INFO:teuthology.orchestra.run.smithi033.stdout: bdev enable discard = True 2023-10-30T13:33:56.673 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore allocator = bitmap 2023-10-30T13:33:56.673 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2023-10-30T13:33:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2023-10-30T13:33:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 1/20 2023-10-30T13:33:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 1/20 2023-10-30T13:33:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-30T13:33:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2023-10-30T13:33:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 4/10 2023-10-30T13:33:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2023-10-30T13:33:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T13:33:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2023-10-30T13:33:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2023-10-30T13:33:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject delay max = 1 2023-10-30T13:33:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject delay probability = 0.005 2023-10-30T13:33:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject delay type = client mds 2023-10-30T13:33:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 2500 2023-10-30T13:33:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: osd blocked scrub grace period = 3600 2023-10-30T13:33:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2023-10-30T13:33:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: osd max markdown count = 1000 2023-10-30T13:33:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2023-10-30T13:33:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: osd op complaint time = 180 2023-10-30T13:33:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.677 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2023-10-30T13:33:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-30T13:33:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2023-10-30T13:33:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-10-30T13:33:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-10-30T13:33:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:33:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:33:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2023-10-30T13:33:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.679 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2023-10-30T13:33:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-30T13:33:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-10-30T13:33:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2023-10-30T13:33:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-10-30T13:33:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2023-10-30T13:33:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2023-10-30T13:33:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:33:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2023-10-30T13:33:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2023-10-30T13:33:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:33:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T13:33:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T13:33:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T13:33:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T13:33:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T13:33:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T13:33:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2023-10-30T13:33:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2023-10-30T13:33:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2023-10-30T13:33:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: mon op complaint time = 120 2023-10-30T13:33:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-30T13:33:56.684 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2023-10-30T13:33:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2023-10-30T13:33:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2023-10-30T13:33:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2023-10-30T13:33:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T13:33:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T13:33:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: client mount timeout = 600 2023-10-30T13:33:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: client_shutdown_timeout = 120 2023-10-30T13:33:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: debug client = 20 2023-10-30T13:33:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-30T13:33:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: rados mon op timeout = 900 2023-10-30T13:33:56.686 INFO:teuthology.orchestra.run.smithi033.stdout: rados osd op timeout = 900 2023-10-30T13:33:56.686 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2023-10-30T13:33:56.686 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.b] 2023-10-30T13:33:56.686 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.c] 2023-10-30T13:33:56.686 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.a] 2023-10-30T13:33:56.686 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.c] 2023-10-30T13:33:56.687 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.e] 2023-10-30T13:33:56.687 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.b] 2023-10-30T13:33:56.687 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.d] 2023-10-30T13:33:56.687 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.f] 2023-10-30T13:33:56.687 INFO:teuthology.orchestra.run.smithi033.stdout:[mds] 2023-10-30T13:33:56.687 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds = 20 2023-10-30T13:33:56.688 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds balancer = 20 2023-10-30T13:33:56.688 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-30T13:33:56.688 INFO:teuthology.orchestra.run.smithi033.stdout: mds bal fragment size max = 10000 2023-10-30T13:33:56.688 INFO:teuthology.orchestra.run.smithi033.stdout: mds bal merge size = 5 2023-10-30T13:33:56.688 INFO:teuthology.orchestra.run.smithi033.stdout: mds bal split bits = 3 2023-10-30T13:33:56.688 INFO:teuthology.orchestra.run.smithi033.stdout: mds bal split size = 100 2023-10-30T13:33:56.689 INFO:teuthology.orchestra.run.smithi033.stdout: mds debug frag = True 2023-10-30T13:33:56.689 INFO:teuthology.orchestra.run.smithi033.stdout: mds debug scatterstat = True 2023-10-30T13:33:56.689 INFO:teuthology.orchestra.run.smithi033.stdout: mds oft prefetch dirfrags = True 2023-10-30T13:33:56.689 INFO:teuthology.orchestra.run.smithi033.stdout: mds op complaint time = 180 2023-10-30T13:33:56.689 INFO:teuthology.orchestra.run.smithi033.stdout: mds verify scatter = True 2023-10-30T13:33:56.690 INFO:teuthology.orchestra.run.smithi033.stdout: mds_dir_prefetch = False 2023-10-30T13:33:56.690 INFO:teuthology.orchestra.run.smithi033.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T13:33:56.690 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2023-10-30T13:33:56.690 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject delay max = 1 2023-10-30T13:33:56.690 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject delay probability = 0.005 2023-10-30T13:33:56.690 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject delay type = client mds osd 2023-10-30T13:33:56.691 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 2500 2023-10-30T13:33:56.691 INFO:teuthology.orchestra.run.smithi033.stdout: osd op complaint time = 180 2023-10-30T13:33:56.691 INFO:teuthology.orchestra.run.smithi033.stdout: rados mon op timeout = 900 2023-10-30T13:33:56.691 INFO:teuthology.orchestra.run.smithi033.stdout: rados osd op timeout = 900 2023-10-30T13:33:56.693 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-10-30T13:33:56.693 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-10-30T13:33:56.693 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T13:33:56.693 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2023-10-30T13:33:56.693 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.694 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-10-30T13:33:56.694 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.694 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-10-30T13:33:56.694 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.694 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-10-30T13:33:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-10-30T13:33:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-10-30T13:33:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-10-30T13:33:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T13:33:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-10-30T13:33:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T13:33:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T13:33:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-10-30T13:33:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-10-30T13:33:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-10-30T13:33:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-10-30T13:33:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-10-30T13:33:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-10-30T13:33:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-10-30T13:33:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-10-30T13:33:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T13:33:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T13:33:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-10-30T13:33:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T13:33:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-10-30T13:33:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-10-30T13:33:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-10-30T13:33:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-10-30T13:33:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-10-30T13:33:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = "172.21.15.31,172.21.15.33,[v2:172.21.15.33:3301,v1:172.21.15.33:6790]" 2023-10-30T13:33:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on skipped message = False 2023-10-30T13:33:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 62b651f4-6cfc-4dcb-91a0-e9335b7700dd 2023-10-30T13:33:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.701 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-10-30T13:33:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-10-30T13:33:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-10-30T13:33:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2023-10-30T13:33:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-10-30T13:33:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-10-30T13:33:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-10-30T13:33:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-10-30T13:33:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-10-30T13:33:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-10-30T13:33:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-10-30T13:33:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-10-30T13:33:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-10-30T13:33:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-10-30T13:33:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-10-30T13:33:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-10-30T13:33:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-10-30T13:33:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-10-30T13:33:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-10-30T13:33:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: bdev async discard = True 2023-10-30T13:33:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: bdev enable discard = True 2023-10-30T13:33:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = bitmap 2023-10-30T13:33:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-10-30T13:33:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-10-30T13:33:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2023-10-30T13:33:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2023-10-30T13:33:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-30T13:33:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2023-10-30T13:33:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2023-10-30T13:33:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2023-10-30T13:33:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T13:33:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-10-30T13:33:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-10-30T13:33:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay max = 1 2023-10-30T13:33:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay probability = 0.005 2023-10-30T13:33:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay type = client mds 2023-10-30T13:33:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 2500 2023-10-30T13:33:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: osd blocked scrub grace period = 3600 2023-10-30T13:33:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-10-30T13:33:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: osd max markdown count = 1000 2023-10-30T13:33:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-10-30T13:33:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: osd op complaint time = 180 2023-10-30T13:33:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.711 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-10-30T13:33:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-30T13:33:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-10-30T13:33:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-10-30T13:33:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-10-30T13:33:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:33:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:33:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-10-30T13:33:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.712 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-10-30T13:33:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-30T13:33:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-10-30T13:33:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2023-10-30T13:33:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-10-30T13:33:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-10-30T13:33:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-10-30T13:33:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:33:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-10-30T13:33:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-10-30T13:33:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:33:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T13:33:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T13:33:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T13:33:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T13:33:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T13:33:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T13:33:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-10-30T13:33:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-10-30T13:33:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-10-30T13:33:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: mon op complaint time = 120 2023-10-30T13:33:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:33:56.717 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-10-30T13:33:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-10-30T13:33:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-10-30T13:33:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-10-30T13:33:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T13:33:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T13:33:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: client mount timeout = 600 2023-10-30T13:33:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: client_shutdown_timeout = 120 2023-10-30T13:33:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: debug client = 20 2023-10-30T13:33:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-30T13:33:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: rados mon op timeout = 900 2023-10-30T13:33:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: rados osd op timeout = 900 2023-10-30T13:33:56.719 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-10-30T13:33:56.719 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.b] 2023-10-30T13:33:56.719 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.c] 2023-10-30T13:33:56.720 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a] 2023-10-30T13:33:56.720 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c] 2023-10-30T13:33:56.720 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.e] 2023-10-30T13:33:56.720 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b] 2023-10-30T13:33:56.720 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d] 2023-10-30T13:33:56.720 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.f] 2023-10-30T13:33:56.721 INFO:teuthology.orchestra.run.smithi031.stdout:[mds] 2023-10-30T13:33:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds = 20 2023-10-30T13:33:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds balancer = 20 2023-10-30T13:33:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-30T13:33:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: mds bal fragment size max = 10000 2023-10-30T13:33:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: mds bal merge size = 5 2023-10-30T13:33:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: mds bal split bits = 3 2023-10-30T13:33:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: mds bal split size = 100 2023-10-30T13:33:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: mds debug frag = True 2023-10-30T13:33:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: mds debug scatterstat = True 2023-10-30T13:33:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: mds oft prefetch dirfrags = True 2023-10-30T13:33:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: mds op complaint time = 180 2023-10-30T13:33:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: mds verify scatter = True 2023-10-30T13:33:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: mds_dir_prefetch = False 2023-10-30T13:33:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T13:33:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2023-10-30T13:33:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay max = 1 2023-10-30T13:33:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay probability = 0.005 2023-10-30T13:33:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay type = client mds osd 2023-10-30T13:33:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 2500 2023-10-30T13:33:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: osd op complaint time = 180 2023-10-30T13:33:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: rados mon op timeout = 900 2023-10-30T13:33:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: rados osd op timeout = 900 2023-10-30T13:33:56.725 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T13:33:56.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T13:33:56.797 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T13:33:56.798 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:56.798 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T13:33:56.807 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:56.808 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T13:33:56.863 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T13:33:56.863 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:56.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T13:33:56.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:33:56.944 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:33:56.945 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:33:56.996 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T13:33:56.996 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:56.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T13:33:56.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:33:57.022 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:33:57.023 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:33:57.071 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T13:33:57.071 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T13:33:57.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T13:33:57.129 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T13:33:57.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T13:33:57.179 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T13:33:57.189 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T13:33:57.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T13:33:57.247 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T13:33:57.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-30T13:33:57.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T13:33:57.371 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T13:33:57.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-30T13:33:57.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T13:33:57.496 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T13:33:57.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-30T13:33:57.521 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T13:33:57.569 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T13:33:57.572 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-30T13:33:57.586 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T13:33:57.671 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T13:33:57.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-30T13:33:57.730 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T13:33:57.819 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T13:33:57.822 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-30T13:33:57.882 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T13:33:57.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T13:33:57.929 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T13:33:57.946 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T13:33:57.947 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T13:33:57.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T13:33:57.967 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:33:57.968 INFO:tasks.ceph:role: osd.0 2023-10-30T13:33:57.968 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T13:33:57.968 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T13:33:58.039 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:33:58.039 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:33:58.039 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:33:58.039 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:33:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:33:58.040 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:33:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:33:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:33:58.040 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:33:58.041 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:33:58.486 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-30T13:33:58.488 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-30T13:33:58.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T13:33:58.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T13:33:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:33:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:33:58.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T13:33:58.683 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:33:58.683 INFO:tasks.ceph:role: osd.1 2023-10-30T13:33:58.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T13:33:58.684 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T13:33:58.756 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:33:58.756 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:33:58.756 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:33:58.756 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:33:58.756 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:33:58.757 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:33:58.757 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:33:58.757 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:33:58.757 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:33:58.757 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:33:59.238 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-30T13:33:59.240 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-30T13:33:59.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T13:33:59.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T13:33:59.405 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:33:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:33:59.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T13:33:59.473 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:33:59.474 INFO:tasks.ceph:role: osd.2 2023-10-30T13:33:59.474 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T13:33:59.474 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T13:33:59.548 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:33:59.549 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:33:59.549 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:33:59.549 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:33:59.549 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:33:59.549 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:33:59.550 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:33:59.550 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:33:59.550 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:33:59.550 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:34:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-30T13:34:00.030 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-30T13:34:00.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T13:34:00.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T13:34:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:34:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:34:00.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T13:34:00.232 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:34:00.232 INFO:tasks.ceph:role: osd.3 2023-10-30T13:34:00.232 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T13:34:00.233 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T13:34:00.304 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:34:00.304 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:34:00.305 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:34:00.305 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:34:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:34:00.305 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:34:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:34:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:34:00.306 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:34:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:34:00.746 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-30T13:34:00.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-30T13:34:00.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T13:34:00.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T13:34:00.878 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:34:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:34:00.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:01.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:01.005+0000 7f062f150640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T13:34:01.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:01.009+0000 7f062f150640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T13:34:01.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:01.009+0000 7f062f150640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:34:01.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:01.009+0000 7f062f150640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:34:01.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:01.009+0000 7f062f150640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:34:01.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:01.013+0000 7f062f150640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T13:34:04.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T13:34:04.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:04.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:04.129+0000 7f0db58dc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T13:34:04.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:04.133+0000 7f0db58dc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T13:34:04.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:04.133+0000 7f0db58dc640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:34:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:04.133+0000 7f0db58dc640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:34:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:04.133+0000 7f0db58dc640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:34:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:04.133+0000 7f0db58dc640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T13:34:07.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T13:34:07.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:07.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:07.233+0000 7f6e9275f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T13:34:07.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:07.233+0000 7f6e9275f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T13:34:07.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:07.237+0000 7f6e9275f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:34:07.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:07.237+0000 7f6e9275f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:34:07.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:07.237+0000 7f6e9275f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:34:07.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:07.237+0000 7f6e9275f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T13:34:10.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T13:34:10.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:10.377+0000 7ff458cb8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T13:34:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:10.377+0000 7ff458cb8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T13:34:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:10.377+0000 7ff458cb8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:34:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:10.377+0000 7ff458cb8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:34:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:10.377+0000 7ff458cb8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:34:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:10.381+0000 7ff458cb8640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T13:34:13.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T13:34:13.404 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T13:34:13.419 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:34:13.420 INFO:tasks.ceph:role: osd.4 2023-10-30T13:34:13.420 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T13:34:13.420 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T13:34:13.482 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:34:13.483 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:34:13.483 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:34:13.483 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:34:13.483 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:34:13.483 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:34:13.484 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:34:13.484 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:34:13.484 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:34:13.484 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:34:13.934 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-30T13:34:13.937 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-30T13:34:13.937 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T13:34:14.025 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T13:34:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:34:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:34:14.035 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T13:34:14.088 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:34:14.088 INFO:tasks.ceph:role: osd.5 2023-10-30T13:34:14.089 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T13:34:14.089 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T13:34:14.150 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:34:14.151 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:34:14.151 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:34:14.151 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:34:14.151 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:34:14.152 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:34:14.152 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:34:14.152 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:34:14.152 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:34:14.152 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:34:14.613 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-30T13:34:14.614 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-30T13:34:14.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T13:34:14.756 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T13:34:14.768 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:34:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:34:14.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T13:34:14.822 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:34:14.823 INFO:tasks.ceph:role: osd.6 2023-10-30T13:34:14.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T13:34:14.823 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T13:34:14.887 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:34:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:34:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:34:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:34:14.888 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:34:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:34:14.888 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:34:14.888 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:34:14.889 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:34:14.889 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:34:15.363 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-30T13:34:15.364 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-30T13:34:15.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T13:34:15.512 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T13:34:15.523 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:34:15.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:34:15.525 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T13:34:15.578 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:34:15.578 INFO:tasks.ceph:role: osd.7 2023-10-30T13:34:15.578 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T13:34:15.579 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T13:34:15.643 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:34:15.643 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:34:15.643 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:34:15.643 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:34:15.643 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:34:15.644 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:34:15.644 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:34:15.644 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:34:15.644 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:34:15.644 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:34:16.064 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-30T13:34:16.067 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-30T13:34:16.067 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T13:34:16.167 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T13:34:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:34:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:34:16.178 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:16.275+0000 7f2998312640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T13:34:16.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:16.279+0000 7f2998312640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T13:34:16.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:16.279+0000 7f2998312640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:34:16.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:16.279+0000 7f2998312640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:34:16.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:16.279+0000 7f2998312640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:34:16.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:16.283+0000 7f2998312640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T13:34:19.293 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T13:34:19.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:19.394 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:19.391+0000 7f0ceca88640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T13:34:19.395 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:19.391+0000 7f0ceca88640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T13:34:19.396 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:19.391+0000 7f0ceca88640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:34:19.396 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:19.391+0000 7f0ceca88640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:34:19.396 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:19.391+0000 7f0ceca88640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:34:19.396 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:19.391+0000 7f0ceca88640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T13:34:22.414 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T13:34:22.471 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:22.555 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:22.551+0000 7f459b80a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T13:34:22.556 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:22.551+0000 7f459b80a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T13:34:22.556 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:22.551+0000 7f459b80a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:34:22.557 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:22.551+0000 7f459b80a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:34:22.557 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:22.551+0000 7f459b80a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:34:22.557 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:22.551+0000 7f459b80a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T13:34:24.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:24.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:25.574 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T13:34:25.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:25.670 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:25.667+0000 7f82b7275640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T13:34:25.671 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:25.667+0000 7f82b7275640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T13:34:25.671 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:25.667+0000 7f82b7275640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:34:25.671 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:25.667+0000 7f82b7275640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:34:25.671 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:25.667+0000 7f82b7275640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:34:25.672 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:25.667+0000 7f82b7275640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T13:34:28.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T13:34:28.734 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T13:34:28.735 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:28.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T13:34:28.758 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:28.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-30T13:34:28.820 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:28.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-30T13:34:28.884 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:28.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-30T13:34:28.952 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:28.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T13:34:29.016 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:29.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T13:34:29.084 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:29.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T13:34:29.152 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:29.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T13:34:29.216 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.216 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T13:34:29.230 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-30T13:34:29.286 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.286 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-30T13:34:29.342 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-30T13:34:29.397 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.398 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T13:34:29.453 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.454 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T13:34:29.510 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.510 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T13:34:29.565 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:29.566 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T13:34:29.622 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:29.622 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T13:34:29.632 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T13:34:29.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T13:34:29.682 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T13:34:29.693 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.y] 2023-10-30T13:34:29.694 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9leRmcBxAAX12BXcJnQj4tPFZvP+1jqg== 2023-10-30T13:34:29.694 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.a] 2023-10-30T13:34:29.694 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9lmZGlDhAAAh9/y4Gk0DmjQjEkL7q7DQ== 2023-10-30T13:34:29.694 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.c] 2023-10-30T13:34:29.695 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9lhpINFhAA56vFUGWH5Kc3/lMx3vZzFw== 2023-10-30T13:34:29.695 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.e] 2023-10-30T13:34:29.695 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9llAt1HRAAaBkT/5iZRkGxZRC86ECm3g== 2023-10-30T13:34:29.695 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2023-10-30T13:34:29.695 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDJsD9lMDiFABAAhwoTi2CHakMTGc+37LwCuA== 2023-10-30T13:34:29.695 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2023-10-30T13:34:29.696 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDMsD9liCj0BxAAMR7jQZQxCk3MqTAYtYsf5Q== 2023-10-30T13:34:29.696 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2023-10-30T13:34:29.696 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDPsD9lC3EeDhAAVh3tGFeG7xI5kcUxhgCXAw== 2023-10-30T13:34:29.696 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.3] 2023-10-30T13:34:29.696 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDSsD9lxbapFhAAD3t/RYxSJJbIm/xtIwVwGg== 2023-10-30T13:34:29.697 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2023-10-30T13:34:29.697 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9lP2eWChAAg8lrqlB1XeVsdmKSeO6qig== 2023-10-30T13:34:29.697 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.b] 2023-10-30T13:34:29.697 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9lxT/TIRAACmw6Gwn6NGV0ipk8FJFDtg== 2023-10-30T13:34:29.698 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.d] 2023-10-30T13:34:29.698 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9lMp7nJxAA+o24HNsPS6wEAFepvdQCSg== 2023-10-30T13:34:29.698 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.f] 2023-10-30T13:34:29.698 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9llDu7MBAASdgu9OCc+CWUqLU9ESe17Q== 2023-10-30T13:34:29.698 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.4] 2023-10-30T13:34:29.699 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDYsD9ltlyjEBAAi18Zr8X/E82MHHEnVOKxMQ== 2023-10-30T13:34:29.699 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.5] 2023-10-30T13:34:29.699 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDbsD9lP0RvFxAAK7UfvYxUBg+jilpBMeuSmA== 2023-10-30T13:34:29.699 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.6] 2023-10-30T13:34:29.699 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDesD9l83YFIRAAgr7QQGSpwgQKAGg7+/VEyA== 2023-10-30T13:34:29.700 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.7] 2023-10-30T13:34:29.700 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDhsD9lPBvdJxAAneHWnYOXDtojbbbcWpZV/Q== 2023-10-30T13:34:29.700 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2023-10-30T13:34:29.700 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFsD9lwNBENxAArFrDpruGItzrIDbtUjeIVQ== 2023-10-30T13:34:29.701 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.y] 2023-10-30T13:34:29.702 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9leRmcBxAAX12BXcJnQj4tPFZvP+1jqg== 2023-10-30T13:34:29.702 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a] 2023-10-30T13:34:29.702 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9lmZGlDhAAAh9/y4Gk0DmjQjEkL7q7DQ== 2023-10-30T13:34:29.702 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c] 2023-10-30T13:34:29.702 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9lhpINFhAA56vFUGWH5Kc3/lMx3vZzFw== 2023-10-30T13:34:29.703 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.e] 2023-10-30T13:34:29.703 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9llAt1HRAAaBkT/5iZRkGxZRC86ECm3g== 2023-10-30T13:34:29.703 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2023-10-30T13:34:29.703 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDJsD9lMDiFABAAhwoTi2CHakMTGc+37LwCuA== 2023-10-30T13:34:29.703 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2023-10-30T13:34:29.703 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDMsD9liCj0BxAAMR7jQZQxCk3MqTAYtYsf5Q== 2023-10-30T13:34:29.704 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2023-10-30T13:34:29.704 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDPsD9lC3EeDhAAVh3tGFeG7xI5kcUxhgCXAw== 2023-10-30T13:34:29.704 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2023-10-30T13:34:29.704 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDSsD9lxbapFhAAD3t/RYxSJJbIm/xtIwVwGg== 2023-10-30T13:34:29.704 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2023-10-30T13:34:29.705 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9lP2eWChAAg8lrqlB1XeVsdmKSeO6qig== 2023-10-30T13:34:29.705 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b] 2023-10-30T13:34:29.705 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9lxT/TIRAACmw6Gwn6NGV0ipk8FJFDtg== 2023-10-30T13:34:29.705 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d] 2023-10-30T13:34:29.705 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9lMp7nJxAA+o24HNsPS6wEAFepvdQCSg== 2023-10-30T13:34:29.705 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.f] 2023-10-30T13:34:29.706 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9llDu7MBAASdgu9OCc+CWUqLU9ESe17Q== 2023-10-30T13:34:29.706 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.4] 2023-10-30T13:34:29.706 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDYsD9ltlyjEBAAi18Zr8X/E82MHHEnVOKxMQ== 2023-10-30T13:34:29.706 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.5] 2023-10-30T13:34:29.706 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDbsD9lP0RvFxAAK7UfvYxUBg+jilpBMeuSmA== 2023-10-30T13:34:29.707 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.6] 2023-10-30T13:34:29.707 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDesD9l83YFIRAAgr7QQGSpwgQKAGg7+/VEyA== 2023-10-30T13:34:29.707 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.7] 2023-10-30T13:34:29.707 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDhsD9lPBvdJxAAneHWnYOXDtojbbbcWpZV/Q== 2023-10-30T13:34:29.707 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-10-30T13:34:29.707 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDFsD9lwNBENxAArFrDpruGItzrIDbtUjeIVQ== 2023-10-30T13:34:29.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:34:29.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:34:29.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:29.806 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:29.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:29.893 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:29.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:29.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:30.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.083 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:34:30.340 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:34:30.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:30.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:30.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:30.508 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:30.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:30.595 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:34:30.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.765 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:30.933 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:34:31.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T13:34:31.018 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T13:34:31.101 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T13:34:31.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T13:34:31.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:34:31.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T13:34:31.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T13:34:31.271 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:34:31.373 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T13:34:31.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T13:34:31.436 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:34:31.535 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T13:34:31.547 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:31.551 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:34:31.598 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T13:34:31.598 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T13:34:31.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T13:34:31.605 INFO:tasks.ceph.mon.a:Started 2023-10-30T13:34:31.606 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T13:34:31.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T13:34:31.643 INFO:tasks.ceph.mon.b:Started 2023-10-30T13:34:31.643 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T13:34:31.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T13:34:31.646 INFO:tasks.ceph.mon.c:Started 2023-10-30T13:34:31.646 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T13:34:31.646 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T13:34:31.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-30T13:34:31.650 INFO:tasks.ceph.mgr.y:Started 2023-10-30T13:34:31.651 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T13:34:31.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-30T13:34:31.653 INFO:tasks.ceph.mgr.x:Started 2023-10-30T13:34:31.654 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T13:34:31.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T13:34:31.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81e73e640 1 Processor -- start 2023-10-30T13:34:31.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81e73e640 1 -- start start 2023-10-30T13:34:31.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81e73e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:31.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81e73e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc818153f80 con 0x7fc818153ba0 2023-10-30T13:34:31.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81e73e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc818159df0 con 0x7fc8180c4410 2023-10-30T13:34:31.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81e73e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8181597c0 con 0x7fc8180c4810 2023-10-30T13:34:31.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc817fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc818153ba0 0x7fc81815cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:34930/0 (socket says 172.21.15.31:34930) 2023-10-30T13:34:31.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 learned_addr learned my addr 172.21.15.31:0/1936583275 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:34:31.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:31.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:31.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:31.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:31.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:31.792+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:34:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.000+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.000+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.000+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.000+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.000+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:34:32.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.404+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:32.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.404+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:32.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.404+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:32.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.404+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:32.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:32.404+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:34:33.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:33.208+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:33.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:33.208+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:33.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:33.208+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:33.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:33.208+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:33.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:33.208+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:34:34.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 msgr2=0x7fc8180cb4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:34:34.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc8180c4810 0x7fc8180cb4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:34.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> v1:172.21.15.33:6789/0 conn(0x7fc8180c4410 legacy=0x7fc8180cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:34.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> v1:172.21.15.31:6789/0 conn(0x7fc818153ba0 legacy=0x7fc81815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:34.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:34.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:34.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8181597c0 con 0x7fc804006200 2023-10-30T13:34:34.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.792+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc804008d80 con 0x7fc804003720 2023-10-30T13:34:34.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:34.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:34.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:34:34.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:34.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:34.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:34:34.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:34:34.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:34.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:34.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.796+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:34:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:34:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:34:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:34:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:34.996+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:34:35.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:35.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:35.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:34:35.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:35.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:35.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:34:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:34:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:35.396+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:34:36.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:34:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:34:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:34:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:36.200+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:34:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.792+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 msgr2=0x7fc804005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:34:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.792+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc804003720 0x7fc804005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.792+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 msgr2=0x7fc8040086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:34:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.792+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804006200 0x7fc8040086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 reap_dead start 2023-10-30T13:34:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81815d580 con 0x7fc818153ba0 2023-10-30T13:34:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8040088f0 con 0x7fc804003b60 2023-10-30T13:34:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8181597c0 con 0x7fc804007dd0 2023-10-30T13:34:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:37.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:37.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.796+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:34:38.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.996+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:38.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.996+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:38.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.996+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:38.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.996+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:38.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:37.996+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:34:38.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:38.396+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:38.396+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:38.396+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:38.396+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:38.396+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:34:39.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:39.200+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:39.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:39.200+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:34:39.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:39.200+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:39.200+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:39.200+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:34:40.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 msgr2=0x7fc804008230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:34:40.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.33:3300/0 conn(0x7fc804007dd0 0x7fc804008230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:40.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> v1:172.21.15.33:6789/0 conn(0x7fc804003b60 legacy=0x7fc8040066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:40.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 >> v1:172.21.15.31:6789/0 conn(0x7fc818153ba0 legacy=0x7fc80400b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:40.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:40.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:40.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8181597c0 con 0x7fc80400f5b0 2023-10-30T13:34:40.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc815ffb640 1 -- 172.21.15.31:0/1936583275 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc804008d80 con 0x7fc80400eaa0 2023-10-30T13:34:40.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:40.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:40.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 msgr2=0x7fc80400fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:34:40.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 msgr2=0x7fc80400fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:40.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:40.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:34:40.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 msgr2=0x7fc80400ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:34:40.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 msgr2=0x7fc80400ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:40.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:40.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.796+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:34:41.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 msgr2=0x7fc80400fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:34:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 msgr2=0x7fc80400fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:41.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:34:41.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 msgr2=0x7fc80400ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:34:41.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 msgr2=0x7fc80400ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:41.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:41.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:40.996+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:34:41.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:41.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 msgr2=0x7fc80400fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:34:41.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 msgr2=0x7fc80400fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:41.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:41.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:41.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:34:41.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 msgr2=0x7fc80400ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:34:41.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc817fff640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 msgr2=0x7fc80400ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:34:41.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:34:41.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:41.396+0000 7fc817fff640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:34:42.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.200+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> v2:172.21.15.31:3300/0 conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.200+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 msgr2=0x7fc80400ee80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:34:42.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.200+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc80400eaa0 0x7fc80400ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.200+0000 7fc8177fe640 1 -- 172.21.15.31:0/1936583275 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804010940 con 0x7fc80400f5b0 2023-10-30T13:34:42.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.200+0000 7fc8177fe640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc80400f5b0 0x7fc80400fa60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007fd0 tx=0x7fc80c01d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4936a4499fde6b71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:42.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc816ffd640 1 -- 172.21.15.31:0/1936583275 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c0a7040 con 0x7fc80400f5b0 2023-10-30T13:34:42.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc816ffd640 1 -- 172.21.15.31:0/1936583275 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80c0170f0 con 0x7fc80400f5b0 2023-10-30T13:34:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc816ffd640 1 -- 172.21.15.31:0/1936583275 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c005110 con 0x7fc80400f5b0 2023-10-30T13:34:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 -- 172.21.15.31:0/1936583275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc80400f5b0 msgr2=0x7fc80400fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc80400f5b0 0x7fc80400fa60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007fd0 tx=0x7fc80c01d6a0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81ccdb640 1 -- 172.21.15.31:0/1936583275 reap_dead start 2023-10-30T13:34:42.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 -- 172.21.15.31:0/1936583275 shutdown_connections 2023-10-30T13:34:42.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 --2- 172.21.15.31:0/1936583275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc80400f5b0 0x7fc80400fa60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 -- 172.21.15.31:0/1936583275 >> 172.21.15.31:0/1936583275 conn(0x7fc818071790 msgr2=0x7fc818071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 -- 172.21.15.31:0/1936583275 shutdown_connections 2023-10-30T13:34:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 -- 172.21.15.31:0/1936583275 wait complete. 2023-10-30T13:34:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 Processor -- start 2023-10-30T13:34:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.204+0000 7fc81e73e640 1 -- start start 2023-10-30T13:34:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc80400f5b0 0x7fc81815be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc818156cf0 0x7fc8180c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8180c7b30 0x7fc8181f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8181f53c0 con 0x7fc8180c7b30 2023-10-30T13:34:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8181f2f10 con 0x7fc80400f5b0 2023-10-30T13:34:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc818157220 con 0x7fc818156cf0 2023-10-30T13:34:42.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8180c7b30 0x7fc8181f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8180c7b30 0x7fc8181f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54160/0 (socket says 172.21.15.31:54160) 2023-10-30T13:34:42.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 -- 172.21.15.31:0/4048893869 learned_addr learned my addr 172.21.15.31:0/4048893869 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:34:42.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc8177fe640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc818156cf0 0x7fc8180c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc817fff640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc80400f5b0 0x7fc81815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 -- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc818156cf0 msgr2=0x7fc8180c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc818156cf0 0x7fc8180c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 -- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc80400f5b0 msgr2=0x7fc81815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc80400f5b0 0x7fc81815be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 -- 172.21.15.31:0/4048893869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8040088f0 con 0x7fc8180c7b30 2023-10-30T13:34:42.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc8177fe640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc818156cf0 0x7fc8180c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:42.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81ccdb640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8180c7b30 0x7fc8181f4570 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc810018cd0 tx=0x7fc810016770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:42.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc814ff9640 1 -- 172.21.15.31:0/4048893869 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810018ee0 con 0x7fc8180c7b30 2023-10-30T13:34:42.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc814ff9640 1 -- 172.21.15.31:0/4048893869 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8100039c0 con 0x7fc8180c7b30 2023-10-30T13:34:42.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc814ff9640 1 -- 172.21.15.31:0/4048893869 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810003d50 con 0x7fc8180c7b30 2023-10-30T13:34:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8180c6590 con 0x7fc8180c7b30 2023-10-30T13:34:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8180c6b10 con 0x7fc8180c7b30 2023-10-30T13:34:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc814ff9640 1 -- 172.21.15.31:0/4048893869 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc81001f070 con 0x7fc8180c7b30 2023-10-30T13:34:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc814ff9640 1 -- 172.21.15.31:0/4048893869 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc8100199f0 con 0x7fc8180c7b30 2023-10-30T13:34:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.208+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81804e170 con 0x7fc8180c7b30 2023-10-30T13:34:42.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.216+0000 7fc814ff9640 1 -- 172.21.15.31:0/4048893869 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc8100090e0 con 0x7fc8180c7b30 2023-10-30T13:34:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.284+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc8180cabb0 con 0x7fc8180c7b30 2023-10-30T13:34:42.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc814ff9640 1 -- 172.21.15.31:0/4048893869 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc810019bb0 con 0x7fc8180c7b30 2023-10-30T13:34:42.292 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-10-30T13:34:42.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8180c7b30 msgr2=0x7fc8181f4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8180c7b30 0x7fc8181f4570 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc810018cd0 tx=0x7fc810016770 comp rx=0 tx=0).stop 2023-10-30T13:34:42.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 shutdown_connections 2023-10-30T13:34:42.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8180c7b30 0x7fc8181f4570 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc818156cf0 0x7fc8180c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 --2- 172.21.15.31:0/4048893869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc80400f5b0 0x7fc81815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 >> 172.21.15.31:0/4048893869 conn(0x7fc818071790 msgr2=0x7fc8180cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:42.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 shutdown_connections 2023-10-30T13:34:42.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:42.288+0000 7fc81e73e640 1 -- 172.21.15.31:0/4048893869 wait complete. 2023-10-30T13:34:42.314 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T13:34:42.315 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T13:34:42.315 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:42.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T13:34:42.337 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:42.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T13:34:42.390 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:42.384+0000 7f07ee08d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T13:34:42.399 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:42.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T13:34:42.463 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:34:42.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T13:34:42.527 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:42.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T13:34:42.541 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:42.542 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T13:34:42.550 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:42.547+0000 7f113ebea180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T13:34:42.562 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:42.563 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T13:34:42.617 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-30T13:34:42.618 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T13:34:42.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 1902efe2-909b-4286-9f44-68398744f1b7 0 2023-10-30T13:34:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.839+0000 7fa1ad3b7640 1 Processor -- start 2023-10-30T13:34:42.845 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.839+0000 7fa1ad3b7640 1 -- start start 2023-10-30T13:34:42.846 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1ad3b7640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa1a80c6180 0x7fa1a80c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:42.846 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1ad3b7640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a815cf40 con 0x7fa1a80c7020 2023-10-30T13:34:42.847 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1ad3b7640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a8159c60 con 0x7fa1a815db00 2023-10-30T13:34:42.847 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1ad3b7640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a8159f40 con 0x7fa1a80c6180 2023-10-30T13:34:42.847 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a6ffd640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa1a80c6180 0x7fa1a80c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.847 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a6ffd640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa1a80c6180 0x7fa1a80c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43562/0 (socket says 172.21.15.33:43562) 2023-10-30T13:34:42.847 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/2985853764 learned_addr learned my addr 172.21.15.33:0/2985853764 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:42.848 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/2985853764 >> v1:172.21.15.33:6789/0 conn(0x7fa1a815db00 legacy=0x7fa1a8153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.848 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/2985853764 >> v1:172.21.15.31:6789/0 conn(0x7fa1a80c7020 legacy=0x7fa1a815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.848 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/2985853764 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a815a220 con 0x7fa1a80c6180 2023-10-30T13:34:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a6ffd640 1 --2- 172.21.15.33:0/2985853764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c6180 0x7fa1a80c6560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa198009e50 tx=0x7fa19801f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8290b135c883eb6f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a5ffb640 1 -- 172.21.15.33:0/2985853764 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980af400 con 0x7fa1a80c6180 2023-10-30T13:34:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a5ffb640 1 -- 172.21.15.33:0/2985853764 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1980ab050 con 0x7fa1a80c6180 2023-10-30T13:34:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.843+0000 7fa1a5ffb640 1 -- 172.21.15.33:0/2985853764 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980af780 con 0x7fa1a80c6180 2023-10-30T13:34:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/2985853764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c6180 msgr2=0x7fa1a80c6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 --2- 172.21.15.33:0/2985853764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c6180 0x7fa1a80c6560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa198009e50 tx=0x7fa19801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/2985853764 shutdown_connections 2023-10-30T13:34:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 --2- 172.21.15.33:0/2985853764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c6180 0x7fa1a80c6560 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa198009e50 tx=0x7fa19801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/2985853764 >> 172.21.15.33:0/2985853764 conn(0x7fa1a8069e90 msgr2=0x7fa1a806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/2985853764 shutdown_connections 2023-10-30T13:34:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/2985853764 wait complete. 2023-10-30T13:34:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 Processor -- start 2023-10-30T13:34:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- start start 2023-10-30T13:34:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c7020 0x7fa1a81f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1a815db00 0x7fa1a8152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1a814b630 0x7fa1a815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80c5020 con 0x7fa1a814b630 2023-10-30T13:34:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8150a90 con 0x7fa1a80c7020 2023-10-30T13:34:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a81f2ff0 con 0x7fa1a815db00 2023-10-30T13:34:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c7020 0x7fa1a81f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c7020 0x7fa1a81f2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43578/0 (socket says 172.21.15.33:43578) 2023-10-30T13:34:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/209320853 learned_addr learned my addr 172.21.15.33:0/209320853 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a67fc640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1a815db00 0x7fa1a8152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a77fe640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1a814b630 0x7fa1a815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1a815db00 msgr2=0x7fa1a8152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1a815db00 0x7fa1a8152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/209320853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1a814b630 msgr2=0x7fa1a815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1a814b630 0x7fa1a815ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 -- 172.21.15.33:0/209320853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa198009b30 con 0x7fa1a80c7020 2023-10-30T13:34:42.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1a6ffd640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c7020 0x7fa1a81f2080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa1980b2040 tx=0x7fa19801bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:42.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa18f7fe640 1 -- 172.21.15.33:0/209320853 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980c0070 con 0x7fa1a80c7020 2023-10-30T13:34:42.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa18f7fe640 1 -- 172.21.15.33:0/209320853 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa198009510 con 0x7fa1a80c7020 2023-10-30T13:34:42.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a80c4a60 con 0x7fa1a80c7020 2023-10-30T13:34:42.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa18f7fe640 1 -- 172.21.15.33:0/209320853 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980afe80 con 0x7fa1a80c7020 2023-10-30T13:34:42.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a8150440 con 0x7fa1a80c7020 2023-10-30T13:34:42.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa18f7fe640 1 -- 172.21.15.33:0/209320853 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa1980b6040 con 0x7fa1a80c7020 2023-10-30T13:34:42.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.847+0000 7fa18f7fe640 1 -- 172.21.15.33:0/209320853 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa1980c5450 con 0x7fa1a80c7020 2023-10-30T13:34:42.860 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.851+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a8067540 con 0x7fa1a80c7020 2023-10-30T13:34:42.860 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.855+0000 7fa18f7fe640 1 -- 172.21.15.33:0/209320853 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa1980b4070 con 0x7fa1a80c7020 2023-10-30T13:34:42.928 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.923+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1902efe2-909b-4286-9f44-68398744f1b7", "id": 0} v 0) v1 -- 0x7fa1a814d660 con 0x7fa1a80c7020 2023-10-30T13:34:42.938 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa18f7fe640 1 -- 172.21.15.33:0/209320853 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1902efe2-909b-4286-9f44-68398744f1b7", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa1980ba810 con 0x7fa1a80c7020 2023-10-30T13:34:42.938 INFO:teuthology.orchestra.run.smithi033.stdout:0 2023-10-30T13:34:42.939 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c7020 msgr2=0x7fa1a81f2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:42.939 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c7020 0x7fa1a81f2080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa1980b2040 tx=0x7fa19801bf90 comp rx=0 tx=0).stop 2023-10-30T13:34:42.939 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 shutdown_connections 2023-10-30T13:34:42.940 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1a814b630 0x7fa1a815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.940 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1a815db00 0x7fa1a8152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.940 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 --2- 172.21.15.33:0/209320853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1a80c7020 0x7fa1a81f2080 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:42.940 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 >> 172.21.15.33:0/209320853 conn(0x7fa1a8069e90 msgr2=0x7fa1a80cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:42.940 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 shutdown_connections 2023-10-30T13:34:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:42.935+0000 7fa1ad3b7640 1 -- 172.21.15.33:0/209320853 wait complete. 2023-10-30T13:34:42.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new c659a563-8574-4193-ae70-39bed11809f8 1 2023-10-30T13:34:42.960 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:42.956+0000 7f07ee08d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T13:34:43.075 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:43.071+0000 7f113ebea180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T13:34:43.086 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 Processor -- start 2023-10-30T13:34:43.086 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 -- start start 2023-10-30T13:34:43.087 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f59280c8150 0x7f5928157040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.088 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f59280ca210 0x7f59280c4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.088 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c4a90 0x7f59280c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.088 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592806f6b0 con 0x7f59280ca210 2023-10-30T13:34:43.088 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592806e6a0 con 0x7f59280c8150 2023-10-30T13:34:43.089 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592e3d8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5928157580 con 0x7f59280c4a90 2023-10-30T13:34:43.089 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f59280c8150 0x7f5928157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.089 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592c975640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c4a90 0x7f59280c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.089 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f59280c8150 0x7f5928157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43590/0 (socket says 172.21.15.33:43590) 2023-10-30T13:34:43.089 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 -- 172.21.15.33:0/3742895326 learned_addr learned my addr 172.21.15.33:0/3742895326 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.090 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f592c975640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c4a90 0x7f59280c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56286/0 (socket says 172.21.15.33:56286) 2023-10-30T13:34:43.090 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f59277fe640 1 --2- 172.21.15.33:0/3742895326 >> v2:172.21.15.31:3300/0 conn(0x7f59280ca210 0x7f59280c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.090 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 -- 172.21.15.33:0/3742895326 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c4a90 msgr2=0x7f59280c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.090 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 --2- 172.21.15.33:0/3742895326 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c4a90 0x7f59280c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.090 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 -- 172.21.15.33:0/3742895326 >> v2:172.21.15.31:3300/0 conn(0x7f59280ca210 msgr2=0x7f59280c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.091 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 --2- 172.21.15.33:0/3742895326 >> v2:172.21.15.31:3300/0 conn(0x7f59280ca210 0x7f59280c4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.091 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.083+0000 7f5927fff640 1 -- 172.21.15.33:0/3742895326 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59280ca720 con 0x7f59280c8150 2023-10-30T13:34:43.094 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 --2- 172.21.15.33:0/3742895326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c8150 0x7f5928157040 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5928066480 tx=0x7f592001f1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b25252f52c421dec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.094 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5926ffd640 1 -- 172.21.15.33:0/3742895326 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59200af400 con 0x7f59280c8150 2023-10-30T13:34:43.094 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5926ffd640 1 -- 172.21.15.33:0/3742895326 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59200ab050 con 0x7f59280c8150 2023-10-30T13:34:43.094 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5926ffd640 1 -- 172.21.15.33:0/3742895326 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59200af780 con 0x7f59280c8150 2023-10-30T13:34:43.095 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- 172.21.15.33:0/3742895326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c8150 msgr2=0x7f5928157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.095 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 --2- 172.21.15.33:0/3742895326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c8150 0x7f5928157040 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5928066480 tx=0x7f592001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.095 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- 172.21.15.33:0/3742895326 shutdown_connections 2023-10-30T13:34:43.095 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 --2- 172.21.15.33:0/3742895326 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c4a90 0x7f59280c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.095 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 --2- 172.21.15.33:0/3742895326 >> v2:172.21.15.31:3300/0 conn(0x7f59280ca210 0x7f59280c4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.095 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 --2- 172.21.15.33:0/3742895326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c8150 0x7f5928157040 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- 172.21.15.33:0/3742895326 >> 172.21.15.33:0/3742895326 conn(0x7f592806afa0 msgr2=0x7f592806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- 172.21.15.33:0/3742895326 shutdown_connections 2023-10-30T13:34:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- 172.21.15.33:0/3742895326 wait complete. 2023-10-30T13:34:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 Processor -- start 2023-10-30T13:34:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- start start 2023-10-30T13:34:43.097 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c4a90 0x7f592814e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.097 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c8150 0x7f592814f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.097 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59280ca210 0x7f59281eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.097 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592814b1a0 con 0x7f59280ca210 2023-10-30T13:34:43.097 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59281ee820 con 0x7f59280c4a90 2023-10-30T13:34:43.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592e3d8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592814e7d0 con 0x7f59280c8150 2023-10-30T13:34:43.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c4a90 0x7f592814e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f59277fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c8150 0x7f592814f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c4a90 0x7f592814e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43600/0 (socket says 172.21.15.33:43600) 2023-10-30T13:34:43.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 -- 172.21.15.33:0/1746659311 learned_addr learned my addr 172.21.15.33:0/1746659311 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f592c975640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59280ca210 0x7f59281eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 -- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c8150 msgr2=0x7f592814f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c8150 0x7f592814f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 -- 172.21.15.33:0/1746659311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59280ca210 msgr2=0x7f59281eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59280ca210 0x7f59281eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 -- 172.21.15.33:0/1746659311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5920009b30 con 0x7f59280c4a90 2023-10-30T13:34:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f59277fe640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c8150 0x7f592814f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.087+0000 7f5927fff640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c4a90 0x7f592814e290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f592001d870 tx=0x7f592001bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f5924ff9640 1 -- 172.21.15.33:0/1746659311 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59200afdd0 con 0x7f59280c4a90 2023-10-30T13:34:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f5924ff9640 1 -- 172.21.15.33:0/1746659311 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5920009510 con 0x7f59280c4a90 2023-10-30T13:34:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592814e9f0 con 0x7f59280c4a90 2023-10-30T13:34:43.101 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f5924ff9640 1 -- 172.21.15.33:0/1746659311 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59200ba680 con 0x7f59280c4a90 2023-10-30T13:34:43.101 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592814c150 con 0x7f59280c4a90 2023-10-30T13:34:43.101 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f5924ff9640 1 -- 172.21.15.33:0/1746659311 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f59200b6040 con 0x7f59280c4a90 2023-10-30T13:34:43.101 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f5924ff9640 1 -- 172.21.15.33:0/1746659311 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f59200c4b80 con 0x7f59280c4a90 2023-10-30T13:34:43.102 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.091+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5928067540 con 0x7f59280c4a90 2023-10-30T13:34:43.102 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.095+0000 7f5924ff9640 1 -- 172.21.15.33:0/1746659311 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f59200b4070 con 0x7f59280c4a90 2023-10-30T13:34:43.157 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:43.152+0000 7f07ee08d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T13:34:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.163+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c659a563-8574-4193-ae70-39bed11809f8", "id": 1} v 0) v1 -- 0x7f5928073d50 con 0x7f59280c4a90 2023-10-30T13:34:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f5924ff9640 1 -- 172.21.15.33:0/1746659311 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c659a563-8574-4193-ae70-39bed11809f8", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f59200ce440 con 0x7f59280c4a90 2023-10-30T13:34:43.176 INFO:teuthology.orchestra.run.smithi033.stdout:1 2023-10-30T13:34:43.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c4a90 msgr2=0x7f592814e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c4a90 0x7f592814e290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f592001d870 tx=0x7f592001bf60 comp rx=0 tx=0).stop 2023-10-30T13:34:43.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 shutdown_connections 2023-10-30T13:34:43.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59280ca210 0x7f59281eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f59280c8150 0x7f592814f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 --2- 172.21.15.33:0/1746659311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59280c4a90 0x7f592814e290 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 >> 172.21.15.33:0/1746659311 conn(0x7f592806afa0 msgr2=0x7f59280c5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 shutdown_connections 2023-10-30T13:34:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.171+0000 7f592e3d8640 1 -- 172.21.15.33:0/1746659311 wait complete. 2023-10-30T13:34:43.193 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 2c5a63c6-dd27-4bfd-b9b4-af64e974503e 2 2023-10-30T13:34:43.280 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:43.275+0000 7f113ebea180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T13:34:43.308 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.303+0000 7f3470d98640 1 Processor -- start 2023-10-30T13:34:43.309 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.303+0000 7f3470d98640 1 -- start start 2023-10-30T13:34:43.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.303+0000 7f3470d98640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0c90e0 0x7f346c0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.303+0000 7f3470d98640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c151fd0 con 0x7f346c0cb6c0 2023-10-30T13:34:43.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.303+0000 7f3470d98640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c0cbaa0 con 0x7f346c0c5940 2023-10-30T13:34:43.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.303+0000 7f3470d98640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c15bc10 con 0x7f346c0c90e0 2023-10-30T13:34:43.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f3469d9b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0c90e0 0x7f346c0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f346ad9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f346c0c5940 0x7f346c157a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:43166/0 (socket says 172.21.15.33:43166) 2023-10-30T13:34:43.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f346ad9d640 1 -- 172.21.15.33:0/4016782909 learned_addr learned my addr 172.21.15.33:0/4016782909 (peer_addr_for_me v1:172.21.15.33:0/0) 2023-10-30T13:34:43.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f3469d9b640 1 -- 172.21.15.33:0/4016782909 >> v1:172.21.15.33:6789/0 conn(0x7f346c0c5940 legacy=0x7f346c157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f346959a640 1 -- 172.21.15.33:0/4016782909 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276365123 0 0) 0x7f346c0cbaa0 con 0x7f346c0c5940 2023-10-30T13:34:43.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f3469d9b640 1 -- 172.21.15.33:0/4016782909 >> v1:172.21.15.31:6789/0 conn(0x7f346c0cb6c0 legacy=0x7f346c06e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.312 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f3469d9b640 1 -- 172.21.15.33:0/4016782909 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c15af90 con 0x7f346c0c90e0 2023-10-30T13:34:43.313 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f3469d9b640 1 --2- 172.21.15.33:0/4016782909 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0c90e0 0x7f346c0c5400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f345c0097f0 tx=0x7f345c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd1fec5d0130b911 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.313 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f346959a640 1 -- 172.21.15.33:0/4016782909 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0a8970 con 0x7f346c0c90e0 2023-10-30T13:34:43.313 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f346959a640 1 -- 172.21.15.33:0/4016782909 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c0a8ad0 con 0x7f346c0c90e0 2023-10-30T13:34:43.314 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.307+0000 7f346959a640 1 -- 172.21.15.33:0/4016782909 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0a8e60 con 0x7f346c0c90e0 2023-10-30T13:34:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- 172.21.15.33:0/4016782909 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0c90e0 msgr2=0x7f346c0c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 --2- 172.21.15.33:0/4016782909 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0c90e0 0x7f346c0c5400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f345c0097f0 tx=0x7f345c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:34:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- 172.21.15.33:0/4016782909 shutdown_connections 2023-10-30T13:34:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 --2- 172.21.15.33:0/4016782909 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0c90e0 0x7f346c0c5400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- 172.21.15.33:0/4016782909 >> 172.21.15.33:0/4016782909 conn(0x7f346c06afa0 msgr2=0x7f346c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- 172.21.15.33:0/4016782909 shutdown_connections 2023-10-30T13:34:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- 172.21.15.33:0/4016782909 wait complete. 2023-10-30T13:34:43.318 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 Processor -- start 2023-10-30T13:34:43.318 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- start start 2023-10-30T13:34:43.318 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f346c0c5940 0x7f346c0720b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.319 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f346c0c90e0 0x7f346c14b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.319 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0cb6c0 0x7f346c14bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.319 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c14b0e0 con 0x7f346c0c90e0 2023-10-30T13:34:43.319 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c153450 con 0x7f346c0c5940 2023-10-30T13:34:43.320 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c14a7b0 con 0x7f346c0cb6c0 2023-10-30T13:34:43.320 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0cb6c0 0x7f346c14bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.320 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f346c0c5940 0x7f346c0720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.320 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0cb6c0 0x7f346c14bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56304/0 (socket says 172.21.15.33:56304) 2023-10-30T13:34:43.320 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f346c0c5940 0x7f346c0720b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43608/0 (socket says 172.21.15.33:43608) 2023-10-30T13:34:43.321 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 -- 172.21.15.33:0/257755048 learned_addr learned my addr 172.21.15.33:0/257755048 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.321 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3469d9b640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f346c0c90e0 0x7f346c14b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.321 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 -- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f346c0c5940 msgr2=0x7f346c0720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.321 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f346c0c5940 0x7f346c0720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.321 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 -- 172.21.15.33:0/257755048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f346c0c90e0 msgr2=0x7f346c14b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.322 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f346c0c90e0 0x7f346c14b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.322 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 -- 172.21.15.33:0/257755048 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c009420 con 0x7f346c0cb6c0 2023-10-30T13:34:43.322 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346a59c640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f346c0c5940 0x7f346c0720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:43.322 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f346ad9d640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0cb6c0 0x7f346c14bab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3464001570 tx=0x7f346400cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.322 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f345affd640 1 -- 172.21.15.33:0/257755048 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346400a640 con 0x7f346c0cb6c0 2023-10-30T13:34:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f345affd640 1 -- 172.21.15.33:0/257755048 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346400a7c0 con 0x7f346c0cb6c0 2023-10-30T13:34:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c14aa90 con 0x7f346c0cb6c0 2023-10-30T13:34:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f345affd640 1 -- 172.21.15.33:0/257755048 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346400ab70 con 0x7f346c0cb6c0 2023-10-30T13:34:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c14fd80 con 0x7f346c0cb6c0 2023-10-30T13:34:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f345affd640 1 -- 172.21.15.33:0/257755048 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f34640070d0 con 0x7f346c0cb6c0 2023-10-30T13:34:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.311+0000 7f345affd640 1 -- 172.21.15.33:0/257755048 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f3464012820 con 0x7f346c0cb6c0 2023-10-30T13:34:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.315+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346c14b250 con 0x7f346c0cb6c0 2023-10-30T13:34:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.315+0000 7f345affd640 1 -- 172.21.15.33:0/257755048 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f3464018070 con 0x7f346c0cb6c0 2023-10-30T13:34:43.388 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.383+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2c5a63c6-dd27-4bfd-b9b4-af64e974503e", "id": 2} v 0) v1 -- 0x7f346c15b520 con 0x7f346c0cb6c0 2023-10-30T13:34:43.395 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f345affd640 1 -- 172.21.15.33:0/257755048 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c5a63c6-dd27-4bfd-b9b4-af64e974503e", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f34640124d0 con 0x7f346c0cb6c0 2023-10-30T13:34:43.395 INFO:teuthology.orchestra.run.smithi033.stdout:2 2023-10-30T13:34:43.396 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0cb6c0 msgr2=0x7f346c14bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.397 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0cb6c0 0x7f346c14bab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3464001570 tx=0x7f346400cca0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.397 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 shutdown_connections 2023-10-30T13:34:43.397 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f346c0cb6c0 0x7f346c14bab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.397 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f346c0c90e0 0x7f346c14b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.397 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 --2- 172.21.15.33:0/257755048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f346c0c5940 0x7f346c0720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.398 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 >> 172.21.15.33:0/257755048 conn(0x7f346c06afa0 msgr2=0x7f346c06b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.398 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 shutdown_connections 2023-10-30T13:34:43.398 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.391+0000 7f3470d98640 1 -- 172.21.15.33:0/257755048 wait complete. 2023-10-30T13:34:43.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 66544431-3c45-45bb-9505-2469b825ccc0 3 2023-10-30T13:34:43.525 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.519+0000 7facb72b4640 1 Processor -- start 2023-10-30T13:34:43.526 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.519+0000 7facb72b4640 1 -- start start 2023-10-30T13:34:43.527 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb72b4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb01508f0 0x7facb00c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.527 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb72b4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7facb0159bb0 0x7facb015a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.528 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb72b4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb0071710 con 0x7facb0070680 2023-10-30T13:34:43.528 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb72b4640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb00c4df0 con 0x7facb0159bb0 2023-10-30T13:34:43.528 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb72b4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb00c4830 con 0x7facb01508f0 2023-10-30T13:34:43.528 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb5050640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb01508f0 0x7facb00c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.528 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb484f640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7facb0070680 0x7facb006e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:43170/0 (socket says 172.21.15.33:43170) 2023-10-30T13:34:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb484f640 1 -- 172.21.15.33:0/4229043194 learned_addr learned my addr 172.21.15.33:0/4229043194 (peer_addr_for_me v1:172.21.15.33:0/0) 2023-10-30T13:34:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb5851640 1 --2- 172.21.15.33:0/4229043194 >> v2:172.21.15.31:3300/0 conn(0x7facb0159bb0 0x7facb015a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7faca7fff640 1 -- 172.21.15.33:0/4229043194 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065255011 0 0) 0x7facb0071710 con 0x7facb0070680 2023-10-30T13:34:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7faca7fff640 1 -- 172.21.15.33:0/4229043194 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac9c003530 con 0x7facb0070680 2023-10-30T13:34:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb5050640 1 -- 172.21.15.33:0/4229043194 >> v2:172.21.15.31:3300/0 conn(0x7facb0159bb0 msgr2=0x7facb015a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb5050640 1 --2- 172.21.15.33:0/4229043194 >> v2:172.21.15.31:3300/0 conn(0x7facb0159bb0 0x7facb015a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb5050640 1 -- 172.21.15.33:0/4229043194 >> v1:172.21.15.33:6789/0 conn(0x7facb0070680 legacy=0x7facb006e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb5050640 1 -- 172.21.15.33:0/4229043194 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb00c4b10 con 0x7facb01508f0 2023-10-30T13:34:43.532 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7facb5050640 1 --2- 172.21.15.33:0/4229043194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb01508f0 0x7facb00c4240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7facac0097c0 tx=0x7facac01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f836e1645537654b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.532 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7faca7fff640 1 -- 172.21.15.33:0/4229043194 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0a8970 con 0x7facb01508f0 2023-10-30T13:34:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7faca7fff640 1 -- 172.21.15.33:0/4229043194 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac0a8ad0 con 0x7facb01508f0 2023-10-30T13:34:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.523+0000 7faca7fff640 1 -- 172.21.15.33:0/4229043194 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0a8e60 con 0x7facb01508f0 2023-10-30T13:34:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- 172.21.15.33:0/4229043194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb01508f0 msgr2=0x7facb00c4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 --2- 172.21.15.33:0/4229043194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb01508f0 0x7facb00c4240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7facac0097c0 tx=0x7facac01ec70 comp rx=0 tx=0).stop 2023-10-30T13:34:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- 172.21.15.33:0/4229043194 shutdown_connections 2023-10-30T13:34:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 --2- 172.21.15.33:0/4229043194 >> v2:172.21.15.31:3300/0 conn(0x7facb0159bb0 0x7facb015a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 --2- 172.21.15.33:0/4229043194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb01508f0 0x7facb00c4240 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- 172.21.15.33:0/4229043194 >> 172.21.15.33:0/4229043194 conn(0x7facb0069e90 msgr2=0x7facb006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- 172.21.15.33:0/4229043194 shutdown_connections 2023-10-30T13:34:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- 172.21.15.33:0/4229043194 wait complete. 2023-10-30T13:34:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 Processor -- start 2023-10-30T13:34:43.535 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- start start 2023-10-30T13:34:43.535 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb0070680 0x7facb014cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.535 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facb01508f0 0x7facb01f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.535 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facb0159bb0 0x7facb0146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.535 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb00c2b60 con 0x7facb01508f0 2023-10-30T13:34:43.536 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb0152720 con 0x7facb0159bb0 2023-10-30T13:34:43.536 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb01f23a0 con 0x7facb0070680 2023-10-30T13:34:43.536 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb0070680 0x7facb014cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.536 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb0070680 0x7facb014cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56328/0 (socket says 172.21.15.33:56328) 2023-10-30T13:34:43.536 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 -- 172.21.15.33:0/1910927924 learned_addr learned my addr 172.21.15.33:0/1910927924 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.537 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5851640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facb0159bb0 0x7facb0146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.537 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb484f640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facb01508f0 0x7facb01f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.537 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 -- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facb0159bb0 msgr2=0x7facb0146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.537 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facb0159bb0 0x7facb0146010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.537 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 -- 172.21.15.33:0/1910927924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facb01508f0 msgr2=0x7facb01f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.537 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facb01508f0 0x7facb01f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.538 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 -- 172.21.15.33:0/1910927924 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac009420 con 0x7facb0070680 2023-10-30T13:34:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb5050640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb0070680 0x7facb014cef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7facac01c8a0 tx=0x7facac01ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7faca5ffb640 1 -- 172.21.15.33:0/1910927924 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0b9630 con 0x7facb0070680 2023-10-30T13:34:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7faca5ffb640 1 -- 172.21.15.33:0/1910927924 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac0b97b0 con 0x7facb0070680 2023-10-30T13:34:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb00cbdb0 con 0x7facb0070680 2023-10-30T13:34:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7faca5ffb640 1 -- 172.21.15.33:0/1910927924 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0b9b60 con 0x7facb0070680 2023-10-30T13:34:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb00cc370 con 0x7facb0070680 2023-10-30T13:34:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7faca5ffb640 1 -- 172.21.15.33:0/1910927924 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7facac0b7040 con 0x7facb0070680 2023-10-30T13:34:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.527+0000 7faca5ffb640 1 -- 172.21.15.33:0/1910927924 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7facac0c4be0 con 0x7facb0070680 2023-10-30T13:34:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.531+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb0067540 con 0x7facb0070680 2023-10-30T13:34:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.531+0000 7faca5ffb640 1 -- 172.21.15.33:0/1910927924 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7facac0b5070 con 0x7facb0070680 2023-10-30T13:34:43.603 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.599+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "osd new", "uuid": "66544431-3c45-45bb-9505-2469b825ccc0", "id": 3} v 0) v1 -- 0x7facb01f2170 con 0x7facb0070680 2023-10-30T13:34:43.610 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7faca5ffb640 1 -- 172.21.15.33:0/1910927924 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "66544431-3c45-45bb-9505-2469b825ccc0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7facac0d30b0 con 0x7facb0070680 2023-10-30T13:34:43.610 INFO:teuthology.orchestra.run.smithi033.stdout:3 2023-10-30T13:34:43.612 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb0070680 msgr2=0x7facb014cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.612 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb0070680 0x7facb014cef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7facac01c8a0 tx=0x7facac01ba10 comp rx=0 tx=0).stop 2023-10-30T13:34:43.612 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 shutdown_connections 2023-10-30T13:34:43.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facb0159bb0 0x7facb0146010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facb01508f0 0x7facb01f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 --2- 172.21.15.33:0/1910927924 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facb0070680 0x7facb014cef0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 >> 172.21.15.33:0/1910927924 conn(0x7facb0069e90 msgr2=0x7facb00c4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 shutdown_connections 2023-10-30T13:34:43.614 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.607+0000 7facb72b4640 1 -- 172.21.15.33:0/1910927924 wait complete. 2023-10-30T13:34:43.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 8fa6bf36-d2cf-4588-b248-7ee20a44b370 4 2023-10-30T13:34:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad757f640 1 Processor -- start 2023-10-30T13:34:43.739 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad757f640 1 -- start start 2023-10-30T13:34:43.740 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad757f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fcad00cd5a0 0x7fcad00ca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.740 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad757f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fcad01577e0 0x7fcad00ca550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.740 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad757f640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad00c9b80 con 0x7fcad00cd980 2023-10-30T13:34:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad757f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad0157c90 con 0x7fcad00cd5a0 2023-10-30T13:34:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad757f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad0157f70 con 0x7fcad01577e0 2023-10-30T13:34:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad4b1a640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fcad01577e0 0x7fcad00ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad4b1a640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fcad01577e0 0x7fcad00ca550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43622/0 (socket says 172.21.15.33:43622) 2023-10-30T13:34:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad4b1a640 1 -- 172.21.15.33:0/1487682196 learned_addr learned my addr 172.21.15.33:0/1487682196 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad531b640 1 --2- 172.21.15.33:0/1487682196 >> v2:172.21.15.31:3300/0 conn(0x7fcad00cd5a0 0x7fcad00ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad4b1a640 1 -- 172.21.15.33:0/1487682196 >> v2:172.21.15.31:3300/0 conn(0x7fcad00cd5a0 msgr2=0x7fcad00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad4b1a640 1 --2- 172.21.15.33:0/1487682196 >> v2:172.21.15.31:3300/0 conn(0x7fcad00cd5a0 0x7fcad00ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad4b1a640 1 -- 172.21.15.33:0/1487682196 >> v1:172.21.15.31:6789/0 conn(0x7fcad00cd980 legacy=0x7fcad0151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad4b1a640 1 -- 172.21.15.33:0/1487682196 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad0158250 con 0x7fcad01577e0 2023-10-30T13:34:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.735+0000 7fcad531b640 1 --2- 172.21.15.33:0/1487682196 >> v2:172.21.15.31:3300/0 conn(0x7fcad00cd5a0 0x7fcad00ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:43.743 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad4b1a640 1 --2- 172.21.15.33:0/1487682196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad01577e0 0x7fcad00ca550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcad00ccf00 tx=0x7fcac001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb4199dfb2b6e805 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.743 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcac7fff640 1 -- 172.21.15.33:0/1487682196 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00ad680 con 0x7fcad01577e0 2023-10-30T13:34:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcac7fff640 1 -- 172.21.15.33:0/1487682196 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac00ad7e0 con 0x7fcad01577e0 2023-10-30T13:34:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcac7fff640 1 -- 172.21.15.33:0/1487682196 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00adb70 con 0x7fcad01577e0 2023-10-30T13:34:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 -- 172.21.15.33:0/1487682196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad01577e0 msgr2=0x7fcad00ca550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 --2- 172.21.15.33:0/1487682196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad01577e0 0x7fcad00ca550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcad00ccf00 tx=0x7fcac001c470 comp rx=0 tx=0).stop 2023-10-30T13:34:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 -- 172.21.15.33:0/1487682196 shutdown_connections 2023-10-30T13:34:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 --2- 172.21.15.33:0/1487682196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad01577e0 0x7fcad00ca550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 --2- 172.21.15.33:0/1487682196 >> v2:172.21.15.31:3300/0 conn(0x7fcad00cd5a0 0x7fcad00ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 -- 172.21.15.33:0/1487682196 >> 172.21.15.33:0/1487682196 conn(0x7fcad0072010 msgr2=0x7fcad0072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 -- 172.21.15.33:0/1487682196 shutdown_connections 2023-10-30T13:34:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 -- 172.21.15.33:0/1487682196 wait complete. 2023-10-30T13:34:43.748 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 Processor -- start 2023-10-30T13:34:43.748 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.739+0000 7fcad757f640 1 -- start start 2023-10-30T13:34:43.748 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad00cd5a0 0x7fcad00c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.749 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcad00cd980 0x7fcad00c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.749 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad01577e0 0x7fcad00c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.749 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00c21e0 con 0x7fcad01577e0 2023-10-30T13:34:43.749 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad013f910 con 0x7fcad00cd5a0 2023-10-30T13:34:43.749 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad006b040 con 0x7fcad00cd980 2023-10-30T13:34:43.750 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad531b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad00cd5a0 0x7fcad00c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.750 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcad00cd980 0x7fcad00c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.750 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad531b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad00cd5a0 0x7fcad00c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43626/0 (socket says 172.21.15.33:43626) 2023-10-30T13:34:43.750 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad531b640 1 -- 172.21.15.33:0/594400402 learned_addr learned my addr 172.21.15.33:0/594400402 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.751 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcad00cd980 0x7fcad00c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56334/0 (socket says 172.21.15.33:56334) 2023-10-30T13:34:43.751 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad5b1c640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad01577e0 0x7fcad00c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.751 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 -- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad00cd5a0 msgr2=0x7fcad00c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.751 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad00cd5a0 0x7fcad00c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.751 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 -- 172.21.15.33:0/594400402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad01577e0 msgr2=0x7fcad00c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.752 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad01577e0 0x7fcad00c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.752 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 -- 172.21.15.33:0/594400402 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00094d0 con 0x7fcad00cd980 2023-10-30T13:34:43.752 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad531b640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad00cd5a0 0x7fcad00c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:43.752 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad4b1a640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcad00cd980 0x7fcad00c5e90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcac00ac000 tx=0x7fcac001acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.752 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcac5ffb640 1 -- 172.21.15.33:0/594400402 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00bd070 con 0x7fcad00cd980 2023-10-30T13:34:43.752 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcac5ffb640 1 -- 172.21.15.33:0/594400402 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac00add20 con 0x7fcad00cd980 2023-10-30T13:34:43.753 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad006b320 con 0x7fcad00cd980 2023-10-30T13:34:43.753 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcac5ffb640 1 -- 172.21.15.33:0/594400402 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00b8670 con 0x7fcad00cd980 2023-10-30T13:34:43.753 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad006ced0 con 0x7fcad00cd980 2023-10-30T13:34:43.753 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcac5ffb640 1 -- 172.21.15.33:0/594400402 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcac00b4070 con 0x7fcad00cd980 2023-10-30T13:34:43.753 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcac5ffb640 1 -- 172.21.15.33:0/594400402 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fcac00c2bf0 con 0x7fcad00cd980 2023-10-30T13:34:43.753 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.743+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad006ca50 con 0x7fcad00cd980 2023-10-30T13:34:43.754 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.747+0000 7fcac5ffb640 1 -- 172.21.15.33:0/594400402 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fcad006ca50 con 0x7fcad00cd980 2023-10-30T13:34:43.810 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:43.804+0000 7f07ee08d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T13:34:43.817 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.811+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8fa6bf36-d2cf-4588-b248-7ee20a44b370", "id": 4} v 0) v1 -- 0x7fcad00c5c50 con 0x7fcad00cd980 2023-10-30T13:34:43.822 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcac5ffb640 1 -- 172.21.15.33:0/594400402 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8fa6bf36-d2cf-4588-b248-7ee20a44b370", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcac00b2070 con 0x7fcad00cd980 2023-10-30T13:34:43.822 INFO:teuthology.orchestra.run.smithi033.stdout:4 2023-10-30T13:34:43.823 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcad00cd980 msgr2=0x7fcad00c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.824 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcad00cd980 0x7fcad00c5e90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcac00ac000 tx=0x7fcac001acb0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.824 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 shutdown_connections 2023-10-30T13:34:43.824 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad01577e0 0x7fcad00c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.824 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcad00cd980 0x7fcad00c5e90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.824 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 --2- 172.21.15.33:0/594400402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcad00cd5a0 0x7fcad00c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.825 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 >> 172.21.15.33:0/594400402 conn(0x7fcad0072010 msgr2=0x7fcad00cb830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.825 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 shutdown_connections 2023-10-30T13:34:43.825 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.819+0000 7fcad757f640 1 -- 172.21.15.33:0/594400402 wait complete. 2023-10-30T13:34:43.839 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d 5 2023-10-30T13:34:43.876 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:43.872+0000 7f07ee08d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T13:34:43.902 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:43.899+0000 7f113ebea180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T13:34:43.939 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:43.936+0000 7f07ee08d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T13:34:43.960 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f810de02640 1 Processor -- start 2023-10-30T13:34:43.960 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f810de02640 1 -- start start 2023-10-30T13:34:43.962 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f810de02640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f81080cb6e0 0x7f810806e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.962 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f810de02640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080c90e0 0x7f81080c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.962 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f810de02640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108151fd0 con 0x7f81080c5940 2023-10-30T13:34:43.962 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f810de02640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810815bc10 con 0x7f81080cb6e0 2023-10-30T13:34:43.963 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f810de02640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810815ad80 con 0x7f81080c90e0 2023-10-30T13:34:43.963 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f8106ffd640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080c90e0 0x7f81080c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.963 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f8106ffd640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080c90e0 0x7f81080c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56346/0 (socket says 172.21.15.33:56346) 2023-10-30T13:34:43.963 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f8106ffd640 1 -- 172.21.15.33:0/1196200560 learned_addr learned my addr 172.21.15.33:0/1196200560 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.963 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.955+0000 7f81077fe640 1 --2- 172.21.15.33:0/1196200560 >> v2:172.21.15.31:3300/0 conn(0x7f81080cb6e0 0x7f810806e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.964 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f8106ffd640 1 -- 172.21.15.33:0/1196200560 >> v2:172.21.15.31:3300/0 conn(0x7f81080cb6e0 msgr2=0x7f810806e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.964 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f8106ffd640 1 --2- 172.21.15.33:0/1196200560 >> v2:172.21.15.31:3300/0 conn(0x7f81080cb6e0 0x7f810806e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.964 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f8106ffd640 1 -- 172.21.15.33:0/1196200560 >> v1:172.21.15.31:6789/0 conn(0x7f81080c5940 legacy=0x7f81081579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:43.964 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f8106ffd640 1 -- 172.21.15.33:0/1196200560 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81080cbac0 con 0x7f81080c90e0 2023-10-30T13:34:43.967 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f8106ffd640 1 --2- 172.21.15.33:0/1196200560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080c90e0 0x7f81080c5400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009870 tx=0x7f80fc01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cef4209abf4422a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.968 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f81067fc640 1 -- 172.21.15.33:0/1196200560 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc0aa5f0 con 0x7f81080c90e0 2023-10-30T13:34:43.968 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f81067fc640 1 -- 172.21.15.33:0/1196200560 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc0aa750 con 0x7f81080c90e0 2023-10-30T13:34:43.968 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f81067fc640 1 -- 172.21.15.33:0/1196200560 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc0aaae0 con 0x7f81080c90e0 2023-10-30T13:34:43.968 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 -- 172.21.15.33:0/1196200560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080c90e0 msgr2=0x7f81080c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.969 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 --2- 172.21.15.33:0/1196200560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080c90e0 0x7f81080c5400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009870 tx=0x7f80fc01c4d0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.969 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 -- 172.21.15.33:0/1196200560 shutdown_connections 2023-10-30T13:34:43.969 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 --2- 172.21.15.33:0/1196200560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080c90e0 0x7f81080c5400 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.969 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 --2- 172.21.15.33:0/1196200560 >> v2:172.21.15.31:3300/0 conn(0x7f81080cb6e0 0x7f810806e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.969 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 -- 172.21.15.33:0/1196200560 >> 172.21.15.33:0/1196200560 conn(0x7f810806afa0 msgr2=0x7f810806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:43.970 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 -- 172.21.15.33:0/1196200560 shutdown_connections 2023-10-30T13:34:43.970 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 -- 172.21.15.33:0/1196200560 wait complete. 2023-10-30T13:34:43.970 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 Processor -- start 2023-10-30T13:34:43.970 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.959+0000 7f810de02640 1 -- start start 2023-10-30T13:34:43.970 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81080c5940 0x7f81081f5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.970 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81080c90e0 0x7f810814e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.971 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080cb6e0 0x7f810814e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:43.971 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8108154c30 con 0x7f81080c5940 2023-10-30T13:34:43.971 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081550d0 con 0x7f81080c90e0 2023-10-30T13:34:43.971 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8108072ae0 con 0x7f81080cb6e0 2023-10-30T13:34:43.971 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81080c90e0 0x7f810814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.971 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8107fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080cb6e0 0x7f810814e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.972 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81080c90e0 0x7f810814e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43642/0 (socket says 172.21.15.33:43642) 2023-10-30T13:34:43.972 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8106ffd640 1 -- 172.21.15.33:0/605296637 learned_addr learned my addr 172.21.15.33:0/605296637 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:43.972 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f81077fe640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81080c5940 0x7f81081f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:43.972 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8107fff640 1 -- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81080c90e0 msgr2=0x7f810814e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.972 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8107fff640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81080c90e0 0x7f810814e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.972 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8107fff640 1 -- 172.21.15.33:0/605296637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81080c5940 msgr2=0x7f81081f5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:43.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8107fff640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81080c5940 0x7f81081f5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:43.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8107fff640 1 -- 172.21.15.33:0/605296637 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc0094d0 con 0x7f81080cb6e0 2023-10-30T13:34:43.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8106ffd640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81080c90e0 0x7f810814e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:43.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f8107fff640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080cb6e0 0x7f810814e5d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c440 tx=0x7f80f800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:43.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f80effff640 1 -- 172.21.15.33:0/605296637 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8011790 con 0x7f81080cb6e0 2023-10-30T13:34:43.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f80effff640 1 -- 172.21.15.33:0/605296637 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f80118f0 con 0x7f81080cb6e0 2023-10-30T13:34:43.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8108161110 con 0x7f81080cb6e0 2023-10-30T13:34:43.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f80effff640 1 -- 172.21.15.33:0/605296637 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8011ca0 con 0x7f81080cb6e0 2023-10-30T13:34:43.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8108160e30 con 0x7f81080cb6e0 2023-10-30T13:34:43.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f80effff640 1 -- 172.21.15.33:0/605296637 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f80f801c760 con 0x7f81080cb6e0 2023-10-30T13:34:43.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f80effff640 1 -- 172.21.15.33:0/605296637 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f80f8012c70 con 0x7f81080cb6e0 2023-10-30T13:34:43.975 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.963+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8108160c90 con 0x7f81080cb6e0 2023-10-30T13:34:43.975 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:43.967+0000 7f80effff640 1 -- 172.21.15.33:0/605296637 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f8108160c90 con 0x7f81080cb6e0 2023-10-30T13:34:43.975 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:43.971+0000 7f113ebea180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T13:34:44.029 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:44.023+0000 7f113ebea180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T13:34:44.032 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:44.028+0000 7f07ee08d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T13:34:44.037 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.031+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "osd new", "uuid": "716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d", "id": 5} v 0) v1 -- 0x7f8108067540 con 0x7f81080cb6e0 2023-10-30T13:34:44.042 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f80effff640 1 -- 172.21.15.33:0/605296637 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f80f802b0b0 con 0x7f81080cb6e0 2023-10-30T13:34:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:5 2023-10-30T13:34:44.043 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080cb6e0 msgr2=0x7f810814e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.043 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080cb6e0 0x7f810814e5d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c440 tx=0x7f80f800c900 comp rx=0 tx=0).stop 2023-10-30T13:34:44.043 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 shutdown_connections 2023-10-30T13:34:44.043 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81080cb6e0 0x7f810814e5d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.044 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81080c90e0 0x7f810814e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.044 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 --2- 172.21.15.33:0/605296637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81080c5940 0x7f81081f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.044 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 >> 172.21.15.33:0/605296637 conn(0x7f810806afa0 msgr2=0x7f8108161740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:44.044 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 shutdown_connections 2023-10-30T13:34:44.044 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.039+0000 7f810de02640 1 -- 172.21.15.33:0/605296637 wait complete. 2023-10-30T13:34:44.060 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new e95e1fe0-72b5-4773-8689-99d6417fa36f 6 2023-10-30T13:34:44.096 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:44.092+0000 7f07ee08d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T13:34:44.113 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:44.111+0000 7f113ebea180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T13:34:44.169 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:44.163+0000 7f113ebea180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T13:34:44.175 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa126c02640 1 Processor -- start 2023-10-30T13:34:44.176 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa126c02640 1 -- start start 2023-10-30T13:34:44.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa126c02640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa1200c3840 0x7fa1200c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa126c02640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa1200b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa126c02640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1200c5a60 con 0x7fa120150a60 2023-10-30T13:34:44.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa126c02640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1201577c0 con 0x7fa1200c3840 2023-10-30T13:34:44.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa126c02640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1200c5cb0 con 0x7fa1200b0720 2023-10-30T13:34:44.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa12519f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa1200b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa11ffff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa1200c3840 0x7fa1200c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.178 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa12519f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa1200b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56376/0 (socket says 172.21.15.33:56376) 2023-10-30T13:34:44.179 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa12519f640 1 -- 172.21.15.33:0/3868284959 learned_addr learned my addr 172.21.15.33:0/3868284959 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:44.179 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa12519f640 1 -- 172.21.15.33:0/3868284959 >> v2:172.21.15.33:3300/0 conn(0x7fa1200c3840 msgr2=0x7fa1200c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:44.179 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa12519f640 1 --2- 172.21.15.33:0/3868284959 >> v2:172.21.15.33:3300/0 conn(0x7fa1200c3840 0x7fa1200c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.179 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa12519f640 1 -- 172.21.15.33:0/3868284959 >> v1:172.21.15.31:6789/0 conn(0x7fa120150a60 legacy=0x7fa1200cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.179 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa12519f640 1 -- 172.21.15.33:0/3868284959 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1200c5f90 con 0x7fa1200b0720 2023-10-30T13:34:44.180 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.171+0000 7fa11ffff640 1 --2- 172.21.15.33:0/3868284959 >> v2:172.21.15.33:3300/0 conn(0x7fa1200c3840 0x7fa1200c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:44.181 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa12519f640 1 --2- 172.21.15.33:0/3868284959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa1200b0b70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa118009870 tx=0x7fa11801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a484d5af87e46b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:44.181 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa11f7fe640 1 -- 172.21.15.33:0/3868284959 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1180a8a30 con 0x7fa1200b0720 2023-10-30T13:34:44.181 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa11f7fe640 1 -- 172.21.15.33:0/3868284959 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1180a8b90 con 0x7fa1200b0720 2023-10-30T13:34:44.181 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa11f7fe640 1 -- 172.21.15.33:0/3868284959 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1180b0440 con 0x7fa1200b0720 2023-10-30T13:34:44.182 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 -- 172.21.15.33:0/3868284959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 msgr2=0x7fa1200b0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.182 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 --2- 172.21.15.33:0/3868284959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa1200b0b70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa118009870 tx=0x7fa11801ed30 comp rx=0 tx=0).stop 2023-10-30T13:34:44.182 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 -- 172.21.15.33:0/3868284959 shutdown_connections 2023-10-30T13:34:44.182 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 --2- 172.21.15.33:0/3868284959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa1200b0b70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.182 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 --2- 172.21.15.33:0/3868284959 >> v2:172.21.15.33:3300/0 conn(0x7fa1200c3840 0x7fa1200c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.183 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 -- 172.21.15.33:0/3868284959 >> 172.21.15.33:0/3868284959 conn(0x7fa120069e90 msgr2=0x7fa12006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:44.183 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 -- 172.21.15.33:0/3868284959 shutdown_connections 2023-10-30T13:34:44.183 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.175+0000 7fa126c02640 1 -- 172.21.15.33:0/3868284959 wait complete. 2023-10-30T13:34:44.185 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 Processor -- start 2023-10-30T13:34:44.185 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 -- start start 2023-10-30T13:34:44.186 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa120151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.186 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1200c3840 0x7fa120153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.186 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa120150a60 0x7fa1200bef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.186 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa120149680 con 0x7fa120150a60 2023-10-30T13:34:44.186 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1201f1170 con 0x7fa1200c3840 2023-10-30T13:34:44.187 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12014ee00 con 0x7fa1200b0720 2023-10-30T13:34:44.187 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa12499e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa120151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.187 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1200c3840 0x7fa120153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.187 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa12499e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa120151400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56390/0 (socket says 172.21.15.33:56390) 2023-10-30T13:34:44.187 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa12499e640 1 -- 172.21.15.33:0/2144063428 learned_addr learned my addr 172.21.15.33:0/2144063428 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:44.188 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1200c3840 0x7fa120153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43658/0 (socket says 172.21.15.33:43658) 2023-10-30T13:34:44.188 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa12519f640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa120150a60 0x7fa1200bef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.188 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 -- 172.21.15.33:0/2144063428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 msgr2=0x7fa120151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.188 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa120151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.189 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 -- 172.21.15.33:0/2144063428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa120150a60 msgr2=0x7fa1200bef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.189 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa120150a60 0x7fa1200bef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.189 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 -- 172.21.15.33:0/2144063428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1080093b0 con 0x7fa1200c3840 2023-10-30T13:34:44.189 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa12519f640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa120150a60 0x7fa1200bef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:44.189 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11ffff640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1200c3840 0x7fa120153080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa10800e930 tx=0x7fa10800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:44.189 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11d7fa640 1 -- 172.21.15.33:0/2144063428 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1080059a0 con 0x7fa1200c3840 2023-10-30T13:34:44.190 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11d7fa640 1 -- 172.21.15.33:0/2144063428 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa108005b00 con 0x7fa1200c3840 2023-10-30T13:34:44.190 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1180094d0 con 0x7fa1200c3840 2023-10-30T13:34:44.190 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11d7fa640 1 -- 172.21.15.33:0/2144063428 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10800c9a0 con 0x7fa1200c3840 2023-10-30T13:34:44.190 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1201f03b0 con 0x7fa1200c3840 2023-10-30T13:34:44.190 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11d7fa640 1 -- 172.21.15.33:0/2144063428 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa108006030 con 0x7fa1200c3840 2023-10-30T13:34:44.190 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa11d7fa640 1 -- 172.21.15.33:0/2144063428 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fa108013a60 con 0x7fa1200c3840 2023-10-30T13:34:44.191 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.179+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1200cb770 con 0x7fa1200c3840 2023-10-30T13:34:44.191 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:44.184+0000 7f07ee08d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T13:34:44.191 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.183+0000 7fa11d7fa640 1 -- 172.21.15.33:0/2144063428 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa108020060 con 0x7fa1200c3840 2023-10-30T13:34:44.250 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:44.244+0000 7f07ee08d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T13:34:44.302 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:44.299+0000 7f113ebea180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T13:34:44.330 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.327+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e95e1fe0-72b5-4773-8689-99d6417fa36f", "id": 6} v 0) v1 -- 0x7fa1200c6800 con 0x7fa1200c3840 2023-10-30T13:34:44.338 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa11d7fa640 1 -- 172.21.15.33:0/2144063428 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e95e1fe0-72b5-4773-8689-99d6417fa36f", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa108013cc0 con 0x7fa1200c3840 2023-10-30T13:34:44.338 INFO:teuthology.orchestra.run.smithi033.stdout:6 2023-10-30T13:34:44.339 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1200c3840 msgr2=0x7fa120153080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.340 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1200c3840 0x7fa120153080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa10800e930 tx=0x7fa10800edf0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.340 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 shutdown_connections 2023-10-30T13:34:44.340 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa120150a60 0x7fa1200bef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.340 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1200c3840 0x7fa120153080 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.340 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 --2- 172.21.15.33:0/2144063428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1200b0720 0x7fa120151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.341 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 >> 172.21.15.33:0/2144063428 conn(0x7fa120069e90 msgr2=0x7fa12006e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:44.341 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 shutdown_connections 2023-10-30T13:34:44.341 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.335+0000 7fa126c02640 1 -- 172.21.15.33:0/2144063428 wait complete. 2023-10-30T13:34:44.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 0727ffd4-34a3-48d6-8431-37963964af96 7 2023-10-30T13:34:44.369 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:44.363+0000 7f113ebea180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T13:34:44.414 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:44.408+0000 7f07ee08d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T13:34:44.473 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.467+0000 7fe56e1b3640 1 Processor -- start 2023-10-30T13:34:44.473 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.467+0000 7fe56e1b3640 1 -- start start 2023-10-30T13:34:44.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe56e1b3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe5680c7940 0x7fe56815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe56e1b3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe5680c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe56e1b3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680c3210 con 0x7fe568151a00 2023-10-30T13:34:44.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe56e1b3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5680c3e60 con 0x7fe5680c7940 2023-10-30T13:34:44.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe56e1b3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56815bcc0 con 0x7fe5680c2850 2023-10-30T13:34:44.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe567fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe5680c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe567fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe5680c2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56398/0 (socket says 172.21.15.33:56398) 2023-10-30T13:34:44.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe567fff640 1 -- 172.21.15.33:0/3072550120 learned_addr learned my addr 172.21.15.33:0/3072550120 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:44.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe566ffd640 1 --2- 172.21.15.33:0/3072550120 >> v2:172.21.15.31:3300/0 conn(0x7fe5680c7940 0x7fe56815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe567fff640 1 -- 172.21.15.33:0/3072550120 >> v2:172.21.15.31:3300/0 conn(0x7fe5680c7940 msgr2=0x7fe56815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:44.477 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe567fff640 1 --2- 172.21.15.33:0/3072550120 >> v2:172.21.15.31:3300/0 conn(0x7fe5680c7940 0x7fe56815ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.477 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe567fff640 1 -- 172.21.15.33:0/3072550120 >> v1:172.21.15.31:6789/0 conn(0x7fe568151a00 legacy=0x7fe5680ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:44.477 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.471+0000 7fe567fff640 1 -- 172.21.15.33:0/3072550120 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680c23c0 con 0x7fe5680c2850 2023-10-30T13:34:44.478 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe567fff640 1 --2- 172.21.15.33:0/3072550120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe5680c2cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe558009e50 tx=0x7fe55801c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1cea1efc29bb504 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:44.478 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5667fc640 1 -- 172.21.15.33:0/3072550120 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5580a5850 con 0x7fe5680c2850 2023-10-30T13:34:44.478 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5667fc640 1 -- 172.21.15.33:0/3072550120 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5580a59b0 con 0x7fe5680c2850 2023-10-30T13:34:44.478 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5667fc640 1 -- 172.21.15.33:0/3072550120 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5580a5d40 con 0x7fe5680c2850 2023-10-30T13:34:44.480 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- 172.21.15.33:0/3072550120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 msgr2=0x7fe5680c2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.480 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 --2- 172.21.15.33:0/3072550120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe5680c2cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe558009e50 tx=0x7fe55801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.480 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- 172.21.15.33:0/3072550120 shutdown_connections 2023-10-30T13:34:44.480 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 --2- 172.21.15.33:0/3072550120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe5680c2cd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.480 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 --2- 172.21.15.33:0/3072550120 >> v2:172.21.15.31:3300/0 conn(0x7fe5680c7940 0x7fe56815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.480 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- 172.21.15.33:0/3072550120 >> 172.21.15.33:0/3072550120 conn(0x7fe568072010 msgr2=0x7fe568072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:44.481 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- 172.21.15.33:0/3072550120 shutdown_connections 2023-10-30T13:34:44.481 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- 172.21.15.33:0/3072550120 wait complete. 2023-10-30T13:34:44.481 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 Processor -- start 2023-10-30T13:34:44.481 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- start start 2023-10-30T13:34:44.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe568146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5680c7940 0x7fe568146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe568151a00 0x7fe56814ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.484 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe568148eb0 con 0x7fe5680c7940 2023-10-30T13:34:44.484 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe568149410 con 0x7fe568151a00 2023-10-30T13:34:44.484 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56815aca0 con 0x7fe5680c2850 2023-10-30T13:34:44.484 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe568146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.484 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe567fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe568151a00 0x7fe56814ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe568146860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.33:56402/0 (socket says 172.21.15.33:56402) 2023-10-30T13:34:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 -- 172.21.15.33:0/105029511 learned_addr learned my addr 172.21.15.33:0/105029511 (peer_addr_for_me v2:172.21.15.33:0/0) 2023-10-30T13:34:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe566ffd640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5680c7940 0x7fe568146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 -- 172.21.15.33:0/105029511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe568151a00 msgr2=0x7fe56814ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe568151a00 0x7fe56814ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 -- 172.21.15.33:0/105029511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5680c7940 msgr2=0x7fe568146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5680c7940 0x7fe568146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 -- 172.21.15.33:0/105029511 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe558009b30 con 0x7fe5680c2850 2023-10-30T13:34:44.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe5677fe640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe568146860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe56000b410 tx=0x7fe56000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:44.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe55ffff640 1 -- 172.21.15.33:0/105029511 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560004020 con 0x7fe5680c2850 2023-10-30T13:34:44.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe55ffff640 1 -- 172.21.15.33:0/105029511 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe560004180 con 0x7fe5680c2850 2023-10-30T13:34:44.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56806c250 con 0x7fe5680c2850 2023-10-30T13:34:44.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe55ffff640 1 -- 172.21.15.33:0/105029511 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560011630 con 0x7fe5680c2850 2023-10-30T13:34:44.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.475+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56806c4f0 con 0x7fe5680c2850 2023-10-30T13:34:44.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.479+0000 7fe55ffff640 1 -- 172.21.15.33:0/105029511 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe560011a90 con 0x7fe5680c2850 2023-10-30T13:34:44.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.479+0000 7fe55ffff640 1 -- 172.21.15.33:0/105029511 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fe560003d80 con 0x7fe5680c2850 2023-10-30T13:34:44.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.479+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe568067540 con 0x7fe5680c2850 2023-10-30T13:34:44.488 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:44.480+0000 7f07ee08d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T13:34:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.479+0000 7fe55ffff640 1 -- 172.21.15.33:0/105029511 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fe560015030 con 0x7fe5680c2850 2023-10-30T13:34:44.521 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:44.515+0000 7f113ebea180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T13:34:44.553 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.547+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0727ffd4-34a3-48d6-8431-37963964af96", "id": 7} v 0) v1 -- 0x7fe5681551d0 con 0x7fe5680c2850 2023-10-30T13:34:44.559 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe55ffff640 1 -- 172.21.15.33:0/105029511 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0727ffd4-34a3-48d6-8431-37963964af96", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe560011d40 con 0x7fe5680c2850 2023-10-30T13:34:44.559 INFO:teuthology.orchestra.run.smithi033.stdout:7 2023-10-30T13:34:44.560 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 msgr2=0x7fe568146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.560 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe568146860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe56000b410 tx=0x7fe56000b8d0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.561 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 shutdown_connections 2023-10-30T13:34:44.561 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe568151a00 0x7fe56814ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.561 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5680c7940 0x7fe568146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.561 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 --2- 172.21.15.33:0/105029511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5680c2850 0x7fe568146860 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.561 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 >> 172.21.15.33:0/105029511 conn(0x7fe568072010 msgr2=0x7fe568149670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:44.562 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 shutdown_connections 2023-10-30T13:34:44.562 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-30T13:34:44.555+0000 7fe56e1b3640 1 -- 172.21.15.33:0/105029511 wait complete. 2023-10-30T13:34:44.575 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T13:34:44.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-30T13:34:44.579 INFO:tasks.ceph.osd.0:Started 2023-10-30T13:34:44.580 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T13:34:44.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-30T13:34:44.583 INFO:tasks.ceph.osd.1:Started 2023-10-30T13:34:44.584 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T13:34:44.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-30T13:34:44.585 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:44.579+0000 7f113ebea180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T13:34:44.588 INFO:tasks.ceph.osd.2:Started 2023-10-30T13:34:44.589 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T13:34:44.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-30T13:34:44.594 INFO:tasks.ceph.osd.3:Started 2023-10-30T13:34:44.594 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T13:34:44.595 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-30T13:34:44.597 INFO:tasks.ceph.osd.4:Started 2023-10-30T13:34:44.598 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T13:34:44.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-30T13:34:44.600 INFO:tasks.ceph.osd.5:Started 2023-10-30T13:34:44.601 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T13:34:44.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-30T13:34:44.605 INFO:tasks.ceph.osd.6:Started 2023-10-30T13:34:44.606 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T13:34:44.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-30T13:34:44.608 INFO:tasks.ceph.osd.7:Started 2023-10-30T13:34:44.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:34:44.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.788+0000 7f9289a54640 1 Processor -- start 2023-10-30T13:34:44.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.788+0000 7f9289a54640 1 -- start start 2023-10-30T13:34:44.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.788+0000 7f9289a54640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f92840c4a10 0x7f928415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.788+0000 7f9289a54640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c42a0 0x7f92840ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.788+0000 7f9289a54640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9284159f20 con 0x7f9284153c80 2023-10-30T13:34:44.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.788+0000 7f9289a54640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928415dec0 con 0x7f92840c4a10 2023-10-30T13:34:44.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.788+0000 7f9289a54640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92841599a0 con 0x7f92840c42a0 2023-10-30T13:34:44.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92837fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c42a0 0x7f92840ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f927bfff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f92840c4a10 0x7f928415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92837fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c42a0 0x7f92840ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:56378/0 (socket says 172.21.15.31:56378) 2023-10-30T13:34:44.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92837fe640 1 -- 172.21.15.31:0/4092525418 learned_addr learned my addr 172.21.15.31:0/4092525418 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:34:44.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92837fe640 1 -- 172.21.15.31:0/4092525418 >> v2:172.21.15.33:3300/0 conn(0x7f92840c4a10 msgr2=0x7f928415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:44.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92827fc640 1 -- 172.21.15.31:0/4092525418 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396773578 0 0) 0x7f9284159f20 con 0x7f9284153c80 2023-10-30T13:34:44.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92837fe640 1 --2- 172.21.15.31:0/4092525418 >> v2:172.21.15.33:3300/0 conn(0x7f92840c4a10 0x7f928415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92837fe640 1 -- 172.21.15.31:0/4092525418 >> v1:172.21.15.33:6789/0 conn(0x7f9284153c80 legacy=0x7f928415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f92837fe640 1 -- 172.21.15.31:0/4092525418 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928415d620 con 0x7f92840c42a0 2023-10-30T13:34:44.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.792+0000 7f927bfff640 1 --2- 172.21.15.31:0/4092525418 >> v2:172.21.15.33:3300/0 conn(0x7f92840c4a10 0x7f928415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:44.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f92837fe640 1 --2- 172.21.15.31:0/4092525418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c42a0 0x7f92840ca1e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f927c009e50 tx=0x7f927c01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12ae7288ab0949c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:44.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f92827fc640 1 -- 172.21.15.31:0/4092525418 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c0af400 con 0x7f92840c42a0 2023-10-30T13:34:44.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f92827fc640 1 -- 172.21.15.31:0/4092525418 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c0ab050 con 0x7f92840c42a0 2023-10-30T13:34:44.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f92827fc640 1 -- 172.21.15.31:0/4092525418 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c0af780 con 0x7f92840c42a0 2023-10-30T13:34:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- 172.21.15.31:0/4092525418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c42a0 msgr2=0x7f92840ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 --2- 172.21.15.31:0/4092525418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c42a0 0x7f92840ca1e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f927c009e50 tx=0x7f927c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- 172.21.15.31:0/4092525418 shutdown_connections 2023-10-30T13:34:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 --2- 172.21.15.31:0/4092525418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c42a0 0x7f92840ca1e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 --2- 172.21.15.31:0/4092525418 >> v2:172.21.15.33:3300/0 conn(0x7f92840c4a10 0x7f928415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- 172.21.15.31:0/4092525418 >> 172.21.15.31:0/4092525418 conn(0x7f92840715c0 msgr2=0x7f92840719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:44.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- 172.21.15.31:0/4092525418 shutdown_connections 2023-10-30T13:34:44.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- 172.21.15.31:0/4092525418 wait complete. 2023-10-30T13:34:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 Processor -- start 2023-10-30T13:34:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- start start 2023-10-30T13:34:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92840c42a0 0x7f92841ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c4a10 0x7f92840c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9284153c80 0x7f92840b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92840b2500 con 0x7f92840c42a0 2023-10-30T13:34:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92841f1f20 con 0x7f9284153c80 2023-10-30T13:34:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.796+0000 7f9289a54640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92841f2aa0 con 0x7f92840c4a10 2023-10-30T13:34:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92840c42a0 0x7f92841ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92840c42a0 0x7f92841ef3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51886/0 (socket says 172.21.15.31:51886) 2023-10-30T13:34:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 -- 172.21.15.31:0/1557297609 learned_addr learned my addr 172.21.15.31:0/1557297609 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:34:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f927bfff640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c4a10 0x7f92840c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f92837fe640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9284153c80 0x7f92840b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 -- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c4a10 msgr2=0x7f92840c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c4a10 0x7f92840c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 -- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9284153c80 msgr2=0x7f92840b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9284153c80 0x7f92840b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 -- 172.21.15.31:0/1557297609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c009420 con 0x7f92840c42a0 2023-10-30T13:34:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9282ffd640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92840c42a0 0x7f92841ef3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f927400c450 tx=0x7f927400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9280ff9640 1 -- 172.21.15.31:0/1557297609 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274005480 con 0x7f92840c42a0 2023-10-30T13:34:44.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9280ff9640 1 -- 172.21.15.31:0/1557297609 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9274003600 con 0x7f92840c42a0 2023-10-30T13:34:44.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9280ff9640 1 -- 172.21.15.31:0/1557297609 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740039b0 con 0x7f92840c42a0 2023-10-30T13:34:44.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c009b30 con 0x7f92840c42a0 2023-10-30T13:34:44.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92841ef930 con 0x7f92840c42a0 2023-10-30T13:34:44.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9280ff9640 1 -- 172.21.15.31:0/1557297609 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f927401c440 con 0x7f92840c42a0 2023-10-30T13:34:44.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9280ff9640 1 -- 172.21.15.31:0/1557297609 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9274012ad0 con 0x7f92840c42a0 2023-10-30T13:34:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.800+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928404e170 con 0x7f92840c42a0 2023-10-30T13:34:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.804+0000 7f9280ff9640 1 -- 172.21.15.31:0/1557297609 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9274017070 con 0x7f92840c42a0 2023-10-30T13:34:44.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.900+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92840b1b20 con 0x7f92840c42a0 2023-10-30T13:34:44.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.900+0000 7f9280ff9640 1 -- 172.21.15.31:0/1557297609 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9274012400 con 0x7f92840c42a0 2023-10-30T13:34:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:34:44.907 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:34:44.552560+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:34:44.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92840c42a0 msgr2=0x7f92841ef3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:44.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92840c42a0 0x7f92841ef3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f927400c450 tx=0x7f927400c910 comp rx=0 tx=0).stop 2023-10-30T13:34:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 shutdown_connections 2023-10-30T13:34:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9284153c80 0x7f92840b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92840c4a10 0x7f92840c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 --2- 172.21.15.31:0/1557297609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92840c42a0 0x7f92841ef3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 >> 172.21.15.31:0/1557297609 conn(0x7f92840715c0 msgr2=0x7f92841514b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 shutdown_connections 2023-10-30T13:34:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:34:44.904+0000 7f9289a54640 1 -- 172.21.15.31:0/1557297609 wait complete. 2023-10-30T13:34:44.928 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T13:34:44.928 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T13:34:45.199 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-30T13:34:45.195+0000 7f206d6ec640 -1 Falling back to public interface 2023-10-30T13:34:45.206 INFO:tasks.ceph.osd.6.smithi033.stderr:2023-10-30T13:34:45.203+0000 7fa676869640 -1 Falling back to public interface 2023-10-30T13:34:45.216 INFO:tasks.ceph.mgr.y.smithi031.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T13:34:45.216 INFO:tasks.ceph.mgr.y.smithi031.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T13:34:45.217 INFO:tasks.ceph.mgr.y.smithi031.stderr: from numpy import show_config as show_numpy_config 2023-10-30T13:34:45.218 INFO:tasks.ceph.osd.5.smithi033.stderr:2023-10-30T13:34:45.215+0000 7f5d9f4ba640 -1 Falling back to public interface 2023-10-30T13:34:45.220 INFO:tasks.ceph.mgr.x.smithi033.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T13:34:45.220 INFO:tasks.ceph.mgr.x.smithi033.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T13:34:45.220 INFO:tasks.ceph.mgr.x.smithi033.stderr: from numpy import show_config as show_numpy_config 2023-10-30T13:34:45.234 INFO:tasks.ceph.osd.7.smithi033.stderr:2023-10-30T13:34:45.231+0000 7f31f8125640 -1 Falling back to public interface 2023-10-30T13:34:45.236 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-30T13:34:45.232+0000 7f91e3d79640 -1 Falling back to public interface 2023-10-30T13:34:45.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-30T13:34:45.232+0000 7fc4a1080640 -1 Falling back to public interface 2023-10-30T13:34:45.254 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:45.248+0000 7f07ee08d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T13:34:45.256 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:45.251+0000 7f113ebea180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T13:34:45.272 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-10-30T13:34:45.268+0000 7fa9bdb4c640 -1 Falling back to public interface 2023-10-30T13:34:45.285 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-30T13:34:45.280+0000 7f29d7095640 -1 Falling back to public interface 2023-10-30T13:34:45.389 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:45.383+0000 7f113ebea180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:34:45.403 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:45.400+0000 7f07ee08d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:34:45.553 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:45.547+0000 7f113ebea180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T13:34:45.586 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:45.580+0000 7f07ee08d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T13:34:45.631 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:45.627+0000 7f113ebea180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T13:34:45.663 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:45.656+0000 7f07ee08d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T13:34:45.733 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:34:45.760 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:45.755+0000 7f113ebea180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:34:45.808 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:45.804+0000 7f07ee08d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:34:45.866 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.860+0000 7fb65d091640 1 Processor -- start 2023-10-30T13:34:45.866 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.860+0000 7fb65d091640 1 -- start start 2023-10-30T13:34:45.868 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65d091640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb658150820 0x7fb6580c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:45.868 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65d091640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb6580703c0 0x7fb658156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:45.868 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65d091640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb658150c00 con 0x7fb6580c7b10 2023-10-30T13:34:45.868 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65d091640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580717f0 con 0x7fb6580703c0 2023-10-30T13:34:45.869 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65d091640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580c7f30 con 0x7fb658150820 2023-10-30T13:34:45.869 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65659c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb6580c7b10 0x7fb65806fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52670/0 (socket says 172.21.15.31:52670) 2023-10-30T13:34:45.869 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65659c640 1 -- 172.21.15.31:0/646499478 learned_addr learned my addr 172.21.15.31:0/646499478 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:34:45.869 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65759e640 1 --2- 172.21.15.31:0/646499478 >> v2:172.21.15.31:3300/0 conn(0x7fb6580703c0 0x7fb658156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:45.869 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb656d9d640 1 --2- 172.21.15.31:0/646499478 >> v2:172.21.15.33:3300/0 conn(0x7fb658150820 0x7fb6580c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:45.870 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65759e640 1 -- 172.21.15.31:0/646499478 >> v2:172.21.15.33:3300/0 conn(0x7fb658150820 msgr2=0x7fb6580c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:45.870 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65759e640 1 --2- 172.21.15.31:0/646499478 >> v2:172.21.15.33:3300/0 conn(0x7fb658150820 0x7fb6580c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.871 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65759e640 1 -- 172.21.15.31:0/646499478 >> v1:172.21.15.31:6789/0 conn(0x7fb6580c7b10 legacy=0x7fb65806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:45.871 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb65759e640 1 -- 172.21.15.31:0/646499478 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580c6f90 con 0x7fb6580703c0 2023-10-30T13:34:45.871 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.864+0000 7fb655d9b640 1 -- 172.21.15.31:0/646499478 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976767931 0 0) 0x7fb658150c00 con 0x7fb6580c7b10 2023-10-30T13:34:45.872 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65759e640 1 --2- 172.21.15.31:0/646499478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6580703c0 0x7fb658156440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb65806c420 tx=0x7fb65001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c786673973afa046 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:45.873 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb655d9b640 1 -- 172.21.15.31:0/646499478 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500b0440 con 0x7fb6580703c0 2023-10-30T13:34:45.873 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb655d9b640 1 -- 172.21.15.31:0/646499478 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6500b05a0 con 0x7fb6580703c0 2023-10-30T13:34:45.873 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb655d9b640 1 -- 172.21.15.31:0/646499478 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500b0930 con 0x7fb6580703c0 2023-10-30T13:34:45.873 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 -- 172.21.15.31:0/646499478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6580703c0 msgr2=0x7fb658156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:45.873 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 --2- 172.21.15.31:0/646499478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6580703c0 0x7fb658156440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb65806c420 tx=0x7fb65001f0f0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.874 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 -- 172.21.15.31:0/646499478 shutdown_connections 2023-10-30T13:34:45.874 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 --2- 172.21.15.31:0/646499478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6580703c0 0x7fb658156440 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.874 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 --2- 172.21.15.31:0/646499478 >> v2:172.21.15.33:3300/0 conn(0x7fb658150820 0x7fb6580c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.874 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 -- 172.21.15.31:0/646499478 >> 172.21.15.31:0/646499478 conn(0x7fb65806b5c0 msgr2=0x7fb65806b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:45.874 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 -- 172.21.15.31:0/646499478 shutdown_connections 2023-10-30T13:34:45.875 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 -- 172.21.15.31:0/646499478 wait complete. 2023-10-30T13:34:45.877 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 Processor -- start 2023-10-30T13:34:45.877 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.868+0000 7fb65d091640 1 -- start start 2023-10-30T13:34:45.878 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb6580703c0 0x7fb658147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:45.878 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6580c7b10 0x7fb65814ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:45.878 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb658150820 0x7fb6581426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:45.878 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581437a0 con 0x7fb658150820 2023-10-30T13:34:45.878 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb658144630 con 0x7fb6580c7b10 2023-10-30T13:34:45.879 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb658147a50 con 0x7fb6580703c0 2023-10-30T13:34:45.879 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb658150820 0x7fb6581426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:45.879 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb6580703c0 0x7fb658147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:45.879 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb6580703c0 0x7fb658147510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:56396/0 (socket says 172.21.15.31:56396) 2023-10-30T13:34:45.879 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6580c7b10 0x7fb65814ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:45.879 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 -- 172.21.15.31:0/3853952086 learned_addr learned my addr 172.21.15.31:0/3853952086 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:34:45.880 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 -- 172.21.15.31:0/3853952086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6580c7b10 msgr2=0x7fb65814ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:45.880 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 --2- 172.21.15.31:0/3853952086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6580c7b10 0x7fb65814ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.880 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 -- 172.21.15.31:0/3853952086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb658150820 msgr2=0x7fb6581426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:45.880 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 --2- 172.21.15.31:0/3853952086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb658150820 0x7fb6581426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.880 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 -- 172.21.15.31:0/3853952086 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650009a70 con 0x7fb6580703c0 2023-10-30T13:34:45.881 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb656d9d640 1 --2- 172.21.15.31:0/3853952086 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb6580703c0 0x7fb658147510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb64800e9d0 tx=0x7fb64800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:45.881 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb63f7fe640 1 -- 172.21.15.31:0/3853952086 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800de60 con 0x7fb6580703c0 2023-10-30T13:34:45.881 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb63f7fe640 1 -- 172.21.15.31:0/3853952086 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb648010440 con 0x7fb6580703c0 2023-10-30T13:34:45.881 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb63f7fe640 1 -- 172.21.15.31:0/3853952086 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6480107b0 con 0x7fb6580703c0 2023-10-30T13:34:45.881 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6581e1970 con 0x7fb6580703c0 2023-10-30T13:34:45.881 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6581e1690 con 0x7fb6580703c0 2023-10-30T13:34:45.882 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb63f7fe640 1 -- 172.21.15.31:0/3853952086 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb648010c10 con 0x7fb6580703c0 2023-10-30T13:34:45.882 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb63f7fe640 1 -- 172.21.15.31:0/3853952086 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb648018d90 con 0x7fb6580703c0 2023-10-30T13:34:45.882 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.872+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65804e170 con 0x7fb6580703c0 2023-10-30T13:34:45.883 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.876+0000 7fb63f7fe640 1 -- 172.21.15.31:0/3853952086 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb64801d020 con 0x7fb6580703c0 2023-10-30T13:34:45.939 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6580c7370 con 0x7fb6580703c0 2023-10-30T13:34:45.940 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb63f7fe640 1 -- 172.21.15.31:0/3853952086 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb648026020 con 0x7fb6580703c0 2023-10-30T13:34:45.941 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-30T13:34:45.941 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":10,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:34:44.552560+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:34:45.942 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb6580703c0 msgr2=0x7fb658147510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:45.942 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 --2- 172.21.15.31:0/3853952086 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb6580703c0 0x7fb658147510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb64800e9d0 tx=0x7fb64800ee90 comp rx=0 tx=0).stop 2023-10-30T13:34:45.942 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 shutdown_connections 2023-10-30T13:34:45.943 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 --2- 172.21.15.31:0/3853952086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb658150820 0x7fb6581426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.943 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 --2- 172.21.15.31:0/3853952086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6580c7b10 0x7fb65814ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.943 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 --2- 172.21.15.31:0/3853952086 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb6580703c0 0x7fb658147510 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:45.943 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 >> 172.21.15.31:0/3853952086 conn(0x7fb65806b5c0 msgr2=0x7fb6580723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:45.943 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 shutdown_connections 2023-10-30T13:34:45.944 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:45.936+0000 7fb65d091640 1 -- 172.21.15.31:0/3853952086 wait complete. 2023-10-30T13:34:45.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T13:34:46.412 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:46.407+0000 7f113ebea180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T13:34:46.493 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:46.487+0000 7f113ebea180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T13:34:46.515 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:46.508+0000 7f07ee08d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T13:34:46.606 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:46.600+0000 7f07ee08d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T13:34:46.671 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:46.667+0000 7f113ebea180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T13:34:46.749 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:46.743+0000 7f113ebea180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T13:34:46.806 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:46.800+0000 7f07ee08d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T13:34:46.895 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:46.888+0000 7f07ee08d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T13:34:47.055 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:47.051+0000 7f113ebea180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T13:34:47.117 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:47.111+0000 7f113ebea180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T13:34:47.246 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:47.240+0000 7f07ee08d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T13:34:47.317 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-10-30T13:34:47.312+0000 7f07ee08d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T13:34:48.162 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:48.159+0000 7f10e6361640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T13:34:48.164 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-10-30T13:34:48.159+0000 7f10e6361640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T13:34:49.035 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-30T13:34:49.031+0000 7f206d6ec640 -1 osd.4 0 log_to_monitors true 2023-10-30T13:34:49.089 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-30T13:34:49.084+0000 7f91e3d79640 -1 osd.0 0 log_to_monitors true 2023-10-30T13:34:49.122 INFO:tasks.ceph.osd.6.smithi033.stderr:2023-10-30T13:34:49.119+0000 7fa676869640 -1 osd.6 0 log_to_monitors true 2023-10-30T13:34:49.134 INFO:tasks.ceph.osd.5.smithi033.stderr:2023-10-30T13:34:49.131+0000 7f5d9f4ba640 -1 osd.5 0 log_to_monitors true 2023-10-30T13:34:49.158 INFO:tasks.ceph.osd.7.smithi033.stderr:2023-10-30T13:34:49.155+0000 7f31f8125640 -1 osd.7 0 log_to_monitors true 2023-10-30T13:34:49.225 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-10-30T13:34:49.220+0000 7fa9bdb4c640 -1 osd.3 0 log_to_monitors true 2023-10-30T13:34:49.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-30T13:34:49.428+0000 7fc4a1080640 -1 osd.1 0 log_to_monitors true 2023-10-30T13:34:49.482 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-30T13:34:49.476+0000 7f29d7095640 -1 osd.2 0 log_to_monitors true 2023-10-30T13:34:51.203 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-30T13:34:51.200+0000 7f91e0d36640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T13:34:51.220 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-30T13:34:51.216+0000 7f91d7b45640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:52.067 INFO:tasks.ceph.osd.6.smithi033.stderr:2023-10-30T13:34:52.063+0000 7fa671fee640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T13:34:52.094 INFO:tasks.ceph.osd.6.smithi033.stderr:2023-10-30T13:34:52.091+0000 7fa66a635640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:52.115 INFO:tasks.ceph.osd.5.smithi033.stderr:2023-10-30T13:34:52.111+0000 7f5d9ac3f640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T13:34:52.137 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-30T13:34:52.131+0000 7f2068e71640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T13:34:52.146 INFO:tasks.ceph.osd.5.smithi033.stderr:2023-10-30T13:34:52.143+0000 7f5d93286640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:52.158 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-30T13:34:52.155+0000 7f20614b8640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:52.549 INFO:tasks.ceph.osd.7.smithi033.stderr:2023-10-30T13:34:52.547+0000 7f31f38aa640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T13:34:52.591 INFO:tasks.ceph.osd.7.smithi033.stderr:2023-10-30T13:34:52.587+0000 7f31ebef1640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:52.765 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:34:52.975 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.972+0000 7f7124b2a640 1 Processor -- start 2023-10-30T13:34:52.975 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.972+0000 7f7124b2a640 1 -- start start 2023-10-30T13:34:52.979 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.972+0000 7f7124b2a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f71200c4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.979 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.972+0000 7f7124b2a640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f712015a1f0 0x7f71200cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.979 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.972+0000 7f7124b2a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71200caa30 con 0x7f71200ca040 2023-10-30T13:34:52.979 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.972+0000 7f7124b2a640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7120153f10 con 0x7f712015a1f0 2023-10-30T13:34:52.979 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.972+0000 7f7124b2a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712015cde0 con 0x7f7120153b30 2023-10-30T13:34:52.980 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711ed9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f712015a1f0 0x7f71200cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.980 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711ed9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f712015a1f0 0x7f71200cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:47450/0 (socket says 172.21.15.31:47450) 2023-10-30T13:34:52.980 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711ed9d640 1 -- 172.21.15.31:0/649355280 learned_addr learned my addr 172.21.15.31:0/649355280 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:34:52.981 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711e59c640 1 --2- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f71200c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.981 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711ed9d640 1 -- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 msgr2=0x7f71200c4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:52.981 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711ed9d640 1 --2- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f71200c4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.982 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711ed9d640 1 -- 172.21.15.31:0/649355280 >> v1:172.21.15.33:6789/0 conn(0x7f71200ca040 legacy=0x7f712015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.982 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711ed9d640 1 -- 172.21.15.31:0/649355280 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712015dc80 con 0x7f712015a1f0 2023-10-30T13:34:52.982 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.976+0000 7f711e59c640 1 --2- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f71200c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:52.987 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f711ed9d640 1 --2- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 0x7f71200cf030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7118009e50 tx=0x7f711801f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2d4a0dbdfca8cc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:52.987 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f711dd9b640 1 -- 172.21.15.31:0/649355280 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71180af400 con 0x7f712015a1f0 2023-10-30T13:34:52.988 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f711dd9b640 1 -- 172.21.15.31:0/649355280 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71180ab050 con 0x7f712015a1f0 2023-10-30T13:34:52.988 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f711dd9b640 1 -- 172.21.15.31:0/649355280 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71180af780 con 0x7f712015a1f0 2023-10-30T13:34:52.989 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 -- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 msgr2=0x7f71200cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.989 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 --2- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 0x7f71200cf030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7118009e50 tx=0x7f711801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.989 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 -- 172.21.15.31:0/649355280 shutdown_connections 2023-10-30T13:34:52.989 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 --2- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 0x7f71200cf030 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.989 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 --2- 172.21.15.31:0/649355280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f71200c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.990 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 -- 172.21.15.31:0/649355280 >> 172.21.15.31:0/649355280 conn(0x7f712006bfe0 msgr2=0x7f712006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:52.990 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 -- 172.21.15.31:0/649355280 shutdown_connections 2023-10-30T13:34:52.990 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.984+0000 7f7124b2a640 1 -- 172.21.15.31:0/649355280 wait complete. 2023-10-30T13:34:52.991 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 Processor -- start 2023-10-30T13:34:52.994 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 -- start start 2023-10-30T13:34:52.994 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200ca040 0x7f712014f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.994 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f7120149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.994 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 0x7f71201496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.994 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712014ac10 con 0x7f71200ca040 2023-10-30T13:34:52.995 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7120150620 con 0x7f712015a1f0 2023-10-30T13:34:52.995 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7124b2a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71200c7690 con 0x7f7120153b30 2023-10-30T13:34:52.995 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200ca040 0x7f712014f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.995 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200ca040 0x7f712014f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51968/0 (socket says 172.21.15.31:51968) 2023-10-30T13:34:52.995 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 -- 172.21.15.31:0/2115210206 learned_addr learned my addr 172.21.15.31:0/2115210206 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:34:52.996 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f7117fff640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f7120149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.996 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711ed9d640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 0x7f71201496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.997 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 -- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 msgr2=0x7f7120149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.997 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f7120149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.997 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 -- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 msgr2=0x7f71201496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.997 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 0x7f71201496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.997 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.988+0000 7f711e59c640 1 -- 172.21.15.31:0/2115210206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7118009b30 con 0x7f71200ca040 2023-10-30T13:34:52.998 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.992+0000 7f711e59c640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200ca040 0x7f712014f1c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f711000e810 tx=0x7f711000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:52.998 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.992+0000 7f7116ffd640 1 -- 172.21.15.31:0/2115210206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711000c8d0 con 0x7f71200ca040 2023-10-30T13:34:52.998 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.992+0000 7f7116ffd640 1 -- 172.21.15.31:0/2115210206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711000ca30 con 0x7f71200ca040 2023-10-30T13:34:52.998 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.992+0000 7f7116ffd640 1 -- 172.21.15.31:0/2115210206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711000cda0 con 0x7f71200ca040 2023-10-30T13:34:52.999 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.992+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7120150b10 con 0x7f71200ca040 2023-10-30T13:34:52.999 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.992+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7120150830 con 0x7f71200ca040 2023-10-30T13:34:53.000 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.992+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71200c7010 con 0x7f71200ca040 2023-10-30T13:34:53.000 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.996+0000 7f7116ffd640 1 -- 172.21.15.31:0/2115210206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) v1 ==== 85794+0+0 (secure 0 0 0) 0x7f7110013410 con 0x7f71200ca040 2023-10-30T13:34:53.001 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.996+0000 7f7116ffd640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7100066410 0x7f71000688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:53.001 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.996+0000 7f7116ffd640 1 -- 172.21.15.31:0/2115210206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3351+0+0 (secure 0 0 0) 0x7f7110083210 con 0x7f71200ca040 2023-10-30T13:34:53.002 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:52.996+0000 7f7117fff640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7100066410 0x7f71000688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:53.003 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.000+0000 7f7117fff640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7100066410 0x7f71000688d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f712014db60 tx=0x7f7108009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:53.009 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.004+0000 7f7116ffd640 1 -- 172.21.15.31:0/2115210206 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7110051150 con 0x7f71200ca040 2023-10-30T13:34:53.144 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-30T13:34:53.140+0000 7f29d281c640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T13:34:53.173 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-30T13:34:53.168+0000 7f29cae63640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:53.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-30T13:34:53.220+0000 7fc49c805640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T13:34:53.241 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.236+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f712004e170 con 0x7f71200ca040 2023-10-30T13:34:53.247 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-10-30T13:34:53.244+0000 7fa9b92d1640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T13:34:53.250 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.244+0000 7f7116ffd640 1 -- 172.21.15.31:0/2115210206 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8952 (secure 0 0 0) 0x7f7110054770 con 0x7f71200ca040 2023-10-30T13:34:53.250 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-30T13:34:53.250 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":14,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:34:52.225488+0000","last_up_change":"2023-10-30T13:34:52.225488+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:34:53.253 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7100066410 msgr2=0x7f71000688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:53.253 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7100066410 0x7f71000688d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f712014db60 tx=0x7f7108009040 comp rx=0 tx=0).stop 2023-10-30T13:34:53.254 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200ca040 msgr2=0x7f712014f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:53.254 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200ca040 0x7f712014f1c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f711000e810 tx=0x7f711000ecd0 comp rx=0 tx=0).stop 2023-10-30T13:34:53.255 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 shutdown_connections 2023-10-30T13:34:53.255 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7100066410 0x7f71000688d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:53.255 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f712015a1f0 0x7f71201496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:53.255 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7120153b30 0x7f7120149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:53.255 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 --2- 172.21.15.31:0/2115210206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200ca040 0x7f712014f1c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:53.255 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 >> 172.21.15.31:0/2115210206 conn(0x7f712006bfe0 msgr2=0x7f71200b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:53.256 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 shutdown_connections 2023-10-30T13:34:53.256 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:34:53.248+0000 7f7124b2a640 1 -- 172.21.15.31:0/2115210206 wait complete. 2023-10-30T13:34:53.261 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-30T13:34:53.256+0000 7fc494e4c640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:53.275 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-10-30T13:34:53.284 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-10-30T13:34:53.280+0000 7fa9b1918640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:34:54.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:54.863 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:00.081 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:00.216 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fe53e640 1 Processor -- start 2023-10-30T13:35:00.216 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fe53e640 1 -- start start 2023-10-30T13:35:00.218 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fe53e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f20f8153c30 0x7f20f80c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.218 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fe53e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f815a370 con 0x7f20f80c9c60 2023-10-30T13:35:00.218 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fe53e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f815a8a0 con 0x7f20f80cb0e0 2023-10-30T13:35:00.218 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fe53e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f80c1a10 con 0x7f20f8153c30 2023-10-30T13:35:00.219 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20f7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f20f8153c30 0x7f20f80c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.219 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fcadb640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f20f80c9c60 0x7f20f8164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54734/0 (socket says 172.21.15.31:54734) 2023-10-30T13:35:00.219 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20fcadb640 1 -- 172.21.15.31:0/3206500192 learned_addr learned my addr 172.21.15.31:0/3206500192 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:00.219 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20f7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f20f8153c30 0x7f20f80c67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42442/0 (socket says 172.21.15.31:42442) 2023-10-30T13:35:00.220 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20f7fff640 1 -- 172.21.15.31:0/3206500192 >> v1:172.21.15.33:6789/0 conn(0x7f20f80cb0e0 legacy=0x7f20f806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.220 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20f7fff640 1 -- 172.21.15.31:0/3206500192 >> v1:172.21.15.31:6789/0 conn(0x7f20f80c9c60 legacy=0x7f20f8164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.220 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20f7fff640 1 -- 172.21.15.31:0/3206500192 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f80c1340 con 0x7f20f8153c30 2023-10-30T13:35:00.220 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.212+0000 7f20f6ffd640 1 -- 172.21.15.31:0/3206500192 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722218665 0 0) 0x7f20f815a370 con 0x7f20f80c9c60 2023-10-30T13:35:00.221 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20f7fff640 1 --2- 172.21.15.31:0/3206500192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8153c30 0x7f20f80c67f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f20e80097c0 tx=0x7f20e801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a80999977c85ce5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.222 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20f6ffd640 1 -- 172.21.15.31:0/3206500192 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80ab5f0 con 0x7f20f8153c30 2023-10-30T13:35:00.222 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20f6ffd640 1 -- 172.21.15.31:0/3206500192 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e80ab750 con 0x7f20f8153c30 2023-10-30T13:35:00.223 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20f6ffd640 1 -- 172.21.15.31:0/3206500192 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80abae0 con 0x7f20f8153c30 2023-10-30T13:35:00.223 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20fe53e640 1 -- 172.21.15.31:0/3206500192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8153c30 msgr2=0x7f20f80c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.223 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20fe53e640 1 --2- 172.21.15.31:0/3206500192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8153c30 0x7f20f80c67f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f20e80097c0 tx=0x7f20e801c3f0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.223 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20fe53e640 1 -- 172.21.15.31:0/3206500192 shutdown_connections 2023-10-30T13:35:00.223 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20fe53e640 1 --2- 172.21.15.31:0/3206500192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8153c30 0x7f20f80c67f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.224 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20fe53e640 1 -- 172.21.15.31:0/3206500192 >> 172.21.15.31:0/3206500192 conn(0x7f20f80cc6d0 msgr2=0x7f20f80c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:00.224 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20fe53e640 1 -- 172.21.15.31:0/3206500192 shutdown_connections 2023-10-30T13:35:00.224 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.216+0000 7f20fe53e640 1 -- 172.21.15.31:0/3206500192 wait complete. 2023-10-30T13:35:00.224 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 Processor -- start 2023-10-30T13:35:00.225 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 -- start start 2023-10-30T13:35:00.226 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80c9c60 0x7f20f8150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.227 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20f80cb0e0 0x7f20f815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.227 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f20f8153c30 0x7f20f81f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.227 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f81506b0 con 0x7f20f80c9c60 2023-10-30T13:35:00.227 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f815d6d0 con 0x7f20f80cb0e0 2023-10-30T13:35:00.227 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fe53e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f81f2760 con 0x7f20f8153c30 2023-10-30T13:35:00.228 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80c9c60 0x7f20f8150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.228 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80c9c60 0x7f20f8150170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42454/0 (socket says 172.21.15.31:42454) 2023-10-30T13:35:00.228 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 -- 172.21.15.31:0/2396852448 learned_addr learned my addr 172.21.15.31:0/2396852448 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:00.228 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20fcadb640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f20f8153c30 0x7f20f81f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.228 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f77fe640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20f80cb0e0 0x7f20f815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.229 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 -- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f20f8153c30 msgr2=0x7f20f81f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.229 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f20f8153c30 0x7f20f81f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.229 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 -- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20f80cb0e0 msgr2=0x7f20f815c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.229 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20f80cb0e0 0x7f20f815c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.229 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 -- 172.21.15.31:0/2396852448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e8009420 con 0x7f20f80c9c60 2023-10-30T13:35:00.229 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f77fe640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20f80cb0e0 0x7f20f815c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:00.230 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f7fff640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80c9c60 0x7f20f8150170 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f20e8019000 tx=0x7f20e801ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.230 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f4ff9640 1 -- 172.21.15.31:0/2396852448 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80b5810 con 0x7f20f80c9c60 2023-10-30T13:35:00.230 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.220+0000 7f20f4ff9640 1 -- 172.21.15.31:0/2396852448 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e80b5990 con 0x7f20f80c9c60 2023-10-30T13:35:00.230 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20f4ff9640 1 -- 172.21.15.31:0/2396852448 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80b5d40 con 0x7f20f80c9c60 2023-10-30T13:35:00.230 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f8159450 con 0x7f20f80c9c60 2023-10-30T13:35:00.231 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f815d920 con 0x7f20f80c9c60 2023-10-30T13:35:00.231 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f804e170 con 0x7f20f80c9c60 2023-10-30T13:35:00.232 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20f4ff9640 1 -- 172.21.15.31:0/2396852448 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20e80b4390 con 0x7f20f80c9c60 2023-10-30T13:35:00.232 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20f4ff9640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f20e40664a0 0x7f20e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.232 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20f4ff9640 1 -- 172.21.15.31:0/2396852448 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20e80a8050 con 0x7f20f80c9c60 2023-10-30T13:35:00.232 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.224+0000 7f20f77fe640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f20e40664a0 0x7f20e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.232 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.228+0000 7f20f77fe640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f20e40664a0 0x7f20e4068960 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20f8153320 tx=0x7f20ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.234 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.228+0000 7f20f4ff9640 1 -- 172.21.15.31:0/2396852448 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20e80ebbd0 con 0x7f20f80c9c60 2023-10-30T13:35:00.398 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.392+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20f80c2380 con 0x7f20f80c9c60 2023-10-30T13:35:00.399 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20f4ff9640 1 -- 172.21.15.31:0/2396852448 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12790 (secure 0 0 0) 0x7f20e80c5090 con 0x7f20f80c9c60 2023-10-30T13:35:00.399 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-30T13:35:00.400 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":17,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:34:55.560761+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:00.401 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f20e40664a0 msgr2=0x7f20e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.401 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f20e40664a0 0x7f20e4068960 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20f8153320 tx=0x7f20ec009040 comp rx=0 tx=0).stop 2023-10-30T13:35:00.402 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80c9c60 msgr2=0x7f20f8150170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.402 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80c9c60 0x7f20f8150170 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f20e8019000 tx=0x7f20e801ac30 comp rx=0 tx=0).stop 2023-10-30T13:35:00.402 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 shutdown_connections 2023-10-30T13:35:00.402 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f20e40664a0 0x7f20e4068960 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.403 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f20f8153c30 0x7f20f81f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.403 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20f80cb0e0 0x7f20f815c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.403 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 --2- 172.21.15.31:0/2396852448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80c9c60 0x7f20f8150170 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.403 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 >> 172.21.15.31:0/2396852448 conn(0x7f20f80cc6d0 msgr2=0x7f20f80ca4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:00.403 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 shutdown_connections 2023-10-30T13:35:00.403 INFO:teuthology.misc.health.smithi031.stderr:2023-10-30T13:35:00.396+0000 7f20fe53e640 1 -- 172.21.15.31:0/2396852448 wait complete. 2023-10-30T13:35:00.420 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T13:35:00.420 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T13:35:00.420 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:35:00.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:35:00.424 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:35:00.424 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:35:00.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:35:00.428 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:35:00.428 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:35:00.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:35:00.432 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:35:00.433 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T13:35:00.433 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T13:35:00.436 INFO:tasks.ceph.mds.b:Started 2023-10-30T13:35:00.436 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:35:00.437 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:35:00.439 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:35:00.440 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T13:35:00.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T13:35:00.443 INFO:tasks.ceph.mds.f:Started 2023-10-30T13:35:00.443 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-30T13:35:00.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:00.519 INFO:tasks.ceph.mds.d.smithi033.stdout:starting mds.d at 2023-10-30T13:35:00.520 INFO:tasks.ceph.mds.f.smithi033.stdout:starting mds.f at 2023-10-30T13:35:00.520 INFO:tasks.ceph.mds.b.smithi033.stdout:starting mds.b at 2023-10-30T13:35:00.530 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2023-10-30T13:35:00.547 INFO:tasks.ceph.mds.e.smithi031.stdout:starting mds.e at 2023-10-30T13:35:00.555 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2023-10-30T13:35:00.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e39417640 1 Processor -- start 2023-10-30T13:35:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e39417640 1 -- start start 2023-10-30T13:35:00.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e39417640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9e34159d50 0x7f9e3415a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e39417640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3415a6c0 con 0x7f9e34153b10 2023-10-30T13:35:00.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e39417640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3415d070 con 0x7f9e340c9b90 2023-10-30T13:35:00.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e39417640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e34159900 con 0x7f9e34159d50 2023-10-30T13:35:00.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e32ffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f9e34153b10 0x7f9e340ca8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54772/0 (socket says 172.21.15.31:54772) 2023-10-30T13:35:00.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e32ffd640 1 -- 172.21.15.31:0/3596058024 learned_addr learned my addr 172.21.15.31:0/3596058024 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:00.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e337fe640 1 --2- 172.21.15.31:0/3596058024 >> v2:172.21.15.31:3300/0 conn(0x7f9e34159d50 0x7f9e3415a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.612+0000 7f9e31ffb640 1 -- 172.21.15.31:0/3596058024 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709431236 0 0) 0x7f9e3415a6c0 con 0x7f9e34153b10 2023-10-30T13:35:00.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.616+0000 7f9e31ffb640 1 -- 172.21.15.31:0/3596058024 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e1c003530 con 0x7f9e34153b10 2023-10-30T13:35:00.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.616+0000 7f9e31ffb640 1 -- 172.21.15.31:0/3596058024 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067420255 0 0) 0x7f9e3415d070 con 0x7f9e340c9b90 2023-10-30T13:35:00.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.616+0000 7f9e337fe640 1 -- 172.21.15.31:0/3596058024 >> v1:172.21.15.33:6789/0 conn(0x7f9e340c9b90 legacy=0x7f9e3406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.616+0000 7f9e337fe640 1 -- 172.21.15.31:0/3596058024 >> v1:172.21.15.31:6789/0 conn(0x7f9e34153b10 legacy=0x7f9e340ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.616+0000 7f9e337fe640 1 -- 172.21.15.31:0/3596058024 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3415d350 con 0x7f9e34159d50 2023-10-30T13:35:00.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e337fe640 1 --2- 172.21.15.31:0/3596058024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34159d50 0x7f9e3415a180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009d90 tx=0x7f9e2c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e56a0c57950e08b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e31ffb640 1 -- 172.21.15.31:0/3596058024 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0a5890 con 0x7f9e34159d50 2023-10-30T13:35:00.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e31ffb640 1 -- 172.21.15.31:0/3596058024 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2c0a59f0 con 0x7f9e34159d50 2023-10-30T13:35:00.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e31ffb640 1 -- 172.21.15.31:0/3596058024 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0a5d80 con 0x7f9e34159d50 2023-10-30T13:35:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 -- 172.21.15.31:0/3596058024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34159d50 msgr2=0x7f9e3415a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 --2- 172.21.15.31:0/3596058024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34159d50 0x7f9e3415a180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009d90 tx=0x7f9e2c01c8c0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 -- 172.21.15.31:0/3596058024 shutdown_connections 2023-10-30T13:35:00.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 --2- 172.21.15.31:0/3596058024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34159d50 0x7f9e3415a180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 -- 172.21.15.31:0/3596058024 >> 172.21.15.31:0/3596058024 conn(0x7f9e3406a240 msgr2=0x7f9e3406a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:00.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 -- 172.21.15.31:0/3596058024 shutdown_connections 2023-10-30T13:35:00.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 -- 172.21.15.31:0/3596058024 wait complete. 2023-10-30T13:35:00.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.620+0000 7f9e39417640 1 Processor -- start 2023-10-30T13:35:00.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 -- start start 2023-10-30T13:35:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e340c9b90 0x7f9e3415c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34153b10 0x7f9e3415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e34159d50 0x7f9e34141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e34144070 con 0x7f9e34153b10 2023-10-30T13:35:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e341427d0 con 0x7f9e340c9b90 2023-10-30T13:35:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3415ba00 con 0x7f9e34159d50 2023-10-30T13:35:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34153b10 0x7f9e3415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34153b10 0x7f9e3415c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42506/0 (socket says 172.21.15.31:42506) 2023-10-30T13:35:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 -- 172.21.15.31:0/3413937570 learned_addr learned my addr 172.21.15.31:0/3413937570 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e337fe640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e34159d50 0x7f9e34141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e32ffd640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e340c9b90 0x7f9e3415c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 -- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e34159d50 msgr2=0x7f9e34141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e34159d50 0x7f9e34141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 -- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e340c9b90 msgr2=0x7f9e3415c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e340c9b90 0x7f9e3415c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 -- 172.21.15.31:0/3413937570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c009a70 con 0x7f9e34153b10 2023-10-30T13:35:00.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e327fc640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34153b10 0x7f9e3415c6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800c4d0 tx=0x7f9e2800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e1b7fe640 1 -- 172.21.15.31:0/3413937570 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28005480 con 0x7f9e34153b10 2023-10-30T13:35:00.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e1b7fe640 1 -- 172.21.15.31:0/3413937570 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e28003600 con 0x7f9e34153b10 2023-10-30T13:35:00.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e1b7fe640 1 -- 172.21.15.31:0/3413937570 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280039b0 con 0x7f9e34153b10 2023-10-30T13:35:00.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e34152040 con 0x7f9e34153b10 2023-10-30T13:35:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e34144500 con 0x7f9e34153b10 2023-10-30T13:35:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.624+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e34067540 con 0x7f9e34153b10 2023-10-30T13:35:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.628+0000 7f9e1b7fe640 1 -- 172.21.15.31:0/3413937570 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e28003b10 con 0x7f9e34153b10 2023-10-30T13:35:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.628+0000 7f9e1b7fe640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e200664a0 0x7f9e20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.628+0000 7f9e1b7fe640 1 -- 172.21.15.31:0/3413937570 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9e280831b0 con 0x7f9e34153b10 2023-10-30T13:35:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.628+0000 7f9e32ffd640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e200664a0 0x7f9e20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.628+0000 7f9e32ffd640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e200664a0 0x7f9e20068960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e24005890 tx=0x7f9e24009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.632+0000 7f9e1b7fe640 1 -- 172.21.15.31:0/3413937570 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e280509a0 con 0x7f9e34153b10 2023-10-30T13:35:00.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e340c5f50 con 0x7f9e34153b10 2023-10-30T13:35:00.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e1b7fe640 1 -- 172.21.15.31:0/3413937570 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12790 (secure 0 0 0) 0x7f9e28017090 con 0x7f9e34153b10 2023-10-30T13:35:00.797 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:00.797 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:34:55.560761+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:00.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e200664a0 msgr2=0x7f9e20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e200664a0 0x7f9e20068960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e24005890 tx=0x7f9e24009040 comp rx=0 tx=0).stop 2023-10-30T13:35:00.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34153b10 msgr2=0x7f9e3415c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34153b10 0x7f9e3415c6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800c4d0 tx=0x7f9e2800c990 comp rx=0 tx=0).stop 2023-10-30T13:35:00.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 shutdown_connections 2023-10-30T13:35:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e200664a0 0x7f9e20068960 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e34159d50 0x7f9e34141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e34153b10 0x7f9e3415c6d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 --2- 172.21.15.31:0/3413937570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e340c9b90 0x7f9e3415c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 >> 172.21.15.31:0/3413937570 conn(0x7f9e3406a240 msgr2=0x7f9e340cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 shutdown_connections 2023-10-30T13:35:00.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.792+0000 7f9e39417640 1 -- 172.21.15.31:0/3413937570 wait complete. 2023-10-30T13:35:00.818 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:35:00.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:35:00.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 Processor -- start 2023-10-30T13:35:00.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 -- start start 2023-10-30T13:35:00.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd6d4153ba0 0x7fd6d40cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd6d40c4410 0x7fd6d415a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d4153f80 con 0x7fd6d4153ba0 2023-10-30T13:35:00.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d4159df0 con 0x7fd6d40c4410 2023-10-30T13:35:00.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6da10f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d415a7c0 con 0x7fd6d40c4810 2023-10-30T13:35:00.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd6d4153ba0 0x7fd6d40cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd6d4153ba0 0x7fd6d40cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42516/0 (socket says 172.21.15.31:42516) 2023-10-30T13:35:00.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 -- 172.21.15.31:0/2398482884 learned_addr learned my addr 172.21.15.31:0/2398482884 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:00.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d3fff640 1 --2- 172.21.15.31:0/2398482884 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d2ffd640 1 --2- 172.21.15.31:0/2398482884 >> v2:172.21.15.33:3300/0 conn(0x7fd6d40c4410 0x7fd6d415a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 -- 172.21.15.31:0/2398482884 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 msgr2=0x7fd6d415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:00.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 --2- 172.21.15.31:0/2398482884 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 -- 172.21.15.31:0/2398482884 >> v2:172.21.15.33:3300/0 conn(0x7fd6d40c4410 msgr2=0x7fd6d415a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:00.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 --2- 172.21.15.31:0/2398482884 >> v2:172.21.15.33:3300/0 conn(0x7fd6d40c4410 0x7fd6d415a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.960+0000 7fd6d37fe640 1 -- 172.21.15.31:0/2398482884 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d415d2f0 con 0x7fd6d4153ba0 2023-10-30T13:35:00.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6d37fe640 1 --2- 172.21.15.31:0/2398482884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40cb4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4072540 tx=0x7fd6c401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6ef8551f2a5ac5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6d27fc640 1 -- 172.21.15.31:0/2398482884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40a7070 con 0x7fd6d4153ba0 2023-10-30T13:35:00.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6d27fc640 1 -- 172.21.15.31:0/2398482884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c40ab430 con 0x7fd6d4153ba0 2023-10-30T13:35:00.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6d27fc640 1 -- 172.21.15.31:0/2398482884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40ab7c0 con 0x7fd6d4153ba0 2023-10-30T13:35:00.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 -- 172.21.15.31:0/2398482884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 msgr2=0x7fd6d40cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 --2- 172.21.15.31:0/2398482884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40cb4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4072540 tx=0x7fd6c401c470 comp rx=0 tx=0).stop 2023-10-30T13:35:00.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 -- 172.21.15.31:0/2398482884 shutdown_connections 2023-10-30T13:35:00.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 --2- 172.21.15.31:0/2398482884 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 --2- 172.21.15.31:0/2398482884 >> v2:172.21.15.33:3300/0 conn(0x7fd6d40c4410 0x7fd6d415a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 --2- 172.21.15.31:0/2398482884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40cb4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 -- 172.21.15.31:0/2398482884 >> 172.21.15.31:0/2398482884 conn(0x7fd6d4071790 msgr2=0x7fd6d4071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:00.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 -- 172.21.15.31:0/2398482884 shutdown_connections 2023-10-30T13:35:00.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.964+0000 7fd6da10f640 1 -- 172.21.15.31:0/2398482884 wait complete. 2023-10-30T13:35:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 Processor -- start 2023-10-30T13:35:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 -- start start 2023-10-30T13:35:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d40c4410 0x7fd6d40c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d40c84d0 con 0x7fd6d4153ba0 2023-10-30T13:35:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d415c290 con 0x7fd6d40c4410 2023-10-30T13:35:00.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d4153420 con 0x7fd6d40c4810 2023-10-30T13:35:00.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42530/0 (socket says 172.21.15.31:42530) 2023-10-30T13:35:00.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 -- 172.21.15.31:0/117800766 learned_addr learned my addr 172.21.15.31:0/117800766 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:00.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d37fe640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d40c4410 0x7fd6d40c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d2ffd640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 -- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 msgr2=0x7fd6d415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 -- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d40c4410 msgr2=0x7fd6d40c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:00.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d40c4410 0x7fd6d40c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:00.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 -- 172.21.15.31:0/117800766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c40094d0 con 0x7fd6d4153ba0 2023-10-30T13:35:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6d3fff640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40c8a10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc007e60 tx=0x7fd6cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6bbfff640 1 -- 172.21.15.31:0/117800766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc014620 con 0x7fd6d4153ba0 2023-10-30T13:35:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6bbfff640 1 -- 172.21.15.31:0/117800766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc014780 con 0x7fd6d4153ba0 2023-10-30T13:35:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6bbfff640 1 -- 172.21.15.31:0/117800766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc014b30 con 0x7fd6d4153ba0 2023-10-30T13:35:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d415ae50 con 0x7fd6d4153ba0 2023-10-30T13:35:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.968+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d415b410 con 0x7fd6d4153ba0 2023-10-30T13:35:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.972+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d404e170 con 0x7fd6d4153ba0 2023-10-30T13:35:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.972+0000 7fd6bbfff640 1 -- 172.21.15.31:0/117800766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6cc014c90 con 0x7fd6d4153ba0 2023-10-30T13:35:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.972+0000 7fd6bbfff640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6c00664d0 0x7fd6c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.972+0000 7fd6bbfff640 1 -- 172.21.15.31:0/117800766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd6cc084670 con 0x7fd6d4153ba0 2023-10-30T13:35:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.972+0000 7fd6d37fe640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6c00664d0 0x7fd6c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.972+0000 7fd6d37fe640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6c00664d0 0x7fd6c0068990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40af000 tx=0x7fd6c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:00.976+0000 7fd6bbfff640 1 -- 172.21.15.31:0/117800766 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6cc051d10 con 0x7fd6d4153ba0 2023-10-30T13:35:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6d415c3d0 con 0x7fd6d4153ba0 2023-10-30T13:35:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6bbfff640 1 -- 172.21.15.31:0/117800766 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6cc055330 con 0x7fd6d4153ba0 2023-10-30T13:35:01.137 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:35:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6c00664d0 msgr2=0x7fd6c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6c00664d0 0x7fd6c0068990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40af000 tx=0x7fd6c4008040 comp rx=0 tx=0).stop 2023-10-30T13:35:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 msgr2=0x7fd6d40c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40c8a10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc007e60 tx=0x7fd6cc00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 shutdown_connections 2023-10-30T13:35:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6c00664d0 0x7fd6c0068990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6d4153ba0 0x7fd6d40c8a10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6d40c4810 0x7fd6d415c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 --2- 172.21.15.31:0/117800766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d40c4410 0x7fd6d40c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.132+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 >> 172.21.15.31:0/117800766 conn(0x7fd6d4071790 msgr2=0x7fd6d40b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.136+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 shutdown_connections 2023-10-30T13:35:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.136+0000 7fd6da10f640 1 -- 172.21.15.31:0/117800766 wait complete. 2023-10-30T13:35:01.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:01.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4c491640 1 Processor -- start 2023-10-30T13:35:01.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4c491640 1 -- start start 2023-10-30T13:35:01.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4c491640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f9d44153c80 0x7f9d4415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4c491640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d44159f20 con 0x7f9d440c42a0 2023-10-30T13:35:01.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4c491640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4415d620 con 0x7f9d440c4a10 2023-10-30T13:35:01.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4c491640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4415dec0 con 0x7f9d44153c80 2023-10-30T13:35:01.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4aa2e640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f9d440c42a0 0x7f9d440ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54778/0 (socket says 172.21.15.31:54778) 2023-10-30T13:35:01.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4aa2e640 1 -- 172.21.15.31:0/3054163709 learned_addr learned my addr 172.21.15.31:0/3054163709 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:01.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.296+0000 7f9d4a22d640 1 --2- 172.21.15.31:0/3054163709 >> v2:172.21.15.33:3300/0 conn(0x7f9d44153c80 0x7f9d4415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305724291 0 0) 0x7f9d44159f20 con 0x7f9d440c42a0 2023-10-30T13:35:01.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d30003530 con 0x7f9d440c42a0 2023-10-30T13:35:01.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991775998 0 0) 0x7f9d30003530 con 0x7f9d440c42a0 2023-10-30T13:35:01.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 >> v2:172.21.15.33:3300/0 conn(0x7f9d44153c80 msgr2=0x7f9d4415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:01.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 --2- 172.21.15.31:0/3054163709 >> v2:172.21.15.33:3300/0 conn(0x7f9d44153c80 0x7f9d4415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 >> v1:172.21.15.33:6789/0 conn(0x7f9d440c4a10 legacy=0x7f9d4415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4415d900 con 0x7f9d440c42a0 2023-10-30T13:35:01.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f9d3c001a00 con 0x7f9d440c42a0 2023-10-30T13:35:01.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 >> v1:172.21.15.31:6789/0 conn(0x7f9d440c42a0 legacy=0x7f9d440ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d30003980 0x7f9d30003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d300047d0 0x7f9d30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d30005190 0x7f9d3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4415dec0 con 0x7f9d300047d0 2023-10-30T13:35:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3000c040 con 0x7f9d30005190 2023-10-30T13:35:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3000c200 con 0x7f9d30003980 2023-10-30T13:35:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d49a2c640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d300047d0 0x7f9d30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4a22d640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d30003980 0x7f9d30003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4aa2e640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d30005190 0x7f9d3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d49a2c640 1 -- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d30003980 msgr2=0x7f9d30003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d49a2c640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d30003980 0x7f9d30003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d49a2c640 1 -- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d30005190 msgr2=0x7f9d3000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d49a2c640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d30005190 0x7f9d3000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d49a2c640 1 -- 172.21.15.31:0/3054163709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30003530 con 0x7f9d300047d0 2023-10-30T13:35:01.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.300+0000 7f9d4aa2e640 1 -- 172.21.15.31:0/3054163709 reap_dead start 2023-10-30T13:35:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d49a2c640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d300047d0 0x7f9d30004c50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001750 tx=0x7f9d3401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727e9cba62ce7b19 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340052d0 con 0x7f9d300047d0 2023-10-30T13:35:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d340a7050 con 0x7f9d300047d0 2023-10-30T13:35:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4922b640 1 -- 172.21.15.31:0/3054163709 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34003600 con 0x7f9d300047d0 2023-10-30T13:35:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- 172.21.15.31:0/3054163709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d300047d0 msgr2=0x7f9d30004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d300047d0 0x7f9d30004c50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001750 tx=0x7f9d3401c3a0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- 172.21.15.31:0/3054163709 shutdown_connections 2023-10-30T13:35:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 --2- 172.21.15.31:0/3054163709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d300047d0 0x7f9d30004c50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- 172.21.15.31:0/3054163709 >> 172.21.15.31:0/3054163709 conn(0x7f9d440715c0 msgr2=0x7f9d440719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- 172.21.15.31:0/3054163709 shutdown_connections 2023-10-30T13:35:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- 172.21.15.31:0/3054163709 wait complete. 2023-10-30T13:35:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 Processor -- start 2023-10-30T13:35:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- start start 2023-10-30T13:35:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d300047d0 0x7f9d440b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d440b5110 0x7f9d4415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d441f2f20 0x7f9d441f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44154310 con 0x7f9d441f2f20 2023-10-30T13:35:01.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441f2a90 con 0x7f9d300047d0 2023-10-30T13:35:01.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.304+0000 7f9d4c491640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441f38e0 con 0x7f9d440b5110 2023-10-30T13:35:01.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d441f2f20 0x7f9d441f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d441f2f20 0x7f9d441f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42542/0 (socket says 172.21.15.31:42542) 2023-10-30T13:35:01.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 -- 172.21.15.31:0/1855666520 learned_addr learned my addr 172.21.15.31:0/1855666520 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4a22d640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d300047d0 0x7f9d440b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d49a2c640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d440b5110 0x7f9d4415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 -- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d440b5110 msgr2=0x7f9d4415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d440b5110 0x7f9d4415c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 -- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d300047d0 msgr2=0x7f9d440b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d300047d0 0x7f9d440b2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 -- 172.21.15.31:0/1855666520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44154540 con 0x7f9d441f2f20 2023-10-30T13:35:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d49a2c640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d440b5110 0x7f9d4415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4aa2e640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d441f2f20 0x7f9d441f33a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9d44072840 tx=0x7f9d3c007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d3affd640 1 -- 172.21.15.31:0/1855666520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c00eb00 con 0x7f9d441f2f20 2023-10-30T13:35:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d3affd640 1 -- 172.21.15.31:0/1855666520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c00ec60 con 0x7f9d441f2f20 2023-10-30T13:35:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d44154820 con 0x7f9d441f2f20 2023-10-30T13:35:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d3affd640 1 -- 172.21.15.31:0/1855666520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c014430 con 0x7f9d441f2f20 2023-10-30T13:35:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d44159550 con 0x7f9d441f2f20 2023-10-30T13:35:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d441f2c40 con 0x7f9d441f2f20 2023-10-30T13:35:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d3affd640 1 -- 172.21.15.31:0/1855666520 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9d3c014590 con 0x7f9d441f2f20 2023-10-30T13:35:01.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d3affd640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9d280664a0 0x7f9d28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.308+0000 7f9d3affd640 1 -- 172.21.15.31:0/1855666520 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9d3c083960 con 0x7f9d441f2f20 2023-10-30T13:35:01.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.312+0000 7f9d4a22d640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9d280664a0 0x7f9d28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.312+0000 7f9d4a22d640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9d280664a0 0x7f9d28068960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d44063a60 tx=0x7f9d40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:01.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.312+0000 7f9d3affd640 1 -- 172.21.15.31:0/1855666520 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9d441f2c40 con 0x7f9d441f2f20 2023-10-30T13:35:01.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d441f2c40 con 0x7f9d441f2f20 2023-10-30T13:35:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d3affd640 1 -- 172.21.15.31:0/1855666520 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f9d441f2c40 con 0x7f9d441f2f20 2023-10-30T13:35:01.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:01.509 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T13:35:01.509 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2023-10-30T13:35:01.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9d280664a0 msgr2=0x7f9d28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9d280664a0 0x7f9d28068960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d44063a60 tx=0x7f9d40008040 comp rx=0 tx=0).stop 2023-10-30T13:35:01.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d441f2f20 msgr2=0x7f9d441f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d441f2f20 0x7f9d441f33a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9d44072840 tx=0x7f9d3c007bc0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 shutdown_connections 2023-10-30T13:35:01.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9d280664a0 0x7f9d28068960 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d441f2f20 0x7f9d441f33a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9d440b5110 0x7f9d4415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 --2- 172.21.15.31:0/1855666520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d300047d0 0x7f9d440b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 >> 172.21.15.31:0/1855666520 conn(0x7f9d440715c0 msgr2=0x7f9d440cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:01.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 shutdown_connections 2023-10-30T13:35:01.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:01.504+0000 7f9d4c491640 1 -- 172.21.15.31:0/1855666520 wait complete. 2023-10-30T13:35:03.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:03.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc9c09640 1 Processor -- start 2023-10-30T13:35:03.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc9c09640 1 -- start start 2023-10-30T13:35:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc9c09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc9c09640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc4159f20 con 0x7f5dc40c42a0 2023-10-30T13:35:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc9c09640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc415d620 con 0x7f5dc40c4a10 2023-10-30T13:35:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc9c09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc415dec0 con 0x7f5dc4153c80 2023-10-30T13:35:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc3fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f5dc40c42a0 0x7f5dc40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54792/0 (socket says 172.21.15.31:54792) 2023-10-30T13:35:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc3fff640 1 -- 172.21.15.31:0/2017015249 learned_addr learned my addr 172.21.15.31:0/2017015249 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc37fe640 1 --2- 172.21.15.31:0/2017015249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc27fc640 1 -- 172.21.15.31:0/2017015249 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684417758 0 0) 0x7f5dc4159f20 con 0x7f5dc40c42a0 2023-10-30T13:35:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc27fc640 1 -- 172.21.15.31:0/2017015249 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac003530 con 0x7f5dc40c42a0 2023-10-30T13:35:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc27fc640 1 -- 172.21.15.31:0/2017015249 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1108053141 0 0) 0x7f5dac003530 con 0x7f5dc40c42a0 2023-10-30T13:35:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc37fe640 1 -- 172.21.15.31:0/2017015249 >> v1:172.21.15.33:6789/0 conn(0x7f5dc40c4a10 legacy=0x7f5dc415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc37fe640 1 -- 172.21.15.31:0/2017015249 >> v1:172.21.15.31:6789/0 conn(0x7f5dc40c42a0 legacy=0x7f5dc40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.680+0000 7f5dc37fe640 1 -- 172.21.15.31:0/2017015249 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc415d900 con 0x7f5dc4153c80 2023-10-30T13:35:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc37fe640 1 --2- 172.21.15.31:0/2017015249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc415cf90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5db40097f0 tx=0x7f5db401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4485ffe460aef0ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc27fc640 1 -- 172.21.15.31:0/2017015249 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc001a00 con 0x7f5dc4153c80 2023-10-30T13:35:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc27fc640 1 -- 172.21.15.31:0/2017015249 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db40a7050 con 0x7f5dc4153c80 2023-10-30T13:35:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc27fc640 1 -- 172.21.15.31:0/2017015249 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db40ab600 con 0x7f5dc4153c80 2023-10-30T13:35:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc9c09640 1 -- 172.21.15.31:0/2017015249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 msgr2=0x7f5dc415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/2017015249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc415cf90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5db40097f0 tx=0x7f5db401c320 comp rx=0 tx=0).stop 2023-10-30T13:35:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc9c09640 1 -- 172.21.15.31:0/2017015249 shutdown_connections 2023-10-30T13:35:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/2017015249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc415cf90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc9c09640 1 -- 172.21.15.31:0/2017015249 >> 172.21.15.31:0/2017015249 conn(0x7f5dc40715c0 msgr2=0x7f5dc40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc9c09640 1 -- 172.21.15.31:0/2017015249 shutdown_connections 2023-10-30T13:35:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.684+0000 7f5dc9c09640 1 -- 172.21.15.31:0/2017015249 wait complete. 2023-10-30T13:35:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 Processor -- start 2023-10-30T13:35:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 -- start start 2023-10-30T13:35:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc40c42a0 0x7f5dc4154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5dc40c4a10 0x7f5dc414f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc4157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc4157670 con 0x7f5dc40c4a10 2023-10-30T13:35:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc4154410 con 0x7f5dc40c42a0 2023-10-30T13:35:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc9c09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc4155270 con 0x7f5dc4153c80 2023-10-30T13:35:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5dc40c4a10 0x7f5dc414f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5dc40c4a10 0x7f5dc414f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42554/0 (socket says 172.21.15.31:42554) 2023-10-30T13:35:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 -- 172.21.15.31:0/3825273168 learned_addr learned my addr 172.21.15.31:0/3825273168 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc3fff640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc4157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc37fe640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc40c42a0 0x7f5dc4154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 -- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 msgr2=0x7f5dc4157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc4157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 -- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc40c42a0 msgr2=0x7f5dc4154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc40c42a0 0x7f5dc4154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 -- 172.21.15.31:0/3825273168 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db4009420 con 0x7f5dc40c4a10 2023-10-30T13:35:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc37fe640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc40c42a0 0x7f5dc4154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.688+0000 7f5dc2ffd640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5dc40c4a10 0x7f5dc414f780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009e30 tx=0x7f5db800c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dabfff640 1 -- 172.21.15.31:0/3825273168 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db800cdb0 con 0x7f5dc40c4a10 2023-10-30T13:35:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dabfff640 1 -- 172.21.15.31:0/3825273168 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db8004020 con 0x7f5dc40c4a10 2023-10-30T13:35:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc415ab60 con 0x7f5dc40c4a10 2023-10-30T13:35:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dabfff640 1 -- 172.21.15.31:0/3825273168 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db80043d0 con 0x7f5dc40c4a10 2023-10-30T13:35:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc41f2e20 con 0x7f5dc40c4a10 2023-10-30T13:35:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc404e170 con 0x7f5dc40c4a10 2023-10-30T13:35:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dabfff640 1 -- 172.21.15.31:0/3825273168 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5db8003600 con 0x7f5dc40c4a10 2023-10-30T13:35:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dabfff640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5db00664a0 0x7f5db0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dabfff640 1 -- 172.21.15.31:0/3825273168 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5db8085df0 con 0x7f5dc40c4a10 2023-10-30T13:35:03.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.692+0000 7f5dc37fe640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5db00664a0 0x7f5db0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:03.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.696+0000 7f5dc37fe640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5db00664a0 0x7f5db0068960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5db40ad000 tx=0x7f5db40a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:03.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.696+0000 7f5dabfff640 1 -- 172.21.15.31:0/3825273168 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5db8053520 con 0x7f5dc40c4a10 2023-10-30T13:35:03.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.884+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc414dd10 con 0x7f5dc40c4a10 2023-10-30T13:35:03.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dabfff640 1 -- 172.21.15.31:0/3825273168 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5271 (secure 0 0 0) 0x7f5db8018020 con 0x7f5dc40c4a10 2023-10-30T13:35:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:03.892 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T13:35:03.892 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2023-10-30T13:35:03.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5db00664a0 msgr2=0x7f5db0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:03.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5db00664a0 0x7f5db0068960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5db40ad000 tx=0x7f5db40a6000 comp rx=0 tx=0).stop 2023-10-30T13:35:03.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5dc40c4a10 msgr2=0x7f5dc414f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:03.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5dc40c4a10 0x7f5dc414f780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009e30 tx=0x7f5db800c430 comp rx=0 tx=0).stop 2023-10-30T13:35:03.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 shutdown_connections 2023-10-30T13:35:03.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5db00664a0 0x7f5db0068960 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:03.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5dc4153c80 0x7f5dc4157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:03.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5dc40c4a10 0x7f5dc414f780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:03.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 --2- 172.21.15.31:0/3825273168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc40c42a0 0x7f5dc4154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:03.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 >> 172.21.15.31:0/3825273168 conn(0x7f5dc40715c0 msgr2=0x7f5dc40c5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:03.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 shutdown_connections 2023-10-30T13:35:03.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:03.888+0000 7f5dc9c09640 1 -- 172.21.15.31:0/3825273168 wait complete. 2023-10-30T13:35:03.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:04.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6707cc9640 1 Processor -- start 2023-10-30T13:35:04.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6707cc9640 1 -- start start 2023-10-30T13:35:04.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6707cc9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151c20 0x7f67000c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6707cc9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f67000cb8b0 0x7f670006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6707cc9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000c5030 con 0x7f6700151840 2023-10-30T13:35:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6707cc9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6700157830 con 0x7f67000cb8b0 2023-10-30T13:35:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6707cc9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6700157b10 con 0x7f6700151c20 2023-10-30T13:35:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6706266640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f67000cb8b0 0x7f670006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6706266640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f67000cb8b0 0x7f670006b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42556/0 (socket says 172.21.15.31:42556) 2023-10-30T13:35:04.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6706266640 1 -- 172.21.15.31:0/370970631 learned_addr learned my addr 172.21.15.31:0/370970631 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:04.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6705264640 1 --2- 172.21.15.31:0/370970631 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151c20 0x7f67000c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6706266640 1 -- 172.21.15.31:0/370970631 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151c20 msgr2=0x7f67000c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:04.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6706266640 1 --2- 172.21.15.31:0/370970631 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151c20 0x7f67000c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6706266640 1 -- 172.21.15.31:0/370970631 >> v1:172.21.15.33:6789/0 conn(0x7f6700151840 legacy=0x7f6700072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:04.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.052+0000 7f6706266640 1 -- 172.21.15.31:0/370970631 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6700157df0 con 0x7f67000cb8b0 2023-10-30T13:35:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6706266640 1 --2- 172.21.15.31:0/370970631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67000cb8b0 0x7f670006b600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009e50 tx=0x7f66f801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b7b62ba955c32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6704a63640 1 -- 172.21.15.31:0/370970631 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f80af400 con 0x7f67000cb8b0 2023-10-30T13:35:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6704a63640 1 -- 172.21.15.31:0/370970631 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f80ab050 con 0x7f67000cb8b0 2023-10-30T13:35:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6704a63640 1 -- 172.21.15.31:0/370970631 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f80af780 con 0x7f67000cb8b0 2023-10-30T13:35:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 -- 172.21.15.31:0/370970631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67000cb8b0 msgr2=0x7f670006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 --2- 172.21.15.31:0/370970631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67000cb8b0 0x7f670006b600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009e50 tx=0x7f66f801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 -- 172.21.15.31:0/370970631 shutdown_connections 2023-10-30T13:35:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 --2- 172.21.15.31:0/370970631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67000cb8b0 0x7f670006b600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 --2- 172.21.15.31:0/370970631 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151c20 0x7f67000c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 -- 172.21.15.31:0/370970631 >> 172.21.15.31:0/370970631 conn(0x7f67000cac70 msgr2=0x7f67000cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 -- 172.21.15.31:0/370970631 shutdown_connections 2023-10-30T13:35:04.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.056+0000 7f6707cc9640 1 -- 172.21.15.31:0/370970631 wait complete. 2023-10-30T13:35:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 Processor -- start 2023-10-30T13:35:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 -- start start 2023-10-30T13:35:04.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67000cb8b0 0x7f67001f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151840 0x7f67000c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6700151c20 0x7f67001553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6700157130 con 0x7f6700151c20 2023-10-30T13:35:04.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67001de1e0 con 0x7f67000cb8b0 2023-10-30T13:35:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67001f31a0 con 0x7f6700151840 2023-10-30T13:35:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6700151c20 0x7f67001553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6700151c20 0x7f67001553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42560/0 (socket says 172.21.15.31:42560) 2023-10-30T13:35:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 -- 172.21.15.31:0/3399843020 learned_addr learned my addr 172.21.15.31:0/3399843020 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6705a65640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67000cb8b0 0x7f67001f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6705264640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151840 0x7f67000c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 -- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151840 msgr2=0x7f67000c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151840 0x7f67000c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 -- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67000cb8b0 msgr2=0x7f67001f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67000cb8b0 0x7f67001f2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 -- 172.21.15.31:0/3399843020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f8009b30 con 0x7f6700151c20 2023-10-30T13:35:04.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6706266640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6700151c20 0x7f67001553e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009e50 tx=0x7f66f801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f66f67fc640 1 -- 172.21.15.31:0/3399843020 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f80afdc0 con 0x7f6700151c20 2023-10-30T13:35:04.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f66f67fc640 1 -- 172.21.15.31:0/3399843020 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8009510 con 0x7f6700151c20 2023-10-30T13:35:04.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f66f67fc640 1 -- 172.21.15.31:0/3399843020 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f80ba660 con 0x7f6700151c20 2023-10-30T13:35:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67001eece0 con 0x7f6700151c20 2023-10-30T13:35:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67001eefa0 con 0x7f6700151c20 2023-10-30T13:35:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.060+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670004e170 con 0x7f6700151c20 2023-10-30T13:35:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.064+0000 7f66f67fc640 1 -- 172.21.15.31:0/3399843020 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f66f80b6040 con 0x7f6700151c20 2023-10-30T13:35:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.064+0000 7f66f67fc640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66e40664d0 0x7f66e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.064+0000 7f66f67fc640 1 -- 172.21.15.31:0/3399843020 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f66f8121e20 con 0x7f6700151c20 2023-10-30T13:35:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.064+0000 7f6705a65640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66e40664d0 0x7f66e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.064+0000 7f6705a65640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66e40664d0 0x7f66e4068990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f66fc004e30 tx=0x7f66fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.068+0000 7f66f67fc640 1 -- 172.21.15.31:0/3399843020 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f66f80ef670 con 0x7f6700151c20 2023-10-30T13:35:04.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.228+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6700066510 con 0x7f6700151c20 2023-10-30T13:35:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.228+0000 7f66f67fc640 1 -- 172.21.15.31:0/3399843020 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12790 (secure 0 0 0) 0x7f66f80f2c90 con 0x7f6700151c20 2023-10-30T13:35:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:04.234 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:34:55.560761+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66e40664d0 msgr2=0x7f66e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66e40664d0 0x7f66e4068990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f66fc004e30 tx=0x7f66fc009040 comp rx=0 tx=0).stop 2023-10-30T13:35:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6700151c20 msgr2=0x7f67001553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6700151c20 0x7f67001553e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009e50 tx=0x7f66f801bf60 comp rx=0 tx=0).stop 2023-10-30T13:35:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 shutdown_connections 2023-10-30T13:35:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66e40664d0 0x7f66e4068990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6700151c20 0x7f67001553e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6700151840 0x7f67000c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 --2- 172.21.15.31:0/3399843020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67000cb8b0 0x7f67001f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 >> 172.21.15.31:0/3399843020 conn(0x7f67000cac70 msgr2=0x7f67001e0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 shutdown_connections 2023-10-30T13:35:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.232+0000 7f6707cc9640 1 -- 172.21.15.31:0/3399843020 wait complete. 2023-10-30T13:35:04.255 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:35:04.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:35:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.396+0000 7f264ee46640 1 Processor -- start 2023-10-30T13:35:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.396+0000 7f264ee46640 1 -- start start 2023-10-30T13:35:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.396+0000 7f264ee46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f264806afa0 0x7f264815a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.396+0000 7f264ee46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f264815a5b0 0x7f26480cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.396+0000 7f264ee46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264806f830 con 0x7f2648156f20 2023-10-30T13:35:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.396+0000 7f264ee46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264806c200 con 0x7f264815a5b0 2023-10-30T13:35:04.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.396+0000 7f264ee46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264806b5e0 con 0x7f264806afa0 2023-10-30T13:35:04.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f264815a5b0 0x7f26480cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f264815a5b0 0x7f26480cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42570/0 (socket says 172.21.15.31:42570) 2023-10-30T13:35:04.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 -- 172.21.15.31:0/4267919639 learned_addr learned my addr 172.21.15.31:0/4267919639 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:04.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f2647fff640 1 --2- 172.21.15.31:0/4267919639 >> v2:172.21.15.33:3300/0 conn(0x7f264806afa0 0x7f264815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 -- 172.21.15.31:0/4267919639 >> v2:172.21.15.33:3300/0 conn(0x7f264806afa0 msgr2=0x7f264815a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:04.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 --2- 172.21.15.31:0/4267919639 >> v2:172.21.15.33:3300/0 conn(0x7f264806afa0 0x7f264815a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 -- 172.21.15.31:0/4267919639 >> v1:172.21.15.33:6789/0 conn(0x7f2648156f20 legacy=0x7f264806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 -- 172.21.15.31:0/4267919639 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2648159920 con 0x7f264815a5b0 2023-10-30T13:35:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f264d3e3640 1 --2- 172.21.15.31:0/4267919639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f264815a5b0 0x7f26480cce30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f264806beb0 tx=0x7f264001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54519c7da2acd1dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f26477fe640 1 -- 172.21.15.31:0/4267919639 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26400a57a0 con 0x7f264815a5b0 2023-10-30T13:35:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f26477fe640 1 -- 172.21.15.31:0/4267919639 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26400a5900 con 0x7f264815a5b0 2023-10-30T13:35:04.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.400+0000 7f26477fe640 1 -- 172.21.15.31:0/4267919639 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26400a5c90 con 0x7f264815a5b0 2023-10-30T13:35:04.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- 172.21.15.31:0/4267919639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f264815a5b0 msgr2=0x7f26480cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 --2- 172.21.15.31:0/4267919639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f264815a5b0 0x7f26480cce30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f264806beb0 tx=0x7f264001c9d0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- 172.21.15.31:0/4267919639 shutdown_connections 2023-10-30T13:35:04.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 --2- 172.21.15.31:0/4267919639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f264815a5b0 0x7f26480cce30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 --2- 172.21.15.31:0/4267919639 >> v2:172.21.15.33:3300/0 conn(0x7f264806afa0 0x7f264815a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- 172.21.15.31:0/4267919639 >> 172.21.15.31:0/4267919639 conn(0x7f2648065a80 msgr2=0x7f2648065e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- 172.21.15.31:0/4267919639 shutdown_connections 2023-10-30T13:35:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- 172.21.15.31:0/4267919639 wait complete. 2023-10-30T13:35:04.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 Processor -- start 2023-10-30T13:35:04.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- start start 2023-10-30T13:35:04.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f264806afa0 0x7f2648155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2648156f20 0x7f26480cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f264815a5b0 0x7f26481f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26481f1d90 con 0x7f2648156f20 2023-10-30T13:35:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26481f5430 con 0x7f264815a5b0 2023-10-30T13:35:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264ee46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264814ecf0 con 0x7f264806afa0 2023-10-30T13:35:04.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2648156f20 0x7f26480cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2648156f20 0x7f26480cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42572/0 (socket says 172.21.15.31:42572) 2023-10-30T13:35:04.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 -- 172.21.15.31:0/1898938884 learned_addr learned my addr 172.21.15.31:0/1898938884 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:04.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264cbe2640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f264806afa0 0x7f2648155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f264d3e3640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f264815a5b0 0x7f26481f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 -- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f264806afa0 msgr2=0x7f2648155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f264806afa0 0x7f2648155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 -- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f264815a5b0 msgr2=0x7f26481f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f264815a5b0 0x7f26481f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.404+0000 7f2647fff640 1 -- 172.21.15.31:0/1898938884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2640009b30 con 0x7f2648156f20 2023-10-30T13:35:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f2647fff640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2648156f20 0x7f26480cc3e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f263400d800 tx=0x7f263400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f26457fa640 1 -- 172.21.15.31:0/1898938884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263400b6c0 con 0x7f2648156f20 2023-10-30T13:35:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f26457fa640 1 -- 172.21.15.31:0/1898938884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263400b820 con 0x7f2648156f20 2023-10-30T13:35:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f26457fa640 1 -- 172.21.15.31:0/1898938884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263400bb90 con 0x7f2648156f20 2023-10-30T13:35:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26481f56d0 con 0x7f2648156f20 2023-10-30T13:35:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26481f5c90 con 0x7f2648156f20 2023-10-30T13:35:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26481f1f00 con 0x7f2648156f20 2023-10-30T13:35:04.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f26457fa640 1 -- 172.21.15.31:0/1898938884 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f263400bcf0 con 0x7f2648156f20 2023-10-30T13:35:04.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f26457fa640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f262c0664f0 0x7f262c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f26457fa640 1 -- 172.21.15.31:0/1898938884 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2634083f60 con 0x7f2648156f20 2023-10-30T13:35:04.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.408+0000 7f264cbe2640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f262c0664f0 0x7f262c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.412+0000 7f264cbe2640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f262c0664f0 0x7f262c0689b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f263c007fd0 tx=0x7f263c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.412+0000 7f26457fa640 1 -- 172.21.15.31:0/1898938884 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26481f1f00 con 0x7f2648156f20 2023-10-30T13:35:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26481f1ed0 con 0x7f2648156f20 2023-10-30T13:35:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f26457fa640 1 -- 172.21.15.31:0/1898938884 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2634055220 con 0x7f2648156f20 2023-10-30T13:35:04.576 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:35:04.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f262c0664f0 msgr2=0x7f262c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f262c0664f0 0x7f262c0689b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f263c007fd0 tx=0x7f263c009040 comp rx=0 tx=0).stop 2023-10-30T13:35:04.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2648156f20 msgr2=0x7f26480cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2648156f20 0x7f26480cc3e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f263400d800 tx=0x7f263400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 shutdown_connections 2023-10-30T13:35:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f262c0664f0 0x7f262c0689b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f264815a5b0 0x7f26481f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2648156f20 0x7f26480cc3e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 --2- 172.21.15.31:0/1898938884 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f264806afa0 0x7f2648155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 >> 172.21.15.31:0/1898938884 conn(0x7f2648065a80 msgr2=0x7f2648065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 shutdown_connections 2023-10-30T13:35:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.572+0000 7f264ee46640 1 -- 172.21.15.31:0/1898938884 wait complete. 2023-10-30T13:35:04.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-30T13:35:04.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc973e640 1 Processor -- start 2023-10-30T13:35:04.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc973e640 1 -- start start 2023-10-30T13:35:04.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc973e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5fc415cf90 0x7f5fc40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc973e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc4159f20 con 0x7f5fc40c4a10 2023-10-30T13:35:04.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc973e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc40c47e0 con 0x7f5fc4153c80 2023-10-30T13:35:04.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc973e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc415dec0 con 0x7f5fc415cf90 2023-10-30T13:35:04.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc37fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5fc415cf90 0x7f5fc40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fbbfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f5fc40c4a10 0x7f5fc40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45024/0 (socket says 172.21.15.31:45024) 2023-10-30T13:35:04.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fbbfff640 1 -- 172.21.15.31:0/788775296 learned_addr learned my addr 172.21.15.31:0/788775296 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:04.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc37fe640 1 -- 172.21.15.31:0/788775296 >> v1:172.21.15.33:6789/0 conn(0x7f5fc4153c80 legacy=0x7f5fc415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc37fe640 1 -- 172.21.15.31:0/788775296 >> v1:172.21.15.31:6789/0 conn(0x7f5fc40c4a10 legacy=0x7f5fc40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc37fe640 1 -- 172.21.15.31:0/788775296 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc415d620 con 0x7f5fc415cf90 2023-10-30T13:35:04.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.736+0000 7f5fc27fc640 1 -- 172.21.15.31:0/788775296 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425079474 0 0) 0x7f5fc4159f20 con 0x7f5fc40c4a10 2023-10-30T13:35:04.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc37fe640 1 --2- 172.21.15.31:0/788775296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc415cf90 0x7f5fc40c42a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009d90 tx=0x7f5fbc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d2679b03d90c86 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc27fc640 1 -- 172.21.15.31:0/788775296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0af600 con 0x7f5fc415cf90 2023-10-30T13:35:04.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc27fc640 1 -- 172.21.15.31:0/788775296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc0ab050 con 0x7f5fc415cf90 2023-10-30T13:35:04.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc27fc640 1 -- 172.21.15.31:0/788775296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0af980 con 0x7f5fc415cf90 2023-10-30T13:35:04.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 -- 172.21.15.31:0/788775296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc415cf90 msgr2=0x7f5fc40c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 --2- 172.21.15.31:0/788775296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc415cf90 0x7f5fc40c42a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009d90 tx=0x7f5fbc01f130 comp rx=0 tx=0).stop 2023-10-30T13:35:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 -- 172.21.15.31:0/788775296 shutdown_connections 2023-10-30T13:35:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 --2- 172.21.15.31:0/788775296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc415cf90 0x7f5fc40c42a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 -- 172.21.15.31:0/788775296 >> 172.21.15.31:0/788775296 conn(0x7f5fc40715c0 msgr2=0x7f5fc40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 -- 172.21.15.31:0/788775296 shutdown_connections 2023-10-30T13:35:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 -- 172.21.15.31:0/788775296 wait complete. 2023-10-30T13:35:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 Processor -- start 2023-10-30T13:35:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.740+0000 7f5fc973e640 1 -- start start 2023-10-30T13:35:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc40c4a10 0x7f5fc40b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5fc4153c80 0x7f5fc4148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fc415cf90 0x7f5fc41eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc41efa50 con 0x7f5fc40c4a10 2023-10-30T13:35:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40b2020 con 0x7f5fc415cf90 2023-10-30T13:35:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40b4df0 con 0x7f5fc4153c80 2023-10-30T13:35:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc40c4a10 0x7f5fc40b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc40c4a10 0x7f5fc40b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33260/0 (socket says 172.21.15.31:33260) 2023-10-30T13:35:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 -- 172.21.15.31:0/204200345 learned_addr learned my addr 172.21.15.31:0/204200345 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fbbfff640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5fc4153c80 0x7f5fc4148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc37fe640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fc415cf90 0x7f5fc41eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 -- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5fc4153c80 msgr2=0x7f5fc4148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5fc4153c80 0x7f5fc4148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 -- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fc415cf90 msgr2=0x7f5fc41eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fc415cf90 0x7f5fc41eea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 -- 172.21.15.31:0/204200345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc009a70 con 0x7f5fc40c4a10 2023-10-30T13:35:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fbbfff640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5fc4153c80 0x7f5fc4148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc2ffd640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc40c4a10 0x7f5fc40b3410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400c500 tx=0x7f5fb400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fbb7fe640 1 -- 172.21.15.31:0/204200345 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4005220 con 0x7f5fc40c4a10 2023-10-30T13:35:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fbb7fe640 1 -- 172.21.15.31:0/204200345 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4005380 con 0x7f5fc40c4a10 2023-10-30T13:35:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc40c5cc0 con 0x7f5fc40c4a10 2023-10-30T13:35:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fbb7fe640 1 -- 172.21.15.31:0/204200345 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4007270 con 0x7f5fc40c4a10 2023-10-30T13:35:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc40b2270 con 0x7f5fc40c4a10 2023-10-30T13:35:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc404e170 con 0x7f5fc40c4a10 2023-10-30T13:35:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.744+0000 7f5fbb7fe640 1 -- 172.21.15.31:0/204200345 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5fb40073d0 con 0x7f5fc40c4a10 2023-10-30T13:35:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.748+0000 7f5fbb7fe640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5fa40664a0 0x7f5fa4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.748+0000 7f5fbb7fe640 1 -- 172.21.15.31:0/204200345 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5fb4082f50 con 0x7f5fc40c4a10 2023-10-30T13:35:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.748+0000 7f5fbbfff640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5fa40664a0 0x7f5fa4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.748+0000 7f5fbbfff640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5fa40664a0 0x7f5fa4068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fc414dcc0 tx=0x7f5fac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.752+0000 7f5fbb7fe640 1 -- 172.21.15.31:0/204200345 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5fb4050690 con 0x7f5fc40c4a10 2023-10-30T13:35:04.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.908+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5fc40cad40 con 0x7f5fc40c4a10 2023-10-30T13:35:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.908+0000 7f5fbb7fe640 1 -- 172.21.15.31:0/204200345 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5fb4053cb0 con 0x7f5fc40c4a10 2023-10-30T13:35:04.913 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:04.913 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2023-10-30T13:35:04.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-10-30T13:35:04.914 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2023-10-30T13:35:04.914 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2023-10-30T13:35:04.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-10-30T13:35:04.914 INFO:teuthology.orchestra.run.smithi031.stdout:] 2023-10-30T13:35:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5fa40664a0 msgr2=0x7f5fa4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5fa40664a0 0x7f5fa4068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fc414dcc0 tx=0x7f5fac009040 comp rx=0 tx=0).stop 2023-10-30T13:35:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc40c4a10 msgr2=0x7f5fc40b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc40c4a10 0x7f5fc40b3410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400c500 tx=0x7f5fb400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 shutdown_connections 2023-10-30T13:35:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5fa40664a0 0x7f5fa4068960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fc415cf90 0x7f5fc41eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5fc4153c80 0x7f5fc4148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 --2- 172.21.15.31:0/204200345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fc40c4a10 0x7f5fc40b3410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 >> 172.21.15.31:0/204200345 conn(0x7f5fc40715c0 msgr2=0x7f5fc4151640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 shutdown_connections 2023-10-30T13:35:04.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:04.912+0000 7f5fc973e640 1 -- 172.21.15.31:0/204200345 wait complete. 2023-10-30T13:35:04.934 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-30T13:35:04.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-30T13:35:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.076+0000 7f7d0f7c5640 1 Processor -- start 2023-10-30T13:35:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.076+0000 7f7d0f7c5640 1 -- start start 2023-10-30T13:35:05.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0f7c5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7d0806fb20 0x7f7d080c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0f7c5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0815ded0 con 0x7f7d08153bd0 2023-10-30T13:35:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0f7c5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0806b130 con 0x7f7d0806ff00 2023-10-30T13:35:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0f7c5640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0806ee00 con 0x7f7d0806fb20 2023-10-30T13:35:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0d561640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7d0806fb20 0x7f7d080c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0d561640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7d0806fb20 0x7f7d080c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33274/0 (socket says 172.21.15.31:33274) 2023-10-30T13:35:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0dd62640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7d08153bd0 0x7f7d080c20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45034/0 (socket says 172.21.15.31:45034) 2023-10-30T13:35:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0d561640 1 -- 172.21.15.31:0/2951212290 learned_addr learned my addr 172.21.15.31:0/2951212290 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0d561640 1 -- 172.21.15.31:0/2951212290 >> v1:172.21.15.33:6789/0 conn(0x7f7d0806ff00 legacy=0x7f7d081534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0d561640 1 -- 172.21.15.31:0/2951212290 >> v1:172.21.15.31:6789/0 conn(0x7f7d08153bd0 legacy=0x7f7d080c20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.080+0000 7f7d0d561640 1 -- 172.21.15.31:0/2951212290 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0806cff0 con 0x7f7d0806fb20 2023-10-30T13:35:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0d561640 1 --2- 172.21.15.31:0/2951212290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d0806fb20 0x7f7d080c7990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009870 tx=0x7f7d0401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a316c63d4c09391c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7cfffff640 1 -- 172.21.15.31:0/2951212290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040ab600 con 0x7f7d0806fb20 2023-10-30T13:35:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7cfffff640 1 -- 172.21.15.31:0/2951212290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d040a7050 con 0x7f7d0806fb20 2023-10-30T13:35:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7cfffff640 1 -- 172.21.15.31:0/2951212290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040ab980 con 0x7f7d0806fb20 2023-10-30T13:35:05.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2951212290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d0806fb20 msgr2=0x7f7d080c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2951212290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d0806fb20 0x7f7d080c7990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009870 tx=0x7f7d0401c4b0 comp rx=0 tx=0).stop 2023-10-30T13:35:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2951212290 shutdown_connections 2023-10-30T13:35:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2951212290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d0806fb20 0x7f7d080c7990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2951212290 >> 172.21.15.31:0/2951212290 conn(0x7f7d080cd010 msgr2=0x7f7d080cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2951212290 shutdown_connections 2023-10-30T13:35:05.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2951212290 wait complete. 2023-10-30T13:35:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 Processor -- start 2023-10-30T13:35:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- start start 2023-10-30T13:35:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7d0806fb20 0x7f7d080c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0806ff00 0x7f7d08153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d08153bd0 0x7f7d080b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d080b0070 con 0x7f7d08153bd0 2023-10-30T13:35:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d080c5550 con 0x7f7d0806ff00 2023-10-30T13:35:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.084+0000 7f7d0f7c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d080c5810 con 0x7f7d0806fb20 2023-10-30T13:35:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d08153bd0 0x7f7d080b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d08153bd0 0x7f7d080b11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33288/0 (socket says 172.21.15.31:33288) 2023-10-30T13:35:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 -- 172.21.15.31:0/2066211241 learned_addr learned my addr 172.21.15.31:0/2066211241 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0d561640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7d0806fb20 0x7f7d080c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0cd60640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0806ff00 0x7f7d08153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 -- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7d0806fb20 msgr2=0x7f7d080c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7d0806fb20 0x7f7d080c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 -- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0806ff00 msgr2=0x7f7d08153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0806ff00 0x7f7d08153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 -- 172.21.15.31:0/2066211241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d040094d0 con 0x7f7d08153bd0 2023-10-30T13:35:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0dd62640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d08153bd0 0x7f7d080b11d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000c7a0 tx=0x7f7d0000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7cfdffb640 1 -- 172.21.15.31:0/2066211241 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0000a640 con 0x7f7d08153bd0 2023-10-30T13:35:05.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7cfdffb640 1 -- 172.21.15.31:0/2066211241 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0000a7a0 con 0x7f7d08153bd0 2023-10-30T13:35:05.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7cfdffb640 1 -- 172.21.15.31:0/2066211241 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0000ab10 con 0x7f7d08153bd0 2023-10-30T13:35:05.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d081f5c90 con 0x7f7d08153bd0 2023-10-30T13:35:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d081f6250 con 0x7f7d08153bd0 2023-10-30T13:35:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0804e170 con 0x7f7d08153bd0 2023-10-30T13:35:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7cfdffb640 1 -- 172.21.15.31:0/2066211241 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d0000ac70 con 0x7f7d08153bd0 2023-10-30T13:35:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7cfdffb640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7cec0664a0 0x7f7cec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.088+0000 7f7cfdffb640 1 -- 172.21.15.31:0/2066211241 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7d000833b0 con 0x7f7d08153bd0 2023-10-30T13:35:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.092+0000 7f7d0d561640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7cec0664a0 0x7f7cec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.092+0000 7f7d0d561640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7cec0664a0 0x7f7cec068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d040ad000 tx=0x7f7d04008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:05.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.092+0000 7f7cfdffb640 1 -- 172.21.15.31:0/2066211241 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d00050b00 con 0x7f7d08153bd0 2023-10-30T13:35:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:05.256+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7d0806c680 con 0x7f7d08153bd0 2023-10-30T13:35:06.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.240+0000 7f7cfdffb640 1 -- 172.21.15.31:0/2066211241 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7d000170b0 con 0x7f7d08153bd0 2023-10-30T13:35:06.246 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2023-10-30T13:35:06.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7cec0664a0 msgr2=0x7f7cec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:06.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7cec0664a0 0x7f7cec068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d040ad000 tx=0x7f7d04008040 comp rx=0 tx=0).stop 2023-10-30T13:35:06.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d08153bd0 msgr2=0x7f7d080b11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:06.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d08153bd0 0x7f7d080b11d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000c7a0 tx=0x7f7d0000cc60 comp rx=0 tx=0).stop 2023-10-30T13:35:06.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 shutdown_connections 2023-10-30T13:35:06.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7cec0664a0 0x7f7cec068960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d08153bd0 0x7f7d080b11d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0806ff00 0x7f7d08153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 --2- 172.21.15.31:0/2066211241 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7d0806fb20 0x7f7d080c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 >> 172.21.15.31:0/2066211241 conn(0x7f7d080cd010 msgr2=0x7f7d0815b800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:06.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 shutdown_connections 2023-10-30T13:35:06.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.244+0000 7f7d0f7c5640 1 -- 172.21.15.31:0/2066211241 wait complete. 2023-10-30T13:35:06.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-30T13:35:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaef217640 1 Processor -- start 2023-10-30T13:35:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaef217640 1 -- start start 2023-10-30T13:35:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaef217640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ffae80c4810 0x7ffae80cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaef217640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae8153f80 con 0x7ffae80c4410 2023-10-30T13:35:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaef217640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae8159df0 con 0x7ffae8153ba0 2023-10-30T13:35:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaef217640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae81597c0 con 0x7ffae80c4810 2023-10-30T13:35:06.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffae7fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ffae80c4410 0x7ffae80cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45044/0 (socket says 172.21.15.31:45044) 2023-10-30T13:35:06.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffae7fff640 1 -- 172.21.15.31:0/2877626845 learned_addr learned my addr 172.21.15.31:0/2877626845 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:06.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaed7b4640 1 --2- 172.21.15.31:0/2877626845 >> v2:172.21.15.33:3300/0 conn(0x7ffae80c4810 0x7ffae80cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffae77fe640 1 -- 172.21.15.31:0/2877626845 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610675231 0 0) 0x7ffae8153f80 con 0x7ffae80c4410 2023-10-30T13:35:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffae77fe640 1 -- 172.21.15.31:0/2877626845 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003530 con 0x7ffae80c4410 2023-10-30T13:35:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffae77fe640 1 -- 172.21.15.31:0/2877626845 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2156259389 0 0) 0x7ffad8003530 con 0x7ffae80c4410 2023-10-30T13:35:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaed7b4640 1 -- 172.21.15.31:0/2877626845 >> v1:172.21.15.33:6789/0 conn(0x7ffae8153ba0 legacy=0x7ffae815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaed7b4640 1 -- 172.21.15.31:0/2877626845 >> v1:172.21.15.31:6789/0 conn(0x7ffae80c4410 legacy=0x7ffae80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.428+0000 7ffaed7b4640 1 -- 172.21.15.31:0/2877626845 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae815d580 con 0x7ffae80c4810 2023-10-30T13:35:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaed7b4640 1 --2- 172.21.15.31:0/2877626845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4810 0x7ffae80cb4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009d90 tx=0x7ffae001c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47959e3c4139c15e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:06.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffae77fe640 1 -- 172.21.15.31:0/2877626845 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4001680 con 0x7ffae80c4810 2023-10-30T13:35:06.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffae77fe640 1 -- 172.21.15.31:0/2877626845 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae00a55f0 con 0x7ffae80c4810 2023-10-30T13:35:06.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffae77fe640 1 -- 172.21.15.31:0/2877626845 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae00a5960 con 0x7ffae80c4810 2023-10-30T13:35:06.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaef217640 1 -- 172.21.15.31:0/2877626845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4810 msgr2=0x7ffae80cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:06.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaef217640 1 --2- 172.21.15.31:0/2877626845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4810 0x7ffae80cb4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009d90 tx=0x7ffae001c7b0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaef217640 1 -- 172.21.15.31:0/2877626845 shutdown_connections 2023-10-30T13:35:06.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaef217640 1 --2- 172.21.15.31:0/2877626845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4810 0x7ffae80cb4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaef217640 1 -- 172.21.15.31:0/2877626845 >> 172.21.15.31:0/2877626845 conn(0x7ffae8071790 msgr2=0x7ffae8071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:06.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaef217640 1 -- 172.21.15.31:0/2877626845 shutdown_connections 2023-10-30T13:35:06.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.432+0000 7ffaef217640 1 -- 172.21.15.31:0/2877626845 wait complete. 2023-10-30T13:35:06.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 Processor -- start 2023-10-30T13:35:06.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 -- start start 2023-10-30T13:35:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4410 0x7ffae80cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae80c4810 0x7ffae814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffae8153ba0 0x7ffae814d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae814f2b0 con 0x7ffae80c4810 2023-10-30T13:35:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8151090 con 0x7ffae80c4410 2023-10-30T13:35:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8151350 con 0x7ffae8153ba0 2023-10-30T13:35:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae80c4810 0x7ffae814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae80c4810 0x7ffae814d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33298/0 (socket says 172.21.15.31:33298) 2023-10-30T13:35:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 -- 172.21.15.31:0/2231712496 learned_addr learned my addr 172.21.15.31:0/2231712496 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaed7b4640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffae8153ba0 0x7ffae814d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaecfb3640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4410 0x7ffae80cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 -- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffae8153ba0 msgr2=0x7ffae814d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffae8153ba0 0x7ffae814d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 -- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4410 msgr2=0x7ffae80cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4410 0x7ffae80cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 -- 172.21.15.31:0/2231712496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0009a70 con 0x7ffae80c4810 2023-10-30T13:35:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae7fff640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae80c4810 0x7ffae814d0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffad40011f0 tx=0x7ffad400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae57fa640 1 -- 172.21.15.31:0/2231712496 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4005ae0 con 0x7ffae80c4810 2023-10-30T13:35:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae57fa640 1 -- 172.21.15.31:0/2231712496 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad4005c40 con 0x7ffae80c4810 2023-10-30T13:35:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae815a7f0 con 0x7ffae80c4810 2023-10-30T13:35:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffae57fa640 1 -- 172.21.15.31:0/2231712496 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40072b0 con 0x7ffae80c4810 2023-10-30T13:35:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.436+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae80c8840 con 0x7ffae80c4810 2023-10-30T13:35:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.440+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae804e170 con 0x7ffae80c4810 2023-10-30T13:35:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.440+0000 7ffae57fa640 1 -- 172.21.15.31:0/2231712496 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffad4007410 con 0x7ffae80c4810 2023-10-30T13:35:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.440+0000 7ffae57fa640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffacc0664a0 0x7ffacc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.440+0000 7ffae57fa640 1 -- 172.21.15.31:0/2231712496 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ffad4083480 con 0x7ffae80c4810 2023-10-30T13:35:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.440+0000 7ffaecfb3640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffacc0664a0 0x7ffacc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.440+0000 7ffaecfb3640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffacc0664a0 0x7ffacc068960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000f70 tx=0x7ffadc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:06.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.444+0000 7ffae57fa640 1 -- 172.21.15.31:0/2231712496 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffad4050a40 con 0x7ffae80c4810 2023-10-30T13:35:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:06.604+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ffae80cabb0 con 0x7ffae80c4810 2023-10-30T13:35:07.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.244+0000 7ffae57fa640 1 -- 172.21.15.31:0/2231712496 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7ffad4054060 con 0x7ffae80c4810 2023-10-30T13:35:07.250 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2023-10-30T13:35:07.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffacc0664a0 msgr2=0x7ffacc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffacc0664a0 0x7ffacc068960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000f70 tx=0x7ffadc009040 comp rx=0 tx=0).stop 2023-10-30T13:35:07.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae80c4810 msgr2=0x7ffae814d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae80c4810 0x7ffae814d0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffad40011f0 tx=0x7ffad400cbe0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 shutdown_connections 2023-10-30T13:35:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffacc0664a0 0x7ffacc068960 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffae8153ba0 0x7ffae814d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae80c4810 0x7ffae814d0e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 --2- 172.21.15.31:0/2231712496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffae80c4410 0x7ffae80cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 >> 172.21.15.31:0/2231712496 conn(0x7ffae8071790 msgr2=0x7ffae80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 shutdown_connections 2023-10-30T13:35:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.248+0000 7ffaef217640 1 -- 172.21.15.31:0/2231712496 wait complete. 2023-10-30T13:35:07.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-30T13:35:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746fa9c640 1 Processor -- start 2023-10-30T13:35:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746fa9c640 1 -- start start 2023-10-30T13:35:07.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746fa9c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7468150630 0x7f74680c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746fa9c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f74680c5be0 0x7f7468159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746fa9c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7468157100 con 0x7f74680c1430 2023-10-30T13:35:07.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746fa9c640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7468150a10 con 0x7f74680c5be0 2023-10-30T13:35:07.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746fa9c640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7468072000 con 0x7f7468150630 2023-10-30T13:35:07.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d037640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f74680c5be0 0x7f7468159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d037640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f74680c5be0 0x7f7468159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33304/0 (socket says 172.21.15.31:33304) 2023-10-30T13:35:07.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d037640 1 -- 172.21.15.31:0/2949145159 learned_addr learned my addr 172.21.15.31:0/2949145159 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:07.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d838640 1 --2- 172.21.15.31:0/2949145159 >> v2:172.21.15.33:3300/0 conn(0x7f7468150630 0x7f74680c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d037640 1 -- 172.21.15.31:0/2949145159 >> v2:172.21.15.33:3300/0 conn(0x7f7468150630 msgr2=0x7f74680c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d037640 1 --2- 172.21.15.31:0/2949145159 >> v2:172.21.15.33:3300/0 conn(0x7f7468150630 0x7f74680c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d037640 1 -- 172.21.15.31:0/2949145159 >> v1:172.21.15.33:6789/0 conn(0x7f74680c1430 legacy=0x7f74680cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.440+0000 7f746d037640 1 -- 172.21.15.31:0/2949145159 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7468156510 con 0x7f74680c5be0 2023-10-30T13:35:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746d037640 1 --2- 172.21.15.31:0/2949145159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c5be0 0x7f7468159b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74680c13c0 tx=0x7f745801e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd72dd1c2a5ca152 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746c836640 1 -- 172.21.15.31:0/2949145159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580aa050 con 0x7f74680c5be0 2023-10-30T13:35:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746c836640 1 -- 172.21.15.31:0/2949145159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74580ae5f0 con 0x7f74680c5be0 2023-10-30T13:35:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746c836640 1 -- 172.21.15.31:0/2949145159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580ae980 con 0x7f74680c5be0 2023-10-30T13:35:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 -- 172.21.15.31:0/2949145159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c5be0 msgr2=0x7f7468159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 --2- 172.21.15.31:0/2949145159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c5be0 0x7f7468159b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74680c13c0 tx=0x7f745801e680 comp rx=0 tx=0).stop 2023-10-30T13:35:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 -- 172.21.15.31:0/2949145159 shutdown_connections 2023-10-30T13:35:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 --2- 172.21.15.31:0/2949145159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c5be0 0x7f7468159b00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 --2- 172.21.15.31:0/2949145159 >> v2:172.21.15.33:3300/0 conn(0x7f7468150630 0x7f74680c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 -- 172.21.15.31:0/2949145159 >> 172.21.15.31:0/2949145159 conn(0x7f746806da40 msgr2=0x7f746806de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 -- 172.21.15.31:0/2949145159 shutdown_connections 2023-10-30T13:35:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.444+0000 7f746fa9c640 1 -- 172.21.15.31:0/2949145159 wait complete. 2023-10-30T13:35:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 Processor -- start 2023-10-30T13:35:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 -- start start 2023-10-30T13:35:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c1430 0x7f746814b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f74680c5be0 0x7f7468158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7468150630 0x7f746814c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746814a720 con 0x7f74680c1430 2023-10-30T13:35:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74680c9410 con 0x7f7468150630 2023-10-30T13:35:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746814f510 con 0x7f74680c5be0 2023-10-30T13:35:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c1430 0x7f746814b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c1430 0x7f746814b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33312/0 (socket says 172.21.15.31:33312) 2023-10-30T13:35:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 -- 172.21.15.31:0/45687553 learned_addr learned my addr 172.21.15.31:0/45687553 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d037640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f74680c5be0 0x7f7468158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746e039640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7468150630 0x7f746814c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 -- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f74680c5be0 msgr2=0x7f7468158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f74680c5be0 0x7f7468158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 -- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7468150630 msgr2=0x7f746814c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7468150630 0x7f746814c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 -- 172.21.15.31:0/45687553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74580094d0 con 0x7f74680c1430 2023-10-30T13:35:07.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746d838640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c1430 0x7f746814b5e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f746400e930 tx=0x7f746400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:07.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f745e7fc640 1 -- 172.21.15.31:0/45687553 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464009690 con 0x7f74680c1430 2023-10-30T13:35:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f745e7fc640 1 -- 172.21.15.31:0/45687553 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746400ca00 con 0x7f74680c1430 2023-10-30T13:35:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f745e7fc640 1 -- 172.21.15.31:0/45687553 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746400cd70 con 0x7f74680c1430 2023-10-30T13:35:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746814f7f0 con 0x7f74680c1430 2023-10-30T13:35:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.448+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74681590d0 con 0x7f74680c1430 2023-10-30T13:35:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.452+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746804e170 con 0x7f74680c1430 2023-10-30T13:35:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.452+0000 7f745e7fc640 1 -- 172.21.15.31:0/45687553 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7464013470 con 0x7f74680c1430 2023-10-30T13:35:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.452+0000 7f745e7fc640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f744c0664a0 0x7f744c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.452+0000 7f745e7fc640 1 -- 172.21.15.31:0/45687553 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7464083ff0 con 0x7f74680c1430 2023-10-30T13:35:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.452+0000 7f746d037640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f744c0664a0 0x7f744c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.452+0000 7f746d037640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f744c0664a0 0x7f744c068960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74580b0000 tx=0x7f74580a9000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.456+0000 7f745e7fc640 1 -- 172.21.15.31:0/45687553 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7464051690 con 0x7f74680c1430 2023-10-30T13:35:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:07.652+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f74680c91e0 con 0x7f74680c1430 2023-10-30T13:35:08.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.268+0000 7f745e7fc640 1 -- 172.21.15.31:0/45687553 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7464054cb0 con 0x7f74680c1430 2023-10-30T13:35:08.273 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2023-10-30T13:35:08.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f744c0664a0 msgr2=0x7f744c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f744c0664a0 0x7f744c068960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74580b0000 tx=0x7f74580a9000 comp rx=0 tx=0).stop 2023-10-30T13:35:08.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c1430 msgr2=0x7f746814b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c1430 0x7f746814b5e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f746400e930 tx=0x7f746400edf0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 shutdown_connections 2023-10-30T13:35:08.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f744c0664a0 0x7f744c068960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7468150630 0x7f746814c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f74680c5be0 0x7f7468158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 --2- 172.21.15.31:0/45687553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74680c1430 0x7f746814b5e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 >> 172.21.15.31:0/45687553 conn(0x7f746806da40 msgr2=0x7f74681e7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:08.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 shutdown_connections 2023-10-30T13:35:08.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.272+0000 7f746fa9c640 1 -- 172.21.15.31:0/45687553 wait complete. 2023-10-30T13:35:08.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:08.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc890edc640 1 Processor -- start 2023-10-30T13:35:08.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc890edc640 1 -- start start 2023-10-30T13:35:08.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc890edc640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc890edc640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c15dcc0 con 0x7fc88c15ac70 2023-10-30T13:35:08.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc890edc640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c159d80 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc890edc640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c159800 con 0x7fc88c15cd70 2023-10-30T13:35:08.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc88c15ac70 0x7fc88c06aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45056/0 (socket says 172.21.15.31:45056) 2023-10-30T13:35:08.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88affd640 1 -- 172.21.15.31:0/2355617135 learned_addr learned my addr 172.21.15.31:0/2355617135 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:08.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88bfff640 1 --2- 172.21.15.31:0/2355617135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88a7fc640 1 -- 172.21.15.31:0/2355617135 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191614097 0 0) 0x7fc880001120 con 0x7fc88c15ac70 2023-10-30T13:35:08.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88a7fc640 1 -- 172.21.15.31:0/2355617135 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc874003530 con 0x7fc88c15ac70 2023-10-30T13:35:08.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88bfff640 1 -- 172.21.15.31:0/2355617135 >> v1:172.21.15.33:6789/0 conn(0x7fc88c0c8eb0 legacy=0x7fc88c15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88a7fc640 1 -- 172.21.15.31:0/2355617135 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893975617 0 0) 0x7fc88c159d80 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88bfff640 1 -- 172.21.15.31:0/2355617135 >> v1:172.21.15.31:6789/0 conn(0x7fc88c15ac70 legacy=0x7fc88c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.440+0000 7fc88bfff640 1 -- 172.21.15.31:0/2355617135 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c15d420 con 0x7fc88c15cd70 2023-10-30T13:35:08.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc88bfff640 1 --2- 172.21.15.31:0/2355617135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c069e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc884009dc0 tx=0x7fc88401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cdfd30961b13c84 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:08.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc88a7fc640 1 -- 172.21.15.31:0/2355617135 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8840af600 con 0x7fc88c15cd70 2023-10-30T13:35:08.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc88a7fc640 1 -- 172.21.15.31:0/2355617135 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8840ab050 con 0x7fc88c15cd70 2023-10-30T13:35:08.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc88a7fc640 1 -- 172.21.15.31:0/2355617135 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8840af960 con 0x7fc88c15cd70 2023-10-30T13:35:08.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc890edc640 1 -- 172.21.15.31:0/2355617135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 msgr2=0x7fc88c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc890edc640 1 --2- 172.21.15.31:0/2355617135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c069e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc884009dc0 tx=0x7fc88401f150 comp rx=0 tx=0).stop 2023-10-30T13:35:08.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc890edc640 1 -- 172.21.15.31:0/2355617135 shutdown_connections 2023-10-30T13:35:08.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc890edc640 1 --2- 172.21.15.31:0/2355617135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c069e90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc890edc640 1 -- 172.21.15.31:0/2355617135 >> 172.21.15.31:0/2355617135 conn(0x7fc88c0cef00 msgr2=0x7fc88c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:08.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc890edc640 1 -- 172.21.15.31:0/2355617135 shutdown_connections 2023-10-30T13:35:08.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.444+0000 7fc890edc640 1 -- 172.21.15.31:0/2355617135 wait complete. 2023-10-30T13:35:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 Processor -- start 2023-10-30T13:35:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 -- start start 2023-10-30T13:35:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c0c8eb0 0x7fc88c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c15ac70 0x7fc88c0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c1578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c1595c0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0c4000 con 0x7fc88c15ac70 2023-10-30T13:35:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0c2b00 con 0x7fc88c15cd70 2023-10-30T13:35:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c0c8eb0 0x7fc88c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c0c8eb0 0x7fc88c0c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33324/0 (socket says 172.21.15.31:33324) 2023-10-30T13:35:08.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 -- 172.21.15.31:0/4231785812 learned_addr learned my addr 172.21.15.31:0/4231785812 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:08.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88bfff640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c1578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88affd640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c15ac70 0x7fc88c0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 -- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 msgr2=0x7fc88c1578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c1578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 -- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c15ac70 msgr2=0x7fc88c0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c15ac70 0x7fc88c0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 -- 172.21.15.31:0/4231785812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884009a70 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc88b7fe640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c0c8eb0 0x7fc88c0c6960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00c410 tx=0x7fc87c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc873fff640 1 -- 172.21.15.31:0/4231785812 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0054e0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc873fff640 1 -- 172.21.15.31:0/4231785812 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c003600 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc873fff640 1 -- 172.21.15.31:0/4231785812 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0039b0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c0c2de0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c1591e0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.448+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c04e170 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.452+0000 7fc873fff640 1 -- 172.21.15.31:0/4231785812 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc87c003b10 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.452+0000 7fc873fff640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664a0 0x7fc878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.452+0000 7fc873fff640 1 -- 172.21.15.31:0/4231785812 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc87c0834b0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.452+0000 7fc88affd640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664a0 0x7fc878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.452+0000 7fc88affd640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664a0 0x7fc878068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc88c152750 tx=0x7fc880009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:08.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.456+0000 7fc873fff640 1 -- 172.21.15.31:0/4231785812 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc87c0508b0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc88c0ca960 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc873fff640 1 -- 172.21.15.31:0/4231785812 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16458 (secure 0 0 0) 0x7fc87c053ed0 con 0x7fc88c0c8eb0 2023-10-30T13:35:08.621 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:08.621 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:07.654426+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664a0 msgr2=0x7fc878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664a0 0x7fc878068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc88c152750 tx=0x7fc880009040 comp rx=0 tx=0).stop 2023-10-30T13:35:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c0c8eb0 msgr2=0x7fc88c0c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c0c8eb0 0x7fc88c0c6960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00c410 tx=0x7fc87c00c8d0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 shutdown_connections 2023-10-30T13:35:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664a0 0x7fc878068960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc88c15cd70 0x7fc88c1578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c15ac70 0x7fc88c0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 --2- 172.21.15.31:0/4231785812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c0c8eb0 0x7fc88c0c6960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.616+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 >> 172.21.15.31:0/4231785812 conn(0x7fc88c0cef00 msgr2=0x7fc88c151f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:08.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.620+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 shutdown_connections 2023-10-30T13:35:08.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.620+0000 7fc890edc640 1 -- 172.21.15.31:0/4231785812 wait complete. 2023-10-30T13:35:08.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:08.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.780+0000 7fd71bdd3640 1 Processor -- start 2023-10-30T13:35:08.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.780+0000 7fd71bdd3640 1 -- start start 2023-10-30T13:35:08.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71bdd3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd71406bb40 0x7fd714157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71bdd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406c420 0x7fd714157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71bdd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd714071e00 con 0x7fd71406b760 2023-10-30T13:35:08.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71bdd3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd714070020 con 0x7fd71406bb40 2023-10-30T13:35:08.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71bdd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd714070f10 con 0x7fd71406c420 2023-10-30T13:35:08.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71936e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd71406bb40 0x7fd714157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71936e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd71406bb40 0x7fd714157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33332/0 (socket says 172.21.15.31:33332) 2023-10-30T13:35:08.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71936e640 1 -- 172.21.15.31:0/3034078620 learned_addr learned my addr 172.21.15.31:0/3034078620 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:08.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71a370640 1 --2- 172.21.15.31:0/3034078620 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406c420 0x7fd714157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71936e640 1 -- 172.21.15.31:0/3034078620 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406c420 msgr2=0x7fd714157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:08.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71936e640 1 --2- 172.21.15.31:0/3034078620 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406c420 0x7fd714157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71936e640 1 -- 172.21.15.31:0/3034078620 >> v1:172.21.15.33:6789/0 conn(0x7fd71406b760 legacy=0x7fd71415e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.784+0000 7fd71936e640 1 -- 172.21.15.31:0/3034078620 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7140711f0 con 0x7fd71406bb40 2023-10-30T13:35:08.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71936e640 1 --2- 172.21.15.31:0/3034078620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406bb40 0x7fd714157a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd704009870 tx=0x7fd70401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4337500ebca0e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd718b6d640 1 -- 172.21.15.31:0/3034078620 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7040aa670 con 0x7fd71406bb40 2023-10-30T13:35:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd718b6d640 1 -- 172.21.15.31:0/3034078620 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7040aa7d0 con 0x7fd71406bb40 2023-10-30T13:35:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd718b6d640 1 -- 172.21.15.31:0/3034078620 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7040aab60 con 0x7fd71406bb40 2023-10-30T13:35:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 -- 172.21.15.31:0/3034078620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406bb40 msgr2=0x7fd714157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/3034078620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406bb40 0x7fd714157a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd704009870 tx=0x7fd70401c510 comp rx=0 tx=0).stop 2023-10-30T13:35:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 -- 172.21.15.31:0/3034078620 shutdown_connections 2023-10-30T13:35:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/3034078620 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406c420 0x7fd714157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/3034078620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406bb40 0x7fd714157a20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 -- 172.21.15.31:0/3034078620 >> 172.21.15.31:0/3034078620 conn(0x7fd7140cac70 msgr2=0x7fd7140cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:08.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 -- 172.21.15.31:0/3034078620 shutdown_connections 2023-10-30T13:35:08.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 -- 172.21.15.31:0/3034078620 wait complete. 2023-10-30T13:35:08.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 Processor -- start 2023-10-30T13:35:08.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.788+0000 7fd71bdd3640 1 -- start start 2023-10-30T13:35:08.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406b760 0x7fd71415b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406bb40 0x7fd7140c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71406c420 0x7fd714158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71415a790 con 0x7fd71406b760 2023-10-30T13:35:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7141f9980 con 0x7fd71406c420 2023-10-30T13:35:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71415cb20 con 0x7fd71406bb40 2023-10-30T13:35:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406b760 0x7fd71415b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406b760 0x7fd71415b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33348/0 (socket says 172.21.15.31:33348) 2023-10-30T13:35:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 -- 172.21.15.31:0/2068769401 learned_addr learned my addr 172.21.15.31:0/2068769401 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71936e640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406bb40 0x7fd7140c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71a370640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71406c420 0x7fd714158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 -- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406bb40 msgr2=0x7fd7140c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406bb40 0x7fd7140c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 -- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71406c420 msgr2=0x7fd714158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71406c420 0x7fd714158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 -- 172.21.15.31:0/2068769401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7040094d0 con 0x7fd71406b760 2023-10-30T13:35:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71936e640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406bb40 0x7fd7140c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd719b6f640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406b760 0x7fd71415b170 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd71000c500 tx=0x7fd71000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2068769401 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710005220 con 0x7fd71406b760 2023-10-30T13:35:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2068769401 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd710005380 con 0x7fd71406b760 2023-10-30T13:35:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2068769401 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710007270 con 0x7fd71406b760 2023-10-30T13:35:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71415ce00 con 0x7fd71406b760 2023-10-30T13:35:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71415d3c0 con 0x7fd71406b760 2023-10-30T13:35:08.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71404e170 con 0x7fd71406b760 2023-10-30T13:35:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.792+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2068769401 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd7100073d0 con 0x7fd71406b760 2023-10-30T13:35:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.796+0000 7fd70a7fc640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664a0 0x7fd6f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.796+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2068769401 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd710083880 con 0x7fd71406b760 2023-10-30T13:35:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.796+0000 7fd71936e640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664a0 0x7fd6f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.796+0000 7fd71936e640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664a0 0x7fd6f8068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd7040ae000 tx=0x7fd7040a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.800+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2068769401 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd710050c70 con 0x7fd71406b760 2023-10-30T13:35:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.956+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd714156650 con 0x7fd71406b760 2023-10-30T13:35:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.956+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2068769401 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fd710054290 con 0x7fd71406b760 2023-10-30T13:35:08.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:08.962 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:08.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664a0 msgr2=0x7fd6f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664a0 0x7fd6f8068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd7040ae000 tx=0x7fd7040a7000 comp rx=0 tx=0).stop 2023-10-30T13:35:08.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406b760 msgr2=0x7fd71415b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:08.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406b760 0x7fd71415b170 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd71000c500 tx=0x7fd71000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 shutdown_connections 2023-10-30T13:35:08.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664a0 0x7fd6f8068960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71406c420 0x7fd714158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd71406bb40 0x7fd7140c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 --2- 172.21.15.31:0/2068769401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd71406b760 0x7fd71415b170 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 >> 172.21.15.31:0/2068769401 conn(0x7fd7140cac70 msgr2=0x7fd714152340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 shutdown_connections 2023-10-30T13:35:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:08.960+0000 7fd71bdd3640 1 -- 172.21.15.31:0/2068769401 wait complete. 2023-10-30T13:35:08.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-30T13:35:09.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3b14c9640 1 Processor -- start 2023-10-30T13:35:09.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3b14c9640 1 -- start start 2023-10-30T13:35:09.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3b14c9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe3ac153ba0 0x7fe3ac0cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3b14c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4810 0x7fe3ac15a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3b14c9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac153f80 con 0x7fe3ac0c4410 2023-10-30T13:35:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3b14c9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ac159df0 con 0x7fe3ac153ba0 2023-10-30T13:35:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3b14c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac15a7c0 con 0x7fe3ac0c4810 2023-10-30T13:35:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aaffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe3ac153ba0 0x7fe3ac0cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aaffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe3ac153ba0 0x7fe3ac0cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33360/0 (socket says 172.21.15.31:33360) 2023-10-30T13:35:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aaffd640 1 -- 172.21.15.31:0/3323106523 learned_addr learned my addr 172.21.15.31:0/3323106523 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3ab7fe640 1 --2- 172.21.15.31:0/3323106523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4810 0x7fe3ac15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aaffd640 1 -- 172.21.15.31:0/3323106523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4810 msgr2=0x7fe3ac15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aaffd640 1 --2- 172.21.15.31:0/3323106523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4810 0x7fe3ac15a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aaffd640 1 -- 172.21.15.31:0/3323106523 >> v1:172.21.15.31:6789/0 conn(0x7fe3ac0c4410 legacy=0x7fe3ac15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aaffd640 1 -- 172.21.15.31:0/3323106523 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac1597c0 con 0x7fe3ac153ba0 2023-10-30T13:35:09.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.124+0000 7fe3aa7fc640 1 -- 172.21.15.31:0/3323106523 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452510559 0 0) 0x7fe3ac153f80 con 0x7fe3ac0c4410 2023-10-30T13:35:09.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3aaffd640 1 --2- 172.21.15.31:0/3323106523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac153ba0 0x7fe3ac0cb4e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009870 tx=0x7fe39c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fca0019ee9bf76 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:09.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3aa7fc640 1 -- 172.21.15.31:0/3323106523 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0ab680 con 0x7fe3ac153ba0 2023-10-30T13:35:09.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3aa7fc640 1 -- 172.21.15.31:0/3323106523 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c0ab7e0 con 0x7fe3ac153ba0 2023-10-30T13:35:09.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3aa7fc640 1 -- 172.21.15.31:0/3323106523 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0abb70 con 0x7fe3ac153ba0 2023-10-30T13:35:09.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 -- 172.21.15.31:0/3323106523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac153ba0 msgr2=0x7fe3ac0cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/3323106523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac153ba0 0x7fe3ac0cb4e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009870 tx=0x7fe39c01c480 comp rx=0 tx=0).stop 2023-10-30T13:35:09.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 -- 172.21.15.31:0/3323106523 shutdown_connections 2023-10-30T13:35:09.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/3323106523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4810 0x7fe3ac15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/3323106523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac153ba0 0x7fe3ac0cb4e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 -- 172.21.15.31:0/3323106523 >> 172.21.15.31:0/3323106523 conn(0x7fe3ac071790 msgr2=0x7fe3ac071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:09.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 -- 172.21.15.31:0/3323106523 shutdown_connections 2023-10-30T13:35:09.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.128+0000 7fe3b14c9640 1 -- 172.21.15.31:0/3323106523 wait complete. 2023-10-30T13:35:09.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 Processor -- start 2023-10-30T13:35:09.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 -- start start 2023-10-30T13:35:09.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4410 0x7fe3ac142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac0c4810 0x7fe3ac142e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3ac153ba0 0x7fe3ac152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac145150 con 0x7fe3ac0c4810 2023-10-30T13:35:09.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0ccff0 con 0x7fe3ac153ba0 2023-10-30T13:35:09.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0cd290 con 0x7fe3ac0c4410 2023-10-30T13:35:09.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac0c4810 0x7fe3ac142e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac0c4810 0x7fe3ac142e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33364/0 (socket says 172.21.15.31:33364) 2023-10-30T13:35:09.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 -- 172.21.15.31:0/1197000523 learned_addr learned my addr 172.21.15.31:0/1197000523 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:09.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3aaffd640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4410 0x7fe3ac142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3ab7fe640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3ac153ba0 0x7fe3ac152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 -- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4410 msgr2=0x7fe3ac142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4410 0x7fe3ac142930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 -- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3ac153ba0 msgr2=0x7fe3ac152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3ac153ba0 0x7fe3ac152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 -- 172.21.15.31:0/1197000523 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c0094d0 con 0x7fe3ac0c4810 2023-10-30T13:35:09.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a3fff640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac0c4810 0x7fe3ac142e70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c4d0 tx=0x7fe39400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:09.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a37fe640 1 -- 172.21.15.31:0/1197000523 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394005480 con 0x7fe3ac0c4810 2023-10-30T13:35:09.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a37fe640 1 -- 172.21.15.31:0/1197000523 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe394003600 con 0x7fe3ac0c4810 2023-10-30T13:35:09.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac144260 con 0x7fe3ac0c4810 2023-10-30T13:35:09.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3a37fe640 1 -- 172.21.15.31:0/1197000523 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3940039b0 con 0x7fe3ac0c4810 2023-10-30T13:35:09.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.132+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac144820 con 0x7fe3ac0c4810 2023-10-30T13:35:09.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.136+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac04e170 con 0x7fe3ac0c4810 2023-10-30T13:35:09.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.136+0000 7fe3a37fe640 1 -- 172.21.15.31:0/1197000523 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe394003b10 con 0x7fe3ac0c4810 2023-10-30T13:35:09.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.136+0000 7fe3a37fe640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe38c0664d0 0x7fe38c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.136+0000 7fe3a37fe640 1 -- 172.21.15.31:0/1197000523 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3940834d0 con 0x7fe3ac0c4810 2023-10-30T13:35:09.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.136+0000 7fe3aaffd640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe38c0664d0 0x7fe38c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.136+0000 7fe3aaffd640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe38c0664d0 0x7fe38c068990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0af000 tx=0x7fe39c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:09.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.140+0000 7fe3a37fe640 1 -- 172.21.15.31:0/1197000523 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe394050970 con 0x7fe3ac0c4810 2023-10-30T13:35:09.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:09.332+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe3ac0cabb0 con 0x7fe3ac0c4810 2023-10-30T13:35:10.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.292+0000 7fe3a37fe640 1 -- 172.21.15.31:0/1197000523 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe3940170b0 con 0x7fe3ac0c4810 2023-10-30T13:35:10.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe38c0664d0 msgr2=0x7fe38c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:10.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe38c0664d0 0x7fe38c068990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0af000 tx=0x7fe39c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:35:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac0c4810 msgr2=0x7fe3ac142e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac0c4810 0x7fe3ac142e70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c4d0 tx=0x7fe39400c990 comp rx=0 tx=0).stop 2023-10-30T13:35:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 shutdown_connections 2023-10-30T13:35:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe38c0664d0 0x7fe38c068990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3ac153ba0 0x7fe3ac152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3ac0c4810 0x7fe3ac142e70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 --2- 172.21.15.31:0/1197000523 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe3ac0c4410 0x7fe3ac142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 >> 172.21.15.31:0/1197000523 conn(0x7fe3ac071790 msgr2=0x7fe3ac071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:10.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 shutdown_connections 2023-10-30T13:35:10.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.296+0000 7fe3b14c9640 1 -- 172.21.15.31:0/1197000523 wait complete. 2023-10-30T13:35:10.323 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2023-10-30T13:35:10.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T13:35:10.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.468+0000 7fa461e46640 1 Processor -- start 2023-10-30T13:35:10.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.468+0000 7fa461e46640 1 -- start start 2023-10-30T13:35:10.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.468+0000 7fa461e46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa45c0c56a0 0x7fa45c1500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa461e46640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45c0c5e90 con 0x7fa45c072520 2023-10-30T13:35:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa461e46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45c0c6170 con 0x7fa45c0c64d0 2023-10-30T13:35:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa461e46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45c0c5ca0 con 0x7fa45c0c56a0 2023-10-30T13:35:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fa45c072520 0x7fa45c0cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45066/0 (socket says 172.21.15.31:45066) 2023-10-30T13:35:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 -- 172.21.15.31:0/4028262727 learned_addr learned my addr 172.21.15.31:0/4028262727 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45affd640 1 --2- 172.21.15.31:0/4028262727 >> v2:172.21.15.33:3300/0 conn(0x7fa45c0c56a0 0x7fa45c1500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916774243 0 0) 0x7fa45c0c5e90 con 0x7fa45c072520 2023-10-30T13:35:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa444005740 con 0x7fa45c072520 2023-10-30T13:35:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916818720 0 0) 0x7fa45c0c6170 con 0x7fa45c0c64d0 2023-10-30T13:35:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c0c5e90 con 0x7fa45c0c64d0 2023-10-30T13:35:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732853988 0 0) 0x7fa444005740 con 0x7fa45c072520 2023-10-30T13:35:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 >> v2:172.21.15.33:3300/0 conn(0x7fa45c0c56a0 msgr2=0x7fa45c1500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 --2- 172.21.15.31:0/4028262727 >> v2:172.21.15.33:3300/0 conn(0x7fa45c0c56a0 0x7fa45c1500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 >> v1:172.21.15.33:6789/0 conn(0x7fa45c0c64d0 legacy=0x7fa45c0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c06fe70 con 0x7fa45c072520 2023-10-30T13:35:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fa44c001680 con 0x7fa45c072520 2023-10-30T13:35:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 >> v1:172.21.15.31:6789/0 conn(0x7fa45c072520 legacy=0x7fa45c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa444006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4440069e0 0x7fa444006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4440073a0 0x7fa44400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44400e190 con 0x7fa444005d20 2023-10-30T13:35:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44400e2d0 con 0x7fa4440073a0 2023-10-30T13:35:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44400e490 con 0x7fa4440069e0 2023-10-30T13:35:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa444006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45affd640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4440069e0 0x7fa444006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45bfff640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4440073a0 0x7fa44400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 -- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4440069e0 msgr2=0x7fa444006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4440069e0 0x7fa444006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 -- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4440073a0 msgr2=0x7fa44400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4440073a0 0x7fa44400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45b7fe640 1 -- 172.21.15.31:0/4028262727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa444005740 con 0x7fa444005d20 2023-10-30T13:35:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.472+0000 7fa45bfff640 1 -- 172.21.15.31:0/4028262727 reap_dead start 2023-10-30T13:35:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa45b7fe640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa444006120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa44c001ec0 tx=0x7fa44c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e3d216465ea660 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0058e0 con 0x7fa444005d20 2023-10-30T13:35:10.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44c005a60 con 0x7fa444005d20 2023-10-30T13:35:10.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa45a7fc640 1 -- 172.21.15.31:0/4028262727 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c005e10 con 0x7fa444005d20 2023-10-30T13:35:10.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 -- 172.21.15.31:0/4028262727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 msgr2=0x7fa444006120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:10.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa444006120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa44c001ec0 tx=0x7fa44c01cba0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 -- 172.21.15.31:0/4028262727 shutdown_connections 2023-10-30T13:35:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 --2- 172.21.15.31:0/4028262727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa444006120 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 -- 172.21.15.31:0/4028262727 >> 172.21.15.31:0/4028262727 conn(0x7fa45c06da40 msgr2=0x7fa45c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 -- 172.21.15.31:0/4028262727 shutdown_connections 2023-10-30T13:35:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 -- 172.21.15.31:0/4028262727 wait complete. 2023-10-30T13:35:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.476+0000 7fa461e46640 1 Processor -- start 2023-10-30T13:35:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 -- start start 2023-10-30T13:35:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa45c0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa45c0ca680 0x7fa45c0c0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa45c1f2f40 0x7fa45c1f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c0addf0 con 0x7fa444005d20 2023-10-30T13:35:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1f2810 con 0x7fa45c0ca680 2023-10-30T13:35:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c153c30 con 0x7fa45c1f2f40 2023-10-30T13:35:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa45c0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa45c0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33386/0 (socket says 172.21.15.31:33386) 2023-10-30T13:35:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 -- 172.21.15.31:0/2665015097 learned_addr learned my addr 172.21.15.31:0/2665015097 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45bfff640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa45c1f2f40 0x7fa45c1f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45affd640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa45c0ca680 0x7fa45c0c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 -- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa45c1f2f40 msgr2=0x7fa45c1f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa45c1f2f40 0x7fa45c1f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 -- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa45c0ca680 msgr2=0x7fa45c0c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa45c0ca680 0x7fa45c0c0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:10.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 -- 172.21.15.31:0/2665015097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c153f10 con 0x7fa444005d20 2023-10-30T13:35:10.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45affd640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa45c0ca680 0x7fa45c0c0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:10.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa45b7fe640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa45c0b0000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0ab040 tx=0x7fa444007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:10.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa443fff640 1 -- 172.21.15.31:0/2665015097 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c004c00 con 0x7fa444005d20 2023-10-30T13:35:10.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa443fff640 1 -- 172.21.15.31:0/2665015097 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44c004d60 con 0x7fa444005d20 2023-10-30T13:35:10.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa443fff640 1 -- 172.21.15.31:0/2665015097 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0072d0 con 0x7fa444005d20 2023-10-30T13:35:10.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c1541f0 con 0x7fa444005d20 2023-10-30T13:35:10.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c1ef2a0 con 0x7fa444005d20 2023-10-30T13:35:10.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.480+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c04e170 con 0x7fa444005d20 2023-10-30T13:35:10.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.484+0000 7fa443fff640 1 -- 172.21.15.31:0/2665015097 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa44c007430 con 0x7fa444005d20 2023-10-30T13:35:10.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.484+0000 7fa443fff640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4480664d0 0x7fa448068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:10.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.484+0000 7fa443fff640 1 -- 172.21.15.31:0/2665015097 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa44c11d800 con 0x7fa444005d20 2023-10-30T13:35:10.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.484+0000 7fa45affd640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4480664d0 0x7fa448068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:10.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.484+0000 7fa45affd640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4480664d0 0x7fa448068990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa45c072920 tx=0x7fa450008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:10.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.488+0000 7fa443fff640 1 -- 172.21.15.31:0/2665015097 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa44c0eac60 con 0x7fa444005d20 2023-10-30T13:35:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:10.680+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fa45c0c1750 con 0x7fa444005d20 2023-10-30T13:35:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.300+0000 7fa443fff640 1 -- 172.21.15.31:0/2665015097 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa44c0be020 con 0x7fa444005d20 2023-10-30T13:35:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4480664d0 msgr2=0x7fa448068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4480664d0 0x7fa448068990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa45c072920 tx=0x7fa450008040 comp rx=0 tx=0).stop 2023-10-30T13:35:11.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 msgr2=0x7fa45c0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa45c0b0000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0ab040 tx=0x7fa444007a10 comp rx=0 tx=0).stop 2023-10-30T13:35:11.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 shutdown_connections 2023-10-30T13:35:11.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4480664d0 0x7fa448068990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa45c1f2f40 0x7fa45c1f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa45c0ca680 0x7fa45c0c0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 --2- 172.21.15.31:0/2665015097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa444005d20 0x7fa45c0b0000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 >> 172.21.15.31:0/2665015097 conn(0x7fa45c06da40 msgr2=0x7fa45c0c0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 shutdown_connections 2023-10-30T13:35:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.304+0000 7fa461e46640 1 -- 172.21.15.31:0/2665015097 wait complete. 2023-10-30T13:35:11.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-30T13:35:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.476+0000 7f10135d3640 1 Processor -- start 2023-10-30T13:35:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.476+0000 7f10135d3640 1 -- start start 2023-10-30T13:35:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f10135d3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f100c158070 0x7f100c1611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f10135d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 0x7f100c0720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f10135d3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c0717a0 con 0x7f100c15e650 2023-10-30T13:35:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f10135d3640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c162170 con 0x7f100c158070 2023-10-30T13:35:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f10135d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1616f0 con 0x7f100c071c30 2023-10-30T13:35:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f100c15e650 0x7f100c0c0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45074/0 (socket says 172.21.15.31:45074) 2023-10-30T13:35:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 -- 172.21.15.31:0/2378566573 learned_addr learned my addr 172.21.15.31:0/2378566573 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f101136f640 1 --2- 172.21.15.31:0/2378566573 >> v2:172.21.15.33:3300/0 conn(0x7f100c158070 0x7f100c1611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1011b70640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 0x7f100c0720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268968391 0 0) 0x7f100c0717a0 con 0x7f100c15e650 2023-10-30T13:35:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff8003530 con 0x7f100c15e650 2023-10-30T13:35:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 886842167 0 0) 0x7f0ff8003530 con 0x7f100c15e650 2023-10-30T13:35:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 msgr2=0x7f100c0720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 0x7f100c0720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 >> v2:172.21.15.33:3300/0 conn(0x7f100c158070 msgr2=0x7f100c1611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 --2- 172.21.15.31:0/2378566573 >> v2:172.21.15.33:3300/0 conn(0x7f100c158070 0x7f100c1611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c072600 con 0x7f100c15e650 2023-10-30T13:35:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f0ffc001680 con 0x7f100c15e650 2023-10-30T13:35:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f101136f640 1 --2- 172.21.15.31:0/2378566573 >> v2:172.21.15.33:3300/0 conn(0x7f100c158070 0x7f100c1611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 >> v1:172.21.15.31:6789/0 conn(0x7f100c15e650 legacy=0x7f100c0c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff8003980 0x7f0ff8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ff80047d0 0x7f0ff8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 0x7f0ff8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c162170 con 0x7f0ff80047d0 2023-10-30T13:35:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1616f0 con 0x7f0ff8003980 2023-10-30T13:35:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff800b990 con 0x7f100c071c30 2023-10-30T13:35:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ff80047d0 0x7f0ff8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f101136f640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff8003980 0x7f0ff8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1011b70640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 0x7f0ff8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 -- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 msgr2=0x7f0ff8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c071c30 0x7f0ff8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 -- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff8003980 msgr2=0x7f0ff8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:11.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff8003980 0x7f0ff8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.480+0000 7f1010b6e640 1 -- 172.21.15.31:0/2378566573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff8003530 con 0x7f0ff80047d0 2023-10-30T13:35:11.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f1010b6e640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ff80047d0 0x7f0ff8004c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c240 tx=0x7f0ffc01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6ec06f907d6df7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0058e0 con 0x7f0ff80047d0 2023-10-30T13:35:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc005a40 con 0x7f0ff80047d0 2023-10-30T13:35:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f1003fff640 1 -- 172.21.15.31:0/2378566573 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc005e10 con 0x7f0ff80047d0 2023-10-30T13:35:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f10135d3640 1 -- 172.21.15.31:0/2378566573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ff80047d0 msgr2=0x7f0ff8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f10135d3640 1 --2- 172.21.15.31:0/2378566573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ff80047d0 0x7f0ff8004c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c240 tx=0x7f0ffc01cc60 comp rx=0 tx=0).stop 2023-10-30T13:35:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f1011b70640 1 -- 172.21.15.31:0/2378566573 reap_dead start 2023-10-30T13:35:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f10135d3640 1 -- 172.21.15.31:0/2378566573 shutdown_connections 2023-10-30T13:35:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f10135d3640 1 -- 172.21.15.31:0/2378566573 >> 172.21.15.31:0/2378566573 conn(0x7f100c06da40 msgr2=0x7f100c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f10135d3640 1 -- 172.21.15.31:0/2378566573 shutdown_connections 2023-10-30T13:35:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.484+0000 7f10135d3640 1 -- 172.21.15.31:0/2378566573 wait complete. 2023-10-30T13:35:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 Processor -- start 2023-10-30T13:35:11.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 -- start start 2023-10-30T13:35:11.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff80047d0 0x7f100c0c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100c071190 0x7f100c1f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c1f3340 0x7f100c14bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c150ef0 con 0x7f100c071190 2023-10-30T13:35:11.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c150370 con 0x7f0ff80047d0 2023-10-30T13:35:11.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1f37c0 con 0x7f100c1f3340 2023-10-30T13:35:11.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100c071190 0x7f100c1f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f101136f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff80047d0 0x7f100c0c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100c071190 0x7f100c1f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33402/0 (socket says 172.21.15.31:33402) 2023-10-30T13:35:11.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 -- 172.21.15.31:0/2524423964 learned_addr learned my addr 172.21.15.31:0/2524423964 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:11.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1011b70640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c1f3340 0x7f100c14bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 -- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c1f3340 msgr2=0x7f100c14bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c1f3340 0x7f100c14bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 -- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff80047d0 msgr2=0x7f100c0c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff80047d0 0x7f100c0c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 -- 172.21.15.31:0/2524423964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c1521a0 con 0x7f100c071190 2023-10-30T13:35:11.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1010b6e640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100c071190 0x7f100c1f2e00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c240 tx=0x7f0ff8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:11.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1001ffb640 1 -- 172.21.15.31:0/2524423964 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0a5c90 con 0x7f100c071190 2023-10-30T13:35:11.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1001ffb640 1 -- 172.21.15.31:0/2524423964 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc0a5df0 con 0x7f100c071190 2023-10-30T13:35:11.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f1001ffb640 1 -- 172.21.15.31:0/2524423964 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0049f0 con 0x7f100c071190 2023-10-30T13:35:11.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c0c75e0 con 0x7f100c071190 2023-10-30T13:35:11.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c1508a0 con 0x7f100c071190 2023-10-30T13:35:11.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.488+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c1510a0 con 0x7f100c071190 2023-10-30T13:35:11.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.492+0000 7f1001ffb640 1 -- 172.21.15.31:0/2524423964 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ffc004b50 con 0x7f100c071190 2023-10-30T13:35:11.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.492+0000 7f1001ffb640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0ff00664a0 0x7f0ff0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.492+0000 7f1001ffb640 1 -- 172.21.15.31:0/2524423964 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ffc11dd40 con 0x7f100c071190 2023-10-30T13:35:11.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.492+0000 7f101136f640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0ff00664a0 0x7f0ff0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.492+0000 7f101136f640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0ff00664a0 0x7f0ff0068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10080093f0 tx=0x7f1008009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:11.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.496+0000 7f1001ffb640 1 -- 172.21.15.31:0/2524423964 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ffc0eb260 con 0x7f100c071190 2023-10-30T13:35:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:11.688+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f100c04e170 con 0x7f100c071190 2023-10-30T13:35:12.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f1001ffb640 1 -- 172.21.15.31:0/2524423964 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ffc0ee880 con 0x7f100c071190 2023-10-30T13:35:12.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0ff00664a0 msgr2=0x7f0ff0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:12.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0ff00664a0 0x7f0ff0068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10080093f0 tx=0x7f1008009190 comp rx=0 tx=0).stop 2023-10-30T13:35:12.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100c071190 msgr2=0x7f100c1f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:12.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100c071190 0x7f100c1f2e00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c240 tx=0x7f0ff8005800 comp rx=0 tx=0).stop 2023-10-30T13:35:12.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 shutdown_connections 2023-10-30T13:35:12.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0ff00664a0 0x7f0ff0068960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f100c1f3340 0x7f100c14bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100c071190 0x7f100c1f2e00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 --2- 172.21.15.31:0/2524423964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ff80047d0 0x7f100c0c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.312+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 >> 172.21.15.31:0/2524423964 conn(0x7f100c06da40 msgr2=0x7f100c0c02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:12.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.316+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 shutdown_connections 2023-10-30T13:35:12.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.316+0000 7f10135d3640 1 -- 172.21.15.31:0/2524423964 wait complete. 2023-10-30T13:35:12.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-30T13:35:12.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.484+0000 7f30e99aa640 1 Processor -- start 2023-10-30T13:35:12.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.484+0000 7f30e99aa640 1 -- start start 2023-10-30T13:35:12.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e99aa640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e4153ba0 0x7f30e40cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:12.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e99aa640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f30e40c4410 0x7f30e415a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:12.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e99aa640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e4153f80 con 0x7f30e40c4810 2023-10-30T13:35:12.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e99aa640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e4159df0 con 0x7f30e40c4410 2023-10-30T13:35:12.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e99aa640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e415a7c0 con 0x7f30e4153ba0 2023-10-30T13:35:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e2ffd640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e4153ba0 0x7f30e40cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30dbfff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f30e40c4410 0x7f30e415a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e37fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f30e40c4810 0x7f30e415cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:44916/0 (socket says 172.21.15.31:44916) 2023-10-30T13:35:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30e37fe640 1 -- 172.21.15.31:0/2052157055 learned_addr learned my addr 172.21.15.31:0/2052157055 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30dbfff640 1 -- 172.21.15.31:0/2052157055 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e4153ba0 msgr2=0x7f30e40cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30dbfff640 1 --2- 172.21.15.31:0/2052157055 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e4153ba0 0x7f30e40cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30dbfff640 1 -- 172.21.15.31:0/2052157055 >> v1:172.21.15.33:6789/0 conn(0x7f30e40c4810 legacy=0x7f30e415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.488+0000 7f30dbfff640 1 -- 172.21.15.31:0/2052157055 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e41597c0 con 0x7f30e40c4410 2023-10-30T13:35:12.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30dbfff640 1 --2- 172.21.15.31:0/2052157055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4410 0x7f30e415a280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f30cc0097c0 tx=0x7f30cc01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b34e15bcc9c4ab0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:12.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e27fc640 1 -- 172.21.15.31:0/2052157055 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc0ac620 con 0x7f30e40c4410 2023-10-30T13:35:12.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e27fc640 1 -- 172.21.15.31:0/2052157055 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc0ac780 con 0x7f30e40c4410 2023-10-30T13:35:12.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e27fc640 1 -- 172.21.15.31:0/2052157055 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc0acb10 con 0x7f30e40c4410 2023-10-30T13:35:12.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 -- 172.21.15.31:0/2052157055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4410 msgr2=0x7f30e415a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:12.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 --2- 172.21.15.31:0/2052157055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4410 0x7f30e415a280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f30cc0097c0 tx=0x7f30cc01c3c0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 -- 172.21.15.31:0/2052157055 shutdown_connections 2023-10-30T13:35:12.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 --2- 172.21.15.31:0/2052157055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4410 0x7f30e415a280 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 --2- 172.21.15.31:0/2052157055 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e4153ba0 0x7f30e40cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 -- 172.21.15.31:0/2052157055 >> 172.21.15.31:0/2052157055 conn(0x7f30e4071790 msgr2=0x7f30e4071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:12.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 -- 172.21.15.31:0/2052157055 shutdown_connections 2023-10-30T13:35:12.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 -- 172.21.15.31:0/2052157055 wait complete. 2023-10-30T13:35:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 Processor -- start 2023-10-30T13:35:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.492+0000 7f30e99aa640 1 -- start start 2023-10-30T13:35:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e40c4410 0x7f30e40b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4810 0x7f30e40c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4153ba0 0x7f30e4141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e40b4cf0 con 0x7f30e4153ba0 2023-10-30T13:35:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e40cd190 con 0x7f30e40c4810 2023-10-30T13:35:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4143440 con 0x7f30e40c4410 2023-10-30T13:35:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4153ba0 0x7f30e4141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4153ba0 0x7f30e4141120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33404/0 (socket says 172.21.15.31:33404) 2023-10-30T13:35:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 -- 172.21.15.31:0/3111377965 learned_addr learned my addr 172.21.15.31:0/3111377965 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e2ffd640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e40c4410 0x7f30e40b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30dbfff640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4810 0x7f30e40c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 -- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e40c4410 msgr2=0x7f30e40b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:12.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e40c4410 0x7f30e40b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 -- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4810 msgr2=0x7f30e40c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:12.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4810 0x7f30e40c54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:12.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 -- 172.21.15.31:0/3111377965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc009420 con 0x7f30e4153ba0 2023-10-30T13:35:12.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e37fe640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4153ba0 0x7f30e4141120 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00cb20 tx=0x7f30dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e0ff9640 1 -- 172.21.15.31:0/3111377965 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00aca0 con 0x7f30e4153ba0 2023-10-30T13:35:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e0ff9640 1 -- 172.21.15.31:0/3111377965 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc00ae00 con 0x7f30e4153ba0 2023-10-30T13:35:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e0ff9640 1 -- 172.21.15.31:0/3111377965 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc012600 con 0x7f30e4153ba0 2023-10-30T13:35:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e40b4f20 con 0x7f30e4153ba0 2023-10-30T13:35:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e414fd00 con 0x7f30e4153ba0 2023-10-30T13:35:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e40cd340 con 0x7f30e4153ba0 2023-10-30T13:35:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.496+0000 7f30e0ff9640 1 -- 172.21.15.31:0/3111377965 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f30dc007070 con 0x7f30e4153ba0 2023-10-30T13:35:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.500+0000 7f30e0ff9640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f30c40664a0 0x7f30c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.500+0000 7f30e0ff9640 1 -- 172.21.15.31:0/3111377965 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30dc083d80 con 0x7f30e4153ba0 2023-10-30T13:35:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.500+0000 7f30e2ffd640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f30c40664a0 0x7f30c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:12.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.500+0000 7f30e2ffd640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f30c40664a0 0x7f30c4068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009a50 tx=0x7f30d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:12.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.504+0000 7f30e0ff9640 1 -- 172.21.15.31:0/3111377965 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f30e40cd340 con 0x7f30e4153ba0 2023-10-30T13:35:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:12.692+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f30e40cd340 con 0x7f30e4153ba0 2023-10-30T13:35:13.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e0ff9640 1 -- 172.21.15.31:0/3111377965 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f30e40cd340 con 0x7f30e4153ba0 2023-10-30T13:35:13.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f30c40664a0 msgr2=0x7f30c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f30c40664a0 0x7f30c4068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009a50 tx=0x7f30d4008040 comp rx=0 tx=0).stop 2023-10-30T13:35:13.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4153ba0 msgr2=0x7f30e4141120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4153ba0 0x7f30e4141120 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00cb20 tx=0x7f30dc00a060 comp rx=0 tx=0).stop 2023-10-30T13:35:13.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 shutdown_connections 2023-10-30T13:35:13.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f30c40664a0 0x7f30c4068960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4153ba0 0x7f30e4141120 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e40c4810 0x7f30e40c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 --2- 172.21.15.31:0/3111377965 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f30e40c4410 0x7f30e40b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 >> 172.21.15.31:0/3111377965 conn(0x7f30e4071790 msgr2=0x7f30e40c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:13.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 shutdown_connections 2023-10-30T13:35:13.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.324+0000 7f30e99aa640 1 -- 172.21.15.31:0/3111377965 wait complete. 2023-10-30T13:35:13.353 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-30T13:35:13.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:13.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f03733f5640 1 Processor -- start 2023-10-30T13:35:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f03733f5640 1 -- start start 2023-10-30T13:35:13.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f03733f5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c153c80 0x7f036c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f03733f5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f036c0c4a10 0x7f036c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f03733f5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c159f20 con 0x7f036c15a3b0 2023-10-30T13:35:13.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f03733f5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036c15dec0 con 0x7f036c0c4a10 2023-10-30T13:35:13.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f03733f5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1599a0 con 0x7f036c153c80 2023-10-30T13:35:13.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371191640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c153c80 0x7f036c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0370990640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f036c0c4a10 0x7f036c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371191640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c153c80 0x7f036c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:55224/0 (socket says 172.21.15.31:55224) 2023-10-30T13:35:13.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371191640 1 -- 172.21.15.31:0/672548732 learned_addr learned my addr 172.21.15.31:0/672548732 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:13.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371992640 1 --1- 172.21.15.31:0/672548732 >> v1:172.21.15.33:6789/0 conn(0x7f036c15a3b0 0x7f036c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:44932/0 (socket says 172.21.15.31:44932) 2023-10-30T13:35:13.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371191640 1 -- 172.21.15.31:0/672548732 >> v2:172.21.15.33:3300/0 conn(0x7f036c0c4a10 msgr2=0x7f036c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:13.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371191640 1 --2- 172.21.15.31:0/672548732 >> v2:172.21.15.33:3300/0 conn(0x7f036c0c4a10 0x7f036c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371191640 1 -- 172.21.15.31:0/672548732 >> v1:172.21.15.33:6789/0 conn(0x7f036c15a3b0 legacy=0x7f036c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.496+0000 7f0371191640 1 -- 172.21.15.31:0/672548732 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c0c47e0 con 0x7f036c153c80 2023-10-30T13:35:13.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f0371191640 1 --2- 172.21.15.31:0/672548732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c153c80 0x7f036c15cf90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0368009870 tx=0x7f036801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ac93feeb9e63267 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:13.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f0363fff640 1 -- 172.21.15.31:0/672548732 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680a8a30 con 0x7f036c153c80 2023-10-30T13:35:13.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f0363fff640 1 -- 172.21.15.31:0/672548732 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03680a8b90 con 0x7f036c153c80 2023-10-30T13:35:13.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f0363fff640 1 -- 172.21.15.31:0/672548732 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680b0440 con 0x7f036c153c80 2023-10-30T13:35:13.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f03733f5640 1 -- 172.21.15.31:0/672548732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c153c80 msgr2=0x7f036c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f03733f5640 1 --2- 172.21.15.31:0/672548732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c153c80 0x7f036c15cf90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0368009870 tx=0x7f036801ed30 comp rx=0 tx=0).stop 2023-10-30T13:35:13.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f03733f5640 1 -- 172.21.15.31:0/672548732 shutdown_connections 2023-10-30T13:35:13.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f03733f5640 1 --2- 172.21.15.31:0/672548732 >> v2:172.21.15.33:3300/0 conn(0x7f036c0c4a10 0x7f036c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f03733f5640 1 --2- 172.21.15.31:0/672548732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c153c80 0x7f036c15cf90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.500+0000 7f03733f5640 1 -- 172.21.15.31:0/672548732 >> 172.21.15.31:0/672548732 conn(0x7f036c0715c0 msgr2=0x7f036c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:13.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 -- 172.21.15.31:0/672548732 shutdown_connections 2023-10-30T13:35:13.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 -- 172.21.15.31:0/672548732 wait complete. 2023-10-30T13:35:13.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 Processor -- start 2023-10-30T13:35:13.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 -- start start 2023-10-30T13:35:13.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c0c4a10 0x7f036c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f036c153c80 0x7f036c0ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f036c15a3b0 0x7f036c0c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1f3570 con 0x7f036c15a3b0 2023-10-30T13:35:13.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1ef260 con 0x7f036c153c80 2023-10-30T13:35:13.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1ef500 con 0x7f036c0c4a10 2023-10-30T13:35:13.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f036c15a3b0 0x7f036c0c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f036c15a3b0 0x7f036c0c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33412/0 (socket says 172.21.15.31:33412) 2023-10-30T13:35:13.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 -- 172.21.15.31:0/1672176438 learned_addr learned my addr 172.21.15.31:0/1672176438 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:13.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0370990640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f036c153c80 0x7f036c0ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371191640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c0c4a10 0x7f036c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 -- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c0c4a10 msgr2=0x7f036c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c0c4a10 0x7f036c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 -- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f036c153c80 msgr2=0x7f036c0ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f036c153c80 0x7f036c0ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 -- 172.21.15.31:0/1672176438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680094d0 con 0x7f036c15a3b0 2023-10-30T13:35:13.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0370990640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f036c153c80 0x7f036c0ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:13.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371191640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c0c4a10 0x7f036c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:13.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0371992640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f036c15a3b0 0x7f036c0c7b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f036400c700 tx=0x7f036400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:13.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0361ffb640 1 -- 172.21.15.31:0/1672176438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036400ce10 con 0x7f036c15a3b0 2023-10-30T13:35:13.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0361ffb640 1 -- 172.21.15.31:0/1672176438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036400a640 con 0x7f036c15a3b0 2023-10-30T13:35:13.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c1ef7e0 con 0x7f036c15a3b0 2023-10-30T13:35:13.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.504+0000 7f0361ffb640 1 -- 172.21.15.31:0/1672176438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036400a9b0 con 0x7f036c15a3b0 2023-10-30T13:35:13.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.508+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c1efda0 con 0x7f036c15a3b0 2023-10-30T13:35:13.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.508+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036c0c7840 con 0x7f036c15a3b0 2023-10-30T13:35:13.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.508+0000 7f0361ffb640 1 -- 172.21.15.31:0/1672176438 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f036400ab10 con 0x7f036c15a3b0 2023-10-30T13:35:13.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.508+0000 7f0361ffb640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f03500664a0 0x7f0350068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.508+0000 7f0361ffb640 1 -- 172.21.15.31:0/1672176438 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0364083640 con 0x7f036c15a3b0 2023-10-30T13:35:13.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.508+0000 7f0371191640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f03500664a0 0x7f0350068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.508+0000 7f0371191640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f03500664a0 0x7f0350068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03680b2000 tx=0x7f0368008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:13.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.512+0000 7f0361ffb640 1 -- 172.21.15.31:0/1672176438 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f036c0c7840 con 0x7f036c15a3b0 2023-10-30T13:35:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f036c0c7840 con 0x7f036c15a3b0 2023-10-30T13:35:13.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f0361ffb640 1 -- 172.21.15.31:0/1672176438 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6487 (secure 0 0 0) 0x7f036c0c7840 con 0x7f036c15a3b0 2023-10-30T13:35:13.711 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:13.711 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:12.697491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:13.712 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2023-10-30T13:35:13.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f03500664a0 msgr2=0x7f0350068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f03500664a0 0x7f0350068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03680b2000 tx=0x7f0368008040 comp rx=0 tx=0).stop 2023-10-30T13:35:13.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f036c15a3b0 msgr2=0x7f036c0c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f036c15a3b0 0x7f036c0c7b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f036400c700 tx=0x7f036400cbc0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 shutdown_connections 2023-10-30T13:35:13.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f03500664a0 0x7f0350068960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f036c15a3b0 0x7f036c0c7b20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f036c153c80 0x7f036c0ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 --2- 172.21.15.31:0/1672176438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f036c0c4a10 0x7f036c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 >> 172.21.15.31:0/1672176438 conn(0x7f036c0715c0 msgr2=0x7f036c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:13.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 shutdown_connections 2023-10-30T13:35:13.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.708+0000 7f03733f5640 1 -- 172.21.15.31:0/1672176438 wait complete. 2023-10-30T13:35:13.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9045a46640 1 Processor -- start 2023-10-30T13:35:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9045a46640 1 -- start start 2023-10-30T13:35:13.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9045a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040071cb0 0x7f90400c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9045a46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9040153300 0x7f9040153790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9045a46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9040153cd0 con 0x7f90400718d0 2023-10-30T13:35:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9045a46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90400cca10 con 0x7f9040153300 2023-10-30T13:35:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9045a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904015d0a0 con 0x7f9040071cb0 2023-10-30T13:35:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f903f7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9040153300 0x7f9040153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f903f7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9040153300 0x7f9040153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33422/0 (socket says 172.21.15.31:33422) 2023-10-30T13:35:13.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f903f7fe640 1 -- 172.21.15.31:0/1737468001 learned_addr learned my addr 172.21.15.31:0/1737468001 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:13.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f9037fff640 1 --2- 172.21.15.31:0/1737468001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040071cb0 0x7f90400c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f903f7fe640 1 -- 172.21.15.31:0/1737468001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040071cb0 msgr2=0x7f90400c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:13.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f903f7fe640 1 --2- 172.21.15.31:0/1737468001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040071cb0 0x7f90400c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f903f7fe640 1 -- 172.21.15.31:0/1737468001 >> v1:172.21.15.33:6789/0 conn(0x7f90400718d0 legacy=0x7f904015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.872+0000 7f903f7fe640 1 -- 172.21.15.31:0/1737468001 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904015d380 con 0x7f9040153300 2023-10-30T13:35:13.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f903f7fe640 1 --2- 172.21.15.31:0/1737468001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040153300 0x7f9040153790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9038009e50 tx=0x7f903801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8a49b8b9b56591 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:13.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f903e7fc640 1 -- 172.21.15.31:0/1737468001 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90380af400 con 0x7f9040153300 2023-10-30T13:35:13.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f903e7fc640 1 -- 172.21.15.31:0/1737468001 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90380ab050 con 0x7f9040153300 2023-10-30T13:35:13.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f903e7fc640 1 -- 172.21.15.31:0/1737468001 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90380af780 con 0x7f9040153300 2023-10-30T13:35:13.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 -- 172.21.15.31:0/1737468001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040153300 msgr2=0x7f9040153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 --2- 172.21.15.31:0/1737468001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040153300 0x7f9040153790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9038009e50 tx=0x7f903801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 -- 172.21.15.31:0/1737468001 shutdown_connections 2023-10-30T13:35:13.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 --2- 172.21.15.31:0/1737468001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040153300 0x7f9040153790 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 --2- 172.21.15.31:0/1737468001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040071cb0 0x7f90400c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 -- 172.21.15.31:0/1737468001 >> 172.21.15.31:0/1737468001 conn(0x7f904006ab00 msgr2=0x7f9040065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:13.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 -- 172.21.15.31:0/1737468001 shutdown_connections 2023-10-30T13:35:13.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.876+0000 7f9045a46640 1 -- 172.21.15.31:0/1737468001 wait complete. 2023-10-30T13:35:13.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 Processor -- start 2023-10-30T13:35:13.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 -- start start 2023-10-30T13:35:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90400718d0 0x7f9040158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040071cb0 0x7f904014bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040153300 0x7f90401573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9040158450 con 0x7f9040071cb0 2023-10-30T13:35:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90401f3160 con 0x7f90400718d0 2023-10-30T13:35:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90400b40f0 con 0x7f9040153300 2023-10-30T13:35:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040071cb0 0x7f904014bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040071cb0 0x7f904014bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33424/0 (socket says 172.21.15.31:33424) 2023-10-30T13:35:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 -- 172.21.15.31:0/1692669480 learned_addr learned my addr 172.21.15.31:0/1692669480 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f903f7fe640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040153300 0x7f90401573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f903effd640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90400718d0 0x7f9040158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 -- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040153300 msgr2=0x7f90401573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040153300 0x7f90401573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 -- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90400718d0 msgr2=0x7f9040158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:13.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90400718d0 0x7f9040158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:13.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 -- 172.21.15.31:0/1692669480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9038009b30 con 0x7f9040071cb0 2023-10-30T13:35:13.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9037fff640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040071cb0 0x7f904014bfa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f902800e8d0 tx=0x7f902800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:13.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f903cff9640 1 -- 172.21.15.31:0/1692669480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800c940 con 0x7f9040071cb0 2023-10-30T13:35:13.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f903cff9640 1 -- 172.21.15.31:0/1692669480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902800caa0 con 0x7f9040071cb0 2023-10-30T13:35:13.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90400b4390 con 0x7f9040071cb0 2023-10-30T13:35:13.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f903cff9640 1 -- 172.21.15.31:0/1692669480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800ce10 con 0x7f9040071cb0 2023-10-30T13:35:13.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.880+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9040158940 con 0x7f9040071cb0 2023-10-30T13:35:13.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.884+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904004e170 con 0x7f9040071cb0 2023-10-30T13:35:13.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.884+0000 7f903cff9640 1 -- 172.21.15.31:0/1692669480 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9028013470 con 0x7f9040071cb0 2023-10-30T13:35:13.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.884+0000 7f903cff9640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f90200664a0 0x7f9020068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:13.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.884+0000 7f903cff9640 1 -- 172.21.15.31:0/1692669480 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9028084170 con 0x7f9040071cb0 2023-10-30T13:35:13.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.884+0000 7f903effd640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f90200664a0 0x7f9020068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:13.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.884+0000 7f903effd640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f90200664a0 0x7f9020068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9030000f70 tx=0x7f9030009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:13.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:13.888+0000 7f903cff9640 1 -- 172.21.15.31:0/1692669480 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9028051580 con 0x7f9040071cb0 2023-10-30T13:35:14.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.044+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90400b4580 con 0x7f9040071cb0 2023-10-30T13:35:14.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f903cff9640 1 -- 172.21.15.31:0/1692669480 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f90400b4580 con 0x7f9040071cb0 2023-10-30T13:35:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:14.051 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:14.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f90200664a0 msgr2=0x7f9020068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f90200664a0 0x7f9020068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9030000f70 tx=0x7f9030009040 comp rx=0 tx=0).stop 2023-10-30T13:35:14.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040071cb0 msgr2=0x7f904014bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040071cb0 0x7f904014bfa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f902800e8d0 tx=0x7f902800ed90 comp rx=0 tx=0).stop 2023-10-30T13:35:14.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 shutdown_connections 2023-10-30T13:35:14.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f90200664a0 0x7f9020068960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9040153300 0x7f90401573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9040071cb0 0x7f904014bfa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 --2- 172.21.15.31:0/1692669480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90400718d0 0x7f9040158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 >> 172.21.15.31:0/1692669480 conn(0x7f904006ab00 msgr2=0x7f9040151ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:14.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 shutdown_connections 2023-10-30T13:35:14.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.048+0000 7f9045a46640 1 -- 172.21.15.31:0/1692669480 wait complete. 2023-10-30T13:35:14.072 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-30T13:35:14.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:35:14.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d5349640 1 Processor -- start 2023-10-30T13:35:14.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d5349640 1 -- start start 2023-10-30T13:35:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d5349640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d015ac70 0x7ff5d015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d5349640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff5d006aeb0 0x7ff5d0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d5349640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d015dcc0 con 0x7ff5d00c8eb0 2023-10-30T13:35:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d5349640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d0159800 con 0x7ff5d006aeb0 2023-10-30T13:35:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d5349640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d0159d80 con 0x7ff5d015ac70 2023-10-30T13:35:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d4b48640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff5d006aeb0 0x7ff5d0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d4b48640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff5d006aeb0 0x7ff5d0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33430/0 (socket says 172.21.15.31:33430) 2023-10-30T13:35:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d4b48640 1 -- 172.21.15.31:0/147550699 learned_addr learned my addr 172.21.15.31:0/147550699 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5cf7fe640 1 --2- 172.21.15.31:0/147550699 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d015ac70 0x7ff5d015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d4b48640 1 -- 172.21.15.31:0/147550699 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d015ac70 msgr2=0x7ff5d015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d4b48640 1 --2- 172.21.15.31:0/147550699 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d015ac70 0x7ff5d015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d4b48640 1 -- 172.21.15.31:0/147550699 >> v1:172.21.15.33:6789/0 conn(0x7ff5d00c8eb0 legacy=0x7ff5d015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.212+0000 7ff5d4b48640 1 -- 172.21.15.31:0/147550699 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d015d420 con 0x7ff5d006aeb0 2023-10-30T13:35:14.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d4b48640 1 --2- 172.21.15.31:0/147550699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d0069e90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009e50 tx=0x7ff5c801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b753d5eae7877f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5ceffd640 1 -- 172.21.15.31:0/147550699 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c80af400 con 0x7ff5d006aeb0 2023-10-30T13:35:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5ceffd640 1 -- 172.21.15.31:0/147550699 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c80ab050 con 0x7ff5d006aeb0 2023-10-30T13:35:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5ceffd640 1 -- 172.21.15.31:0/147550699 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c80af780 con 0x7ff5d006aeb0 2023-10-30T13:35:14.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 -- 172.21.15.31:0/147550699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 msgr2=0x7ff5d0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 --2- 172.21.15.31:0/147550699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d0069e90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009e50 tx=0x7ff5c801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 -- 172.21.15.31:0/147550699 shutdown_connections 2023-10-30T13:35:14.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 --2- 172.21.15.31:0/147550699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d0069e90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 --2- 172.21.15.31:0/147550699 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d015ac70 0x7ff5d015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 -- 172.21.15.31:0/147550699 >> 172.21.15.31:0/147550699 conn(0x7ff5d00cef00 msgr2=0x7ff5d00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:14.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 -- 172.21.15.31:0/147550699 shutdown_connections 2023-10-30T13:35:14.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 -- 172.21.15.31:0/147550699 wait complete. 2023-10-30T13:35:14.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 Processor -- start 2023-10-30T13:35:14.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.216+0000 7ff5d5349640 1 -- start start 2023-10-30T13:35:14.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d01530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d00c8eb0 0x7ff5d00cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5d015ac70 0x7ff5d00cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00ce6d0 con 0x7ff5d006aeb0 2023-10-30T13:35:14.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00ccea0 con 0x7ff5d015ac70 2023-10-30T13:35:14.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00ce150 con 0x7ff5d00c8eb0 2023-10-30T13:35:14.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d01530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d01530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33432/0 (socket says 172.21.15.31:33432) 2023-10-30T13:35:14.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 -- 172.21.15.31:0/1412076735 learned_addr learned my addr 172.21.15.31:0/1412076735 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:14.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cf7fe640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d00c8eb0 0x7ff5d00cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d4b48640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5d015ac70 0x7ff5d00cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 -- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d00c8eb0 msgr2=0x7ff5d00cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d00c8eb0 0x7ff5d00cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 -- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5d015ac70 msgr2=0x7ff5d00cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5d015ac70 0x7ff5d00cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 -- 172.21.15.31:0/1412076735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c8009b30 con 0x7ff5d006aeb0 2023-10-30T13:35:14.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5cffff640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d01530e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000c530 tx=0x7ff5c000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5ccff9640 1 -- 172.21.15.31:0/1412076735 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0005220 con 0x7ff5d006aeb0 2023-10-30T13:35:14.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5ccff9640 1 -- 172.21.15.31:0/1412076735 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c0005380 con 0x7ff5d006aeb0 2023-10-30T13:35:14.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d01f3060 con 0x7ff5d006aeb0 2023-10-30T13:35:14.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5ccff9640 1 -- 172.21.15.31:0/1412076735 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0007270 con 0x7ff5d006aeb0 2023-10-30T13:35:14.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d0158a20 con 0x7ff5d006aeb0 2023-10-30T13:35:14.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d004e170 con 0x7ff5d006aeb0 2023-10-30T13:35:14.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.220+0000 7ff5ccff9640 1 -- 172.21.15.31:0/1412076735 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5c00073d0 con 0x7ff5d006aeb0 2023-10-30T13:35:14.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.224+0000 7ff5ccff9640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff5bc0664a0 0x7ff5bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.224+0000 7ff5ccff9640 1 -- 172.21.15.31:0/1412076735 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5c0083620 con 0x7ff5d006aeb0 2023-10-30T13:35:14.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.224+0000 7ff5cf7fe640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff5bc0664a0 0x7ff5bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.224+0000 7ff5cf7fe640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff5bc0664a0 0x7ff5bc068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0150c30 tx=0x7ff5c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.228+0000 7ff5ccff9640 1 -- 172.21.15.31:0/1412076735 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5c0050a90 con 0x7ff5d006aeb0 2023-10-30T13:35:14.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.384+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5d015d9e0 con 0x7ff5bc0664a0 2023-10-30T13:35:14.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5ccff9640 1 -- 172.21.15.31:0/1412076735 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264413 (secure 0 0 0) 0x7ff5c400be80 con 0x7ff5bc0664a0 2023-10-30T13:35:14.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:14.407 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:35:14.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff5bc0664a0 msgr2=0x7ff5bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff5bc0664a0 0x7ff5bc068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0150c30 tx=0x7ff5c4008040 comp rx=0 tx=0).stop 2023-10-30T13:35:14.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 msgr2=0x7ff5d01530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d01530e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000c530 tx=0x7ff5c000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 shutdown_connections 2023-10-30T13:35:14.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff5bc0664a0 0x7ff5bc068960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5d015ac70 0x7ff5d00cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff5d00c8eb0 0x7ff5d00cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 --2- 172.21.15.31:0/1412076735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d006aeb0 0x7ff5d01530e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.400+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 >> 172.21.15.31:0/1412076735 conn(0x7ff5d00cef00 msgr2=0x7ff5d00c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:14.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.404+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 shutdown_connections 2023-10-30T13:35:14.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.404+0000 7ff5d5349640 1 -- 172.21.15.31:0/1412076735 wait complete. 2023-10-30T13:35:14.426 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-30T13:35:13.156937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218896,"kb_used_data":2696,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755824,"statfs":{"total":773094113280,"available":772869963776,"internally_reserved":0,"allocated":2760704,"data_stored":1713144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002061"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808399+0000","last_change":"2023-10-30T13:35:07.792742+0000","last_active":"2023-10-30T13:35:07.808399+0000","last_peered":"2023-10-30T13:35:07.808399+0000","last_clean":"2023-10-30T13:35:07.808399+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:35:07.808399+0000","last_undegraded":"2023-10-30T13:35:07.808399+0000","last_fullsized":"2023-10-30T13:35:07.808399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:17:06.350045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744019+0000","last_change":"2023-10-30T13:35:07.755187+0000","last_active":"2023-10-30T13:35:08.744019+0000","last_peered":"2023-10-30T13:35:08.744019+0000","last_clean":"2023-10-30T13:35:08.744019+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:35:08.744019+0000","last_undegraded":"2023-10-30T13:35:08.744019+0000","last_fullsized":"2023-10-30T13:35:08.744019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:22.764808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756473+0000","last_change":"2023-10-30T13:35:07.754357+0000","last_active":"2023-10-30T13:35:07.756473+0000","last_peered":"2023-10-30T13:35:07.756473+0000","last_clean":"2023-10-30T13:35:07.756473+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:35:07.756473+0000","last_undegraded":"2023-10-30T13:35:07.756473+0000","last_fullsized":"2023-10-30T13:35:07.756473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:59.722272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.805049+0000","last_change":"2023-10-30T13:35:07.792466+0000","last_active":"2023-10-30T13:35:07.805049+0000","last_peered":"2023-10-30T13:35:07.805049+0000","last_clean":"2023-10-30T13:35:07.805049+0000","last_became_active":"2023-10-30T13:35:07.789749+0000","last_became_peered":"2023-10-30T13:35:07.789749+0000","last_unstale":"2023-10-30T13:35:07.805049+0000","last_undegraded":"2023-10-30T13:35:07.805049+0000","last_fullsized":"2023-10-30T13:35:07.805049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:48.000408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808438+0000","last_change":"2023-10-30T13:35:07.793403+0000","last_active":"2023-10-30T13:35:07.808438+0000","last_peered":"2023-10-30T13:35:07.808438+0000","last_clean":"2023-10-30T13:35:07.808438+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:35:07.808438+0000","last_undegraded":"2023-10-30T13:35:07.808438+0000","last_fullsized":"2023-10-30T13:35:07.808438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:40.294735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804990+0000","last_change":"2023-10-30T13:35:07.792285+0000","last_active":"2023-10-30T13:35:07.804990+0000","last_peered":"2023-10-30T13:35:07.804990+0000","last_clean":"2023-10-30T13:35:07.804990+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:35:07.804990+0000","last_undegraded":"2023-10-30T13:35:07.804990+0000","last_fullsized":"2023-10-30T13:35:07.804990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:41.177494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808204+0000","last_change":"2023-10-30T13:35:07.776386+0000","last_active":"2023-10-30T13:35:07.808204+0000","last_peered":"2023-10-30T13:35:07.808204+0000","last_clean":"2023-10-30T13:35:07.808204+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:35:07.808204+0000","last_undegraded":"2023-10-30T13:35:07.808204+0000","last_fullsized":"2023-10-30T13:35:07.808204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:03:21.846537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791396+0000","last_change":"2023-10-30T13:35:07.790487+0000","last_active":"2023-10-30T13:35:07.791396+0000","last_peered":"2023-10-30T13:35:07.791396+0000","last_clean":"2023-10-30T13:35:07.791396+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:35:07.791396+0000","last_undegraded":"2023-10-30T13:35:07.791396+0000","last_fullsized":"2023-10-30T13:35:07.791396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:43.765732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.810904+0000","last_change":"2023-10-30T13:35:07.804038+0000","last_active":"2023-10-30T13:35:07.810904+0000","last_peered":"2023-10-30T13:35:07.810904+0000","last_clean":"2023-10-30T13:35:07.810904+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:35:07.810904+0000","last_undegraded":"2023-10-30T13:35:07.810904+0000","last_fullsized":"2023-10-30T13:35:07.810904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:37:23.081000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.802735+0000","last_change":"2023-10-30T13:35:07.800657+0000","last_active":"2023-10-30T13:35:07.802735+0000","last_peered":"2023-10-30T13:35:07.802735+0000","last_clean":"2023-10-30T13:35:07.802735+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:35:07.802735+0000","last_undegraded":"2023-10-30T13:35:07.802735+0000","last_fullsized":"2023-10-30T13:35:07.802735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:15.007172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.805252+0000","last_change":"2023-10-30T13:35:07.802245+0000","last_active":"2023-10-30T13:35:07.805252+0000","last_peered":"2023-10-30T13:35:07.805252+0000","last_clean":"2023-10-30T13:35:07.805252+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:35:07.805252+0000","last_undegraded":"2023-10-30T13:35:07.805252+0000","last_fullsized":"2023-10-30T13:35:07.805252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:59.843727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.787751+0000","last_change":"2023-10-30T13:35:07.783940+0000","last_active":"2023-10-30T13:35:07.787751+0000","last_peered":"2023-10-30T13:35:07.787751+0000","last_clean":"2023-10-30T13:35:07.787751+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:35:07.787751+0000","last_undegraded":"2023-10-30T13:35:07.787751+0000","last_fullsized":"2023-10-30T13:35:07.787751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:36.232521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.781412+0000","last_change":"2023-10-30T13:35:07.780349+0000","last_active":"2023-10-30T13:35:07.781412+0000","last_peered":"2023-10-30T13:35:07.781412+0000","last_clean":"2023-10-30T13:35:07.781412+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:35:07.781412+0000","last_undegraded":"2023-10-30T13:35:07.781412+0000","last_fullsized":"2023-10-30T13:35:07.781412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:18.044765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.755460+0000","last_change":"2023-10-30T13:35:07.750959+0000","last_active":"2023-10-30T13:35:07.755460+0000","last_peered":"2023-10-30T13:35:07.755460+0000","last_clean":"2023-10-30T13:35:07.755460+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:35:07.755460+0000","last_undegraded":"2023-10-30T13:35:07.755460+0000","last_fullsized":"2023-10-30T13:35:07.755460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:59:01.565769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.768466+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:27.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804445+0000","last_change":"2023-10-30T13:35:07.789918+0000","last_active":"2023-10-30T13:35:07.804445+0000","last_peered":"2023-10-30T13:35:07.804445+0000","last_clean":"2023-10-30T13:35:07.804445+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:35:07.804445+0000","last_undegraded":"2023-10-30T13:35:07.804445+0000","last_fullsized":"2023-10-30T13:35:07.804445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:29.496632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.805766+0000","last_change":"2023-10-30T13:35:07.796429+0000","last_active":"2023-10-30T13:35:07.805766+0000","last_peered":"2023-10-30T13:35:07.805766+0000","last_clean":"2023-10-30T13:35:07.805766+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:35:07.805766+0000","last_undegraded":"2023-10-30T13:35:07.805766+0000","last_fullsized":"2023-10-30T13:35:07.805766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:04.970161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.781097+0000","last_change":"2023-10-30T13:35:07.780131+0000","last_active":"2023-10-30T13:35:07.781097+0000","last_peered":"2023-10-30T13:35:07.781097+0000","last_clean":"2023-10-30T13:35:07.781097+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:35:07.781097+0000","last_undegraded":"2023-10-30T13:35:07.781097+0000","last_fullsized":"2023-10-30T13:35:07.781097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:21.748687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808301+0000","last_change":"2023-10-30T13:35:07.791464+0000","last_active":"2023-10-30T13:35:07.808301+0000","last_peered":"2023-10-30T13:35:07.808301+0000","last_clean":"2023-10-30T13:35:07.808301+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:35:07.808301+0000","last_undegraded":"2023-10-30T13:35:07.808301+0000","last_fullsized":"2023-10-30T13:35:07.808301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:29:21.519846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.789106+0000","last_change":"2023-10-30T13:35:07.783079+0000","last_active":"2023-10-30T13:35:07.789106+0000","last_peered":"2023-10-30T13:35:07.789106+0000","last_clean":"2023-10-30T13:35:07.789106+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:35:07.789106+0000","last_undegraded":"2023-10-30T13:35:07.789106+0000","last_fullsized":"2023-10-30T13:35:07.789106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:17:06.190093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.801890+0000","last_change":"2023-10-30T13:35:07.794666+0000","last_active":"2023-10-30T13:35:07.801890+0000","last_peered":"2023-10-30T13:35:07.801890+0000","last_clean":"2023-10-30T13:35:07.801890+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:35:07.801890+0000","last_undegraded":"2023-10-30T13:35:07.801890+0000","last_fullsized":"2023-10-30T13:35:07.801890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:42.312392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.806985+0000","last_change":"2023-10-30T13:35:07.796601+0000","last_active":"2023-10-30T13:35:07.806985+0000","last_peered":"2023-10-30T13:35:07.806985+0000","last_clean":"2023-10-30T13:35:07.806985+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:35:07.806985+0000","last_undegraded":"2023-10-30T13:35:07.806985+0000","last_fullsized":"2023-10-30T13:35:07.806985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:53.943593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.769377+0000","last_change":"2023-10-30T13:35:07.767965+0000","last_active":"2023-10-30T13:35:07.769377+0000","last_peered":"2023-10-30T13:35:07.769377+0000","last_clean":"2023-10-30T13:35:07.769377+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:35:07.769377+0000","last_undegraded":"2023-10-30T13:35:07.769377+0000","last_fullsized":"2023-10-30T13:35:07.769377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:10:00.028378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.788967+0000","last_change":"2023-10-30T13:35:07.772029+0000","last_active":"2023-10-30T13:35:07.788967+0000","last_peered":"2023-10-30T13:35:07.788967+0000","last_clean":"2023-10-30T13:35:07.788967+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:35:07.788967+0000","last_undegraded":"2023-10-30T13:35:07.788967+0000","last_fullsized":"2023-10-30T13:35:07.788967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:13.735624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.759188+0000","last_change":"2023-10-30T13:35:07.758274+0000","last_active":"2023-10-30T13:35:07.759188+0000","last_peered":"2023-10-30T13:35:07.759188+0000","last_clean":"2023-10-30T13:35:07.759188+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:35:07.759188+0000","last_undegraded":"2023-10-30T13:35:07.759188+0000","last_fullsized":"2023-10-30T13:35:07.759188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:04.233120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.807364+0000","last_change":"2023-10-30T13:35:07.803220+0000","last_active":"2023-10-30T13:35:07.807364+0000","last_peered":"2023-10-30T13:35:07.807364+0000","last_clean":"2023-10-30T13:35:07.807364+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:35:07.807364+0000","last_undegraded":"2023-10-30T13:35:07.807364+0000","last_fullsized":"2023-10-30T13:35:07.807364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:50.524288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743680+0000","last_change":"2023-10-30T13:35:07.753288+0000","last_active":"2023-10-30T13:35:08.743680+0000","last_peered":"2023-10-30T13:35:08.743680+0000","last_clean":"2023-10-30T13:35:08.743680+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:35:08.743680+0000","last_undegraded":"2023-10-30T13:35:08.743680+0000","last_fullsized":"2023-10-30T13:35:08.743680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:43.815379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.769395+0000","last_change":"2023-10-30T13:35:07.767931+0000","last_active":"2023-10-30T13:35:07.769395+0000","last_peered":"2023-10-30T13:35:07.769395+0000","last_clean":"2023-10-30T13:35:07.769395+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:35:07.769395+0000","last_undegraded":"2023-10-30T13:35:07.769395+0000","last_fullsized":"2023-10-30T13:35:07.769395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:09.828603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756607+0000","last_change":"2023-10-30T13:35:07.753695+0000","last_active":"2023-10-30T13:35:07.756607+0000","last_peered":"2023-10-30T13:35:07.756607+0000","last_clean":"2023-10-30T13:35:07.756607+0000","last_became_active":"2023-10-30T13:35:07.753358+0000","last_became_peered":"2023-10-30T13:35:07.753358+0000","last_unstale":"2023-10-30T13:35:07.756607+0000","last_undegraded":"2023-10-30T13:35:07.756607+0000","last_fullsized":"2023-10-30T13:35:07.756607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:03:35.697267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804596+0000","last_change":"2023-10-30T13:35:07.790236+0000","last_active":"2023-10-30T13:35:07.804596+0000","last_peered":"2023-10-30T13:35:07.804596+0000","last_clean":"2023-10-30T13:35:07.804596+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:35:07.804596+0000","last_undegraded":"2023-10-30T13:35:07.804596+0000","last_fullsized":"2023-10-30T13:35:07.804596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:48.035840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791500+0000","last_change":"2023-10-30T13:35:07.790720+0000","last_active":"2023-10-30T13:35:07.791500+0000","last_peered":"2023-10-30T13:35:07.791500+0000","last_clean":"2023-10-30T13:35:07.791500+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:35:07.791500+0000","last_undegraded":"2023-10-30T13:35:07.791500+0000","last_fullsized":"2023-10-30T13:35:07.791500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:49:00.265087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.789161+0000","last_change":"2023-10-30T13:35:07.780576+0000","last_active":"2023-10-30T13:35:07.789161+0000","last_peered":"2023-10-30T13:35:07.789161+0000","last_clean":"2023-10-30T13:35:07.789161+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:35:07.789161+0000","last_undegraded":"2023-10-30T13:35:07.789161+0000","last_fullsized":"2023-10-30T13:35:07.789161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:33.677737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.761724+0000","last_change":"2023-10-30T13:35:07.755919+0000","last_active":"2023-10-30T13:35:07.761724+0000","last_peered":"2023-10-30T13:35:07.761724+0000","last_clean":"2023-10-30T13:35:07.761724+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:35:07.761724+0000","last_undegraded":"2023-10-30T13:35:07.761724+0000","last_fullsized":"2023-10-30T13:35:07.761724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:00.465200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.748399+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:35:07.748399+0000","last_peered":"2023-10-30T13:35:07.748399+0000","last_clean":"2023-10-30T13:35:07.748399+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:35:07.748399+0000","last_undegraded":"2023-10-30T13:35:07.748399+0000","last_fullsized":"2023-10-30T13:35:07.748399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.763593+0000","last_change":"2023-10-30T13:35:07.762428+0000","last_active":"2023-10-30T13:35:07.763593+0000","last_peered":"2023-10-30T13:35:07.763593+0000","last_clean":"2023-10-30T13:35:07.763593+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:35:07.763593+0000","last_undegraded":"2023-10-30T13:35:07.763593+0000","last_fullsized":"2023-10-30T13:35:07.763593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:17.453197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.763104+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:35:07.763104+0000","last_peered":"2023-10-30T13:35:07.763104+0000","last_clean":"2023-10-30T13:35:07.763104+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:35:07.763104+0000","last_undegraded":"2023-10-30T13:35:07.763104+0000","last_fullsized":"2023-10-30T13:35:07.763104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.746252+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:35:07.746252+0000","last_peered":"2023-10-30T13:35:07.746252+0000","last_clean":"2023-10-30T13:35:07.746252+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:35:07.746252+0000","last_undegraded":"2023-10-30T13:35:07.746252+0000","last_fullsized":"2023-10-30T13:35:07.746252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.754244+0000","last_change":"2023-10-30T13:35:07.751131+0000","last_active":"2023-10-30T13:35:07.754244+0000","last_peered":"2023-10-30T13:35:07.754244+0000","last_clean":"2023-10-30T13:35:07.754244+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:35:07.754244+0000","last_undegraded":"2023-10-30T13:35:07.754244+0000","last_fullsized":"2023-10-30T13:35:07.754244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:27:03.270510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756967+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:35:07.756967+0000","last_peered":"2023-10-30T13:35:07.756967+0000","last_clean":"2023-10-30T13:35:07.756967+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:35:07.756967+0000","last_undegraded":"2023-10-30T13:35:07.756967+0000","last_fullsized":"2023-10-30T13:35:07.756967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743438+0000","last_change":"2023-10-30T13:35:07.750392+0000","last_active":"2023-10-30T13:35:08.743438+0000","last_peered":"2023-10-30T13:35:08.743438+0000","last_clean":"2023-10-30T13:35:08.743438+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:35:08.743438+0000","last_undegraded":"2023-10-30T13:35:08.743438+0000","last_fullsized":"2023-10-30T13:35:08.743438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:33.554754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.753242+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:35:07.753242+0000","last_peered":"2023-10-30T13:35:07.753242+0000","last_clean":"2023-10-30T13:35:07.753242+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:35:07.753242+0000","last_undegraded":"2023-10-30T13:35:07.753242+0000","last_fullsized":"2023-10-30T13:35:07.753242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.807074+0000","last_change":"2023-10-30T13:35:07.797513+0000","last_active":"2023-10-30T13:35:07.807074+0000","last_peered":"2023-10-30T13:35:07.807074+0000","last_clean":"2023-10-30T13:35:07.807074+0000","last_became_active":"2023-10-30T13:35:07.796748+0000","last_became_peered":"2023-10-30T13:35:07.796748+0000","last_unstale":"2023-10-30T13:35:07.807074+0000","last_undegraded":"2023-10-30T13:35:07.807074+0000","last_fullsized":"2023-10-30T13:35:07.807074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:39.151161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756154+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:35:07.756154+0000","last_peered":"2023-10-30T13:35:07.756154+0000","last_clean":"2023-10-30T13:35:07.756154+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:35:07.756154+0000","last_undegraded":"2023-10-30T13:35:07.756154+0000","last_fullsized":"2023-10-30T13:35:07.756154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791718+0000","last_change":"2023-10-30T13:35:07.789983+0000","last_active":"2023-10-30T13:35:07.791718+0000","last_peered":"2023-10-30T13:35:07.791718+0000","last_clean":"2023-10-30T13:35:07.791718+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:35:07.791718+0000","last_undegraded":"2023-10-30T13:35:07.791718+0000","last_fullsized":"2023-10-30T13:35:07.791718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:40:26.139549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743813+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:35:08.743813+0000","last_peered":"2023-10-30T13:35:08.743813+0000","last_clean":"2023-10-30T13:35:08.743813+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:35:08.743813+0000","last_undegraded":"2023-10-30T13:35:08.743813+0000","last_fullsized":"2023-10-30T13:35:08.743813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.803657+0000","last_change":"2023-10-30T13:35:07.794930+0000","last_active":"2023-10-30T13:35:07.803657+0000","last_peered":"2023-10-30T13:35:07.803657+0000","last_clean":"2023-10-30T13:35:07.803657+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:35:07.803657+0000","last_undegraded":"2023-10-30T13:35:07.803657+0000","last_fullsized":"2023-10-30T13:35:07.803657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:24.359886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.764306+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:35:07.764306+0000","last_peered":"2023-10-30T13:35:07.764306+0000","last_clean":"2023-10-30T13:35:07.764306+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:35:07.764306+0000","last_undegraded":"2023-10-30T13:35:07.764306+0000","last_fullsized":"2023-10-30T13:35:07.764306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808355+0000","last_change":"2023-10-30T13:35:07.791923+0000","last_active":"2023-10-30T13:35:07.808355+0000","last_peered":"2023-10-30T13:35:07.808355+0000","last_clean":"2023-10-30T13:35:07.808355+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:35:07.808355+0000","last_undegraded":"2023-10-30T13:35:07.808355+0000","last_fullsized":"2023-10-30T13:35:07.808355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:23.544401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.746223+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:35:07.746223+0000","last_peered":"2023-10-30T13:35:07.746223+0000","last_clean":"2023-10-30T13:35:07.746223+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:35:07.746223+0000","last_undegraded":"2023-10-30T13:35:07.746223+0000","last_fullsized":"2023-10-30T13:35:07.746223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.801902+0000","last_change":"2023-10-30T13:35:07.794790+0000","last_active":"2023-10-30T13:35:07.801902+0000","last_peered":"2023-10-30T13:35:07.801902+0000","last_clean":"2023-10-30T13:35:07.801902+0000","last_became_active":"2023-10-30T13:35:07.794477+0000","last_became_peered":"2023-10-30T13:35:07.794477+0000","last_unstale":"2023-10-30T13:35:07.801902+0000","last_undegraded":"2023-10-30T13:35:07.801902+0000","last_fullsized":"2023-10-30T13:35:07.801902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:47:29.816491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.771810+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:35:07.771810+0000","last_peered":"2023-10-30T13:35:07.771810+0000","last_clean":"2023-10-30T13:35:07.771810+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:35:07.771810+0000","last_undegraded":"2023-10-30T13:35:07.771810+0000","last_fullsized":"2023-10-30T13:35:07.771810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808744+0000","last_change":"2023-10-30T13:35:07.797247+0000","last_active":"2023-10-30T13:35:07.808744+0000","last_peered":"2023-10-30T13:35:07.808744+0000","last_clean":"2023-10-30T13:35:07.808744+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:35:07.808744+0000","last_undegraded":"2023-10-30T13:35:07.808744+0000","last_fullsized":"2023-10-30T13:35:07.808744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:46:21.871813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.754609+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:35:07.754609+0000","last_peered":"2023-10-30T13:35:07.754609+0000","last_clean":"2023-10-30T13:35:07.754609+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:35:07.754609+0000","last_undegraded":"2023-10-30T13:35:07.754609+0000","last_fullsized":"2023-10-30T13:35:07.754609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.781957+0000","last_change":"2023-10-30T13:35:07.773683+0000","last_active":"2023-10-30T13:35:07.781957+0000","last_peered":"2023-10-30T13:35:07.781957+0000","last_clean":"2023-10-30T13:35:07.781957+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:35:07.781957+0000","last_undegraded":"2023-10-30T13:35:07.781957+0000","last_fullsized":"2023-10-30T13:35:07.781957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:20.750111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.744464+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:35:07.744464+0000","last_peered":"2023-10-30T13:35:07.744464+0000","last_clean":"2023-10-30T13:35:07.744464+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:35:07.744464+0000","last_undegraded":"2023-10-30T13:35:07.744464+0000","last_fullsized":"2023-10-30T13:35:07.744464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.810692+0000","last_change":"2023-10-30T13:35:07.791266+0000","last_active":"2023-10-30T13:35:07.810692+0000","last_peered":"2023-10-30T13:35:07.810692+0000","last_clean":"2023-10-30T13:35:07.810692+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:35:07.810692+0000","last_undegraded":"2023-10-30T13:35:07.810692+0000","last_fullsized":"2023-10-30T13:35:07.810692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:23:01.624384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768918+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:35:07.768918+0000","last_peered":"2023-10-30T13:35:07.768918+0000","last_clean":"2023-10-30T13:35:07.768918+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:35:07.768918+0000","last_undegraded":"2023-10-30T13:35:07.768918+0000","last_fullsized":"2023-10-30T13:35:07.768918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808484+0000","last_change":"2023-10-30T13:35:07.794747+0000","last_active":"2023-10-30T13:35:07.808484+0000","last_peered":"2023-10-30T13:35:07.808484+0000","last_clean":"2023-10-30T13:35:07.808484+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:35:07.808484+0000","last_undegraded":"2023-10-30T13:35:07.808484+0000","last_fullsized":"2023-10-30T13:35:07.808484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:31.479675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.745728+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:35:07.745728+0000","last_peered":"2023-10-30T13:35:07.745728+0000","last_clean":"2023-10-30T13:35:07.745728+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:35:07.745728+0000","last_undegraded":"2023-10-30T13:35:07.745728+0000","last_fullsized":"2023-10-30T13:35:07.745728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808438+0000","last_change":"2023-10-30T13:35:07.793255+0000","last_active":"2023-10-30T13:35:07.808438+0000","last_peered":"2023-10-30T13:35:07.808438+0000","last_clean":"2023-10-30T13:35:07.808438+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:35:07.808438+0000","last_undegraded":"2023-10-30T13:35:07.808438+0000","last_fullsized":"2023-10-30T13:35:07.808438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:36.854385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743869+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:35:08.743869+0000","last_peered":"2023-10-30T13:35:08.743869+0000","last_clean":"2023-10-30T13:35:08.743869+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:35:08.743869+0000","last_undegraded":"2023-10-30T13:35:08.743869+0000","last_fullsized":"2023-10-30T13:35:08.743869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804649+0000","last_change":"2023-10-30T13:35:07.791920+0000","last_active":"2023-10-30T13:35:07.804649+0000","last_peered":"2023-10-30T13:35:07.804649+0000","last_clean":"2023-10-30T13:35:07.804649+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:35:07.804649+0000","last_undegraded":"2023-10-30T13:35:07.804649+0000","last_fullsized":"2023-10-30T13:35:07.804649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:58.172166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.746443+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:35:07.746443+0000","last_peered":"2023-10-30T13:35:07.746443+0000","last_clean":"2023-10-30T13:35:07.746443+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:35:07.746443+0000","last_undegraded":"2023-10-30T13:35:07.746443+0000","last_fullsized":"2023-10-30T13:35:07.746443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.810851+0000","last_change":"2023-10-30T13:35:07.803211+0000","last_active":"2023-10-30T13:35:07.810851+0000","last_peered":"2023-10-30T13:35:07.810851+0000","last_clean":"2023-10-30T13:35:07.810851+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:35:07.810851+0000","last_undegraded":"2023-10-30T13:35:07.810851+0000","last_fullsized":"2023-10-30T13:35:07.810851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:34:13.535807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.289348+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:35:07.289348+0000","last_peered":"2023-10-30T13:35:07.289348+0000","last_clean":"2023-10-30T13:35:07.289348+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:35:07.289348+0000","last_undegraded":"2023-10-30T13:35:07.289348+0000","last_fullsized":"2023-10-30T13:35:07.289348+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.749763+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:35:07.749763+0000","last_peered":"2023-10-30T13:35:07.749763+0000","last_clean":"2023-10-30T13:35:07.749763+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:35:07.749763+0000","last_undegraded":"2023-10-30T13:35:07.749763+0000","last_fullsized":"2023-10-30T13:35:07.749763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.796651+0000","last_change":"2023-10-30T13:35:07.792082+0000","last_active":"2023-10-30T13:35:07.796651+0000","last_peered":"2023-10-30T13:35:07.796651+0000","last_clean":"2023-10-30T13:35:07.796651+0000","last_became_active":"2023-10-30T13:35:07.791831+0000","last_became_peered":"2023-10-30T13:35:07.791831+0000","last_unstale":"2023-10-30T13:35:07.796651+0000","last_undegraded":"2023-10-30T13:35:07.796651+0000","last_fullsized":"2023-10-30T13:35:07.796651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:58:35.766504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768549+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:35:07.768549+0000","last_peered":"2023-10-30T13:35:07.768549+0000","last_clean":"2023-10-30T13:35:07.768549+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:35:07.768549+0000","last_undegraded":"2023-10-30T13:35:07.768549+0000","last_fullsized":"2023-10-30T13:35:07.768549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.761377+0000","last_change":"2023-10-30T13:35:07.760132+0000","last_active":"2023-10-30T13:35:07.761377+0000","last_peered":"2023-10-30T13:35:07.761377+0000","last_clean":"2023-10-30T13:35:07.761377+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:35:07.761377+0000","last_undegraded":"2023-10-30T13:35:07.761377+0000","last_fullsized":"2023-10-30T13:35:07.761377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:58.458839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.745502+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:35:07.745502+0000","last_peered":"2023-10-30T13:35:07.745502+0000","last_clean":"2023-10-30T13:35:07.745502+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:35:07.745502+0000","last_undegraded":"2023-10-30T13:35:07.745502+0000","last_fullsized":"2023-10-30T13:35:07.745502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.750958+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:53.697186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.772257+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:35:07.772257+0000","last_peered":"2023-10-30T13:35:07.772257+0000","last_clean":"2023-10-30T13:35:07.772257+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:35:07.772257+0000","last_undegraded":"2023-10-30T13:35:07.772257+0000","last_fullsized":"2023-10-30T13:35:07.772257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.782470+0000","last_change":"2023-10-30T13:35:07.768640+0000","last_active":"2023-10-30T13:35:07.782470+0000","last_peered":"2023-10-30T13:35:07.782470+0000","last_clean":"2023-10-30T13:35:07.782470+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:35:07.782470+0000","last_undegraded":"2023-10-30T13:35:07.782470+0000","last_fullsized":"2023-10-30T13:35:07.782470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:57.460587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743612+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:35:08.743612+0000","last_peered":"2023-10-30T13:35:08.743612+0000","last_clean":"2023-10-30T13:35:08.743612+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:35:08.743612+0000","last_undegraded":"2023-10-30T13:35:08.743612+0000","last_fullsized":"2023-10-30T13:35:08.743612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808749+0000","last_change":"2023-10-30T13:35:07.805535+0000","last_active":"2023-10-30T13:35:07.808749+0000","last_peered":"2023-10-30T13:35:07.808749+0000","last_clean":"2023-10-30T13:35:07.808749+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:35:07.808749+0000","last_undegraded":"2023-10-30T13:35:07.808749+0000","last_fullsized":"2023-10-30T13:35:07.808749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:48.167900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.766088+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:35:07.766088+0000","last_peered":"2023-10-30T13:35:07.766088+0000","last_clean":"2023-10-30T13:35:07.766088+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:35:07.766088+0000","last_undegraded":"2023-10-30T13:35:07.766088+0000","last_fullsized":"2023-10-30T13:35:07.766088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743474+0000","last_change":"2023-10-30T13:35:07.754171+0000","last_active":"2023-10-30T13:35:08.743474+0000","last_peered":"2023-10-30T13:35:08.743474+0000","last_clean":"2023-10-30T13:35:08.743474+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:35:08.743474+0000","last_undegraded":"2023-10-30T13:35:08.743474+0000","last_fullsized":"2023-10-30T13:35:08.743474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:50.849695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744041+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:35:08.744041+0000","last_peered":"2023-10-30T13:35:08.744041+0000","last_clean":"2023-10-30T13:35:08.744041+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:35:08.744041+0000","last_undegraded":"2023-10-30T13:35:08.744041+0000","last_fullsized":"2023-10-30T13:35:08.744041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.801917+0000","last_change":"2023-10-30T13:35:07.794804+0000","last_active":"2023-10-30T13:35:07.801917+0000","last_peered":"2023-10-30T13:35:07.801917+0000","last_clean":"2023-10-30T13:35:07.801917+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:35:07.801917+0000","last_undegraded":"2023-10-30T13:35:07.801917+0000","last_fullsized":"2023-10-30T13:35:07.801917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:45:13.550352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.753319+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:35:07.753319+0000","last_peered":"2023-10-30T13:35:07.753319+0000","last_clean":"2023-10-30T13:35:07.753319+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:35:07.753319+0000","last_undegraded":"2023-10-30T13:35:07.753319+0000","last_fullsized":"2023-10-30T13:35:07.753319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.755643+0000","last_change":"2023-10-30T13:35:07.752077+0000","last_active":"2023-10-30T13:35:07.755643+0000","last_peered":"2023-10-30T13:35:07.755643+0000","last_clean":"2023-10-30T13:35:07.755643+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:35:07.755643+0000","last_undegraded":"2023-10-30T13:35:07.755643+0000","last_fullsized":"2023-10-30T13:35:07.755643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:39.134825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.767414+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:35:07.767414+0000","last_peered":"2023-10-30T13:35:07.767414+0000","last_clean":"2023-10-30T13:35:07.767414+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:35:07.767414+0000","last_undegraded":"2023-10-30T13:35:07.767414+0000","last_fullsized":"2023-10-30T13:35:07.767414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.766791+0000","last_change":"2023-10-30T13:35:07.759389+0000","last_active":"2023-10-30T13:35:07.766791+0000","last_peered":"2023-10-30T13:35:07.766791+0000","last_clean":"2023-10-30T13:35:07.766791+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:35:07.766791+0000","last_undegraded":"2023-10-30T13:35:07.766791+0000","last_fullsized":"2023-10-30T13:35:07.766791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:48:06.413218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743540+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:35:08.743540+0000","last_peered":"2023-10-30T13:35:08.743540+0000","last_clean":"2023-10-30T13:35:08.743540+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:35:08.743540+0000","last_undegraded":"2023-10-30T13:35:08.743540+0000","last_fullsized":"2023-10-30T13:35:08.743540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.757489+0000","last_change":"2023-10-30T13:35:07.753673+0000","last_active":"2023-10-30T13:35:07.757489+0000","last_peered":"2023-10-30T13:35:07.757489+0000","last_clean":"2023-10-30T13:35:07.757489+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:35:07.757489+0000","last_undegraded":"2023-10-30T13:35:07.757489+0000","last_fullsized":"2023-10-30T13:35:07.757489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:08:30.823529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743863+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:35:08.743863+0000","last_peered":"2023-10-30T13:35:08.743863+0000","last_clean":"2023-10-30T13:35:08.743863+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:35:08.743863+0000","last_undegraded":"2023-10-30T13:35:08.743863+0000","last_fullsized":"2023-10-30T13:35:08.743863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.757579+0000","last_change":"2023-10-30T13:35:07.755302+0000","last_active":"2023-10-30T13:35:07.757579+0000","last_peered":"2023-10-30T13:35:07.757579+0000","last_clean":"2023-10-30T13:35:07.757579+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:35:07.757579+0000","last_undegraded":"2023-10-30T13:35:07.757579+0000","last_fullsized":"2023-10-30T13:35:07.757579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:06:03.470605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.767574+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:35:07.767574+0000","last_peered":"2023-10-30T13:35:07.767574+0000","last_clean":"2023-10-30T13:35:07.767574+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:35:07.767574+0000","last_undegraded":"2023-10-30T13:35:07.767574+0000","last_fullsized":"2023-10-30T13:35:07.767574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.786227+0000","last_change":"2023-10-30T13:35:07.780297+0000","last_active":"2023-10-30T13:35:07.786227+0000","last_peered":"2023-10-30T13:35:07.786227+0000","last_clean":"2023-10-30T13:35:07.786227+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:35:07.786227+0000","last_undegraded":"2023-10-30T13:35:07.786227+0000","last_fullsized":"2023-10-30T13:35:07.786227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:20:12.682537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808790+0000","last_change":"2023-10-30T13:35:07.806203+0000","last_active":"2023-10-30T13:35:07.808790+0000","last_peered":"2023-10-30T13:35:07.808790+0000","last_clean":"2023-10-30T13:35:07.808790+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:35:07.808790+0000","last_undegraded":"2023-10-30T13:35:07.808790+0000","last_fullsized":"2023-10-30T13:35:07.808790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:53.855040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.759765+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:35:07.759765+0000","last_peered":"2023-10-30T13:35:07.759765+0000","last_clean":"2023-10-30T13:35:07.759765+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:35:07.759765+0000","last_undegraded":"2023-10-30T13:35:07.759765+0000","last_fullsized":"2023-10-30T13:35:07.759765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804522+0000","last_change":"2023-10-30T13:35:07.791319+0000","last_active":"2023-10-30T13:35:07.804522+0000","last_peered":"2023-10-30T13:35:07.804522+0000","last_clean":"2023-10-30T13:35:07.804522+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:35:07.804522+0000","last_undegraded":"2023-10-30T13:35:07.804522+0000","last_fullsized":"2023-10-30T13:35:07.804522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:57:23.365203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.764665+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:35:07.764665+0000","last_peered":"2023-10-30T13:35:07.764665+0000","last_clean":"2023-10-30T13:35:07.764665+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:35:07.764665+0000","last_undegraded":"2023-10-30T13:35:07.764665+0000","last_fullsized":"2023-10-30T13:35:07.764665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791738+0000","last_change":"2023-10-30T13:35:07.790084+0000","last_active":"2023-10-30T13:35:07.791738+0000","last_peered":"2023-10-30T13:35:07.791738+0000","last_clean":"2023-10-30T13:35:07.791738+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:35:07.791738+0000","last_undegraded":"2023-10-30T13:35:07.791738+0000","last_fullsized":"2023-10-30T13:35:07.791738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:32:22.844026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.748210+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:35:07.748210+0000","last_peered":"2023-10-30T13:35:07.748210+0000","last_clean":"2023-10-30T13:35:07.748210+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:35:07.748210+0000","last_undegraded":"2023-10-30T13:35:07.748210+0000","last_fullsized":"2023-10-30T13:35:07.748210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.784143+0000","last_change":"2023-10-30T13:35:07.782341+0000","last_active":"2023-10-30T13:35:07.784143+0000","last_peered":"2023-10-30T13:35:07.784143+0000","last_clean":"2023-10-30T13:35:07.784143+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:35:07.784143+0000","last_undegraded":"2023-10-30T13:35:07.784143+0000","last_fullsized":"2023-10-30T13:35:07.784143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:08.779946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.744301+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:35:07.744301+0000","last_peered":"2023-10-30T13:35:07.744301+0000","last_clean":"2023-10-30T13:35:07.744301+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:35:07.744301+0000","last_undegraded":"2023-10-30T13:35:07.744301+0000","last_fullsized":"2023-10-30T13:35:07.744301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":656919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":656919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:35:14.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:35:14.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd3566d6640 1 Processor -- start 2023-10-30T13:35:14.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd3566d6640 1 -- start start 2023-10-30T13:35:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd3566d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd350153c80 0x7fd35015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd3566d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 0x7fd3500c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd3566d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd350159f20 con 0x7fd35015a3b0 2023-10-30T13:35:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd3566d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35015dec0 con 0x7fd350153c80 2023-10-30T13:35:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd3566d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3501599a0 con 0x7fd3500c4a10 2023-10-30T13:35:14.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd354c73640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fd35015a3b0 0x7fd3500ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45080/0 (socket says 172.21.15.31:45080) 2023-10-30T13:35:14.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd354c73640 1 -- 172.21.15.31:0/3538507900 learned_addr learned my addr 172.21.15.31:0/3538507900 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:14.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 0x7fd3500c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.612+0000 7fd34ffff640 1 --2- 172.21.15.31:0/3538507900 >> v2:172.21.15.33:3300/0 conn(0x7fd350153c80 0x7fd35015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201354279 0 0) 0x7fd350159f20 con 0x7fd35015a3b0 2023-10-30T13:35:14.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338003530 con 0x7fd35015a3b0 2023-10-30T13:35:14.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 295293467 0 0) 0x7fd338003530 con 0x7fd35015a3b0 2023-10-30T13:35:14.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 msgr2=0x7fd3500c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 0x7fd3500c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 >> v2:172.21.15.33:3300/0 conn(0x7fd350153c80 msgr2=0x7fd35015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 --2- 172.21.15.31:0/3538507900 >> v2:172.21.15.33:3300/0 conn(0x7fd350153c80 0x7fd35015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500c47e0 con 0x7fd35015a3b0 2023-10-30T13:35:14.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fd348001a00 con 0x7fd35015a3b0 2023-10-30T13:35:14.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 >> v1:172.21.15.31:6789/0 conn(0x7fd35015a3b0 legacy=0x7fd3500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd338003980 0x7fd338003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3380047d0 0x7fd338004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 0x7fd338005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35015dec0 con 0x7fd3380047d0 2023-10-30T13:35:14.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3501599a0 con 0x7fd338003980 2023-10-30T13:35:14.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33800b990 con 0x7fd3500c4a10 2023-10-30T13:35:14.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3380047d0 0x7fd338004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd354c73640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 0x7fd338005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34ffff640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd338003980 0x7fd338003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34f7fe640 1 -- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 msgr2=0x7fd338005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3500c4a10 0x7fd338005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34f7fe640 1 -- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd338003980 msgr2=0x7fd338003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd338003980 0x7fd338003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.616+0000 7fd34f7fe640 1 -- 172.21.15.31:0/3538507900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338003530 con 0x7fd3380047d0 2023-10-30T13:35:14.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3380047d0 0x7fd338004c50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ab0 tx=0x7fd34401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d7ffc89a574447 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a5540 con 0x7fd3380047d0 2023-10-30T13:35:14.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3440a56a0 con 0x7fd3380047d0 2023-10-30T13:35:14.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd34effd640 1 -- 172.21.15.31:0/3538507900 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a5a50 con 0x7fd3380047d0 2023-10-30T13:35:14.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- 172.21.15.31:0/3538507900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3380047d0 msgr2=0x7fd338004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 --2- 172.21.15.31:0/3538507900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3380047d0 0x7fd338004c50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ab0 tx=0x7fd34401ca30 comp rx=0 tx=0).stop 2023-10-30T13:35:14.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd354c73640 1 -- 172.21.15.31:0/3538507900 reap_dead start 2023-10-30T13:35:14.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- 172.21.15.31:0/3538507900 shutdown_connections 2023-10-30T13:35:14.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- 172.21.15.31:0/3538507900 >> 172.21.15.31:0/3538507900 conn(0x7fd3500715c0 msgr2=0x7fd3500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:14.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- 172.21.15.31:0/3538507900 shutdown_connections 2023-10-30T13:35:14.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- 172.21.15.31:0/3538507900 wait complete. 2023-10-30T13:35:14.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 Processor -- start 2023-10-30T13:35:14.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- start start 2023-10-30T13:35:14.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3380047d0 0x7fd35014d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3500cefa0 0x7fd3500cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3501eebc0 0x7fd3501ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35014ae70 con 0x7fd3500cefa0 2023-10-30T13:35:14.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500cd600 con 0x7fd3380047d0 2023-10-30T13:35:14.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.620+0000 7fd3566d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35014b8f0 con 0x7fd3501eebc0 2023-10-30T13:35:14.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3500cefa0 0x7fd3500cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3500cefa0 0x7fd3500cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33452/0 (socket says 172.21.15.31:33452) 2023-10-30T13:35:14.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 -- 172.21.15.31:0/1253479192 learned_addr learned my addr 172.21.15.31:0/1253479192 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:14.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34ffff640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3380047d0 0x7fd35014d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd354c73640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3501eebc0 0x7fd3501ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 -- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3501eebc0 msgr2=0x7fd3501ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3501eebc0 0x7fd3501ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 -- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3380047d0 msgr2=0x7fd35014d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3380047d0 0x7fd35014d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 -- 172.21.15.31:0/1253479192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500cddd0 con 0x7fd3500cefa0 2023-10-30T13:35:14.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34f7fe640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3500cefa0 0x7fd3500cf3e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ab0 tx=0x7fd338005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34cff9640 1 -- 172.21.15.31:0/1253479192 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440bc070 con 0x7fd3500cefa0 2023-10-30T13:35:14.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34cff9640 1 -- 172.21.15.31:0/1253479192 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd344004020 con 0x7fd3500cefa0 2023-10-30T13:35:14.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34cff9640 1 -- 172.21.15.31:0/1253479192 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344004390 con 0x7fd3500cefa0 2023-10-30T13:35:14.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3500cf920 con 0x7fd3500cefa0 2023-10-30T13:35:14.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3500cd320 con 0x7fd3500cefa0 2023-10-30T13:35:14.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35004e170 con 0x7fd3500cefa0 2023-10-30T13:35:14.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.624+0000 7fd34cff9640 1 -- 172.21.15.31:0/1253479192 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3440b4040 con 0x7fd3500cefa0 2023-10-30T13:35:14.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.628+0000 7fd34cff9640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd33c0664a0 0x7fd33c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.628+0000 7fd34cff9640 1 -- 172.21.15.31:0/1253479192 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd34411d8d0 con 0x7fd3500cefa0 2023-10-30T13:35:14.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.628+0000 7fd34ffff640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd33c0664a0 0x7fd33c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.628+0000 7fd34ffff640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd33c0664a0 0x7fd33c068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd340009b60 tx=0x7fd340008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.628+0000 7fd34cff9640 1 -- 172.21.15.31:0/1253479192 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3440eafb0 con 0x7fd3500cefa0 2023-10-30T13:35:14.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.784+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd35015d900 con 0x7fd33c0664a0 2023-10-30T13:35:14.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.804+0000 7fd34cff9640 1 -- 172.21.15.31:0/1253479192 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264413 (secure 0 0 0) 0x7fd340014b50 con 0x7fd33c0664a0 2023-10-30T13:35:14.810 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.804+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd33c0664a0 msgr2=0x7fd33c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.804+0000 7fd3566d6640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd33c0664a0 0x7fd33c068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd340009b60 tx=0x7fd340008040 comp rx=0 tx=0).stop 2023-10-30T13:35:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.804+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3500cefa0 msgr2=0x7fd3500cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.804+0000 7fd3566d6640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3500cefa0 0x7fd3500cf3e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ab0 tx=0x7fd338005800 comp rx=0 tx=0).stop 2023-10-30T13:35:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 shutdown_connections 2023-10-30T13:35:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd33c0664a0 0x7fd33c068960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd3501eebc0 0x7fd3501ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3500cefa0 0x7fd3500cf3e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 --2- 172.21.15.31:0/1253479192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3380047d0 0x7fd35014d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 >> 172.21.15.31:0/1253479192 conn(0x7fd3500715c0 msgr2=0x7fd35006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 shutdown_connections 2023-10-30T13:35:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:14.808+0000 7fd3566d6640 1 -- 172.21.15.31:0/1253479192 wait complete. 2023-10-30T13:35:14.830 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-30T13:35:13.156937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218896,"kb_used_data":2696,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755824,"statfs":{"total":773094113280,"available":772869963776,"internally_reserved":0,"allocated":2760704,"data_stored":1713144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002061"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808399+0000","last_change":"2023-10-30T13:35:07.792742+0000","last_active":"2023-10-30T13:35:07.808399+0000","last_peered":"2023-10-30T13:35:07.808399+0000","last_clean":"2023-10-30T13:35:07.808399+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:35:07.808399+0000","last_undegraded":"2023-10-30T13:35:07.808399+0000","last_fullsized":"2023-10-30T13:35:07.808399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:17:06.350045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744019+0000","last_change":"2023-10-30T13:35:07.755187+0000","last_active":"2023-10-30T13:35:08.744019+0000","last_peered":"2023-10-30T13:35:08.744019+0000","last_clean":"2023-10-30T13:35:08.744019+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:35:08.744019+0000","last_undegraded":"2023-10-30T13:35:08.744019+0000","last_fullsized":"2023-10-30T13:35:08.744019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:22.764808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756473+0000","last_change":"2023-10-30T13:35:07.754357+0000","last_active":"2023-10-30T13:35:07.756473+0000","last_peered":"2023-10-30T13:35:07.756473+0000","last_clean":"2023-10-30T13:35:07.756473+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:35:07.756473+0000","last_undegraded":"2023-10-30T13:35:07.756473+0000","last_fullsized":"2023-10-30T13:35:07.756473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:59.722272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.805049+0000","last_change":"2023-10-30T13:35:07.792466+0000","last_active":"2023-10-30T13:35:07.805049+0000","last_peered":"2023-10-30T13:35:07.805049+0000","last_clean":"2023-10-30T13:35:07.805049+0000","last_became_active":"2023-10-30T13:35:07.789749+0000","last_became_peered":"2023-10-30T13:35:07.789749+0000","last_unstale":"2023-10-30T13:35:07.805049+0000","last_undegraded":"2023-10-30T13:35:07.805049+0000","last_fullsized":"2023-10-30T13:35:07.805049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:48.000408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808438+0000","last_change":"2023-10-30T13:35:07.793403+0000","last_active":"2023-10-30T13:35:07.808438+0000","last_peered":"2023-10-30T13:35:07.808438+0000","last_clean":"2023-10-30T13:35:07.808438+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:35:07.808438+0000","last_undegraded":"2023-10-30T13:35:07.808438+0000","last_fullsized":"2023-10-30T13:35:07.808438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:40.294735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804990+0000","last_change":"2023-10-30T13:35:07.792285+0000","last_active":"2023-10-30T13:35:07.804990+0000","last_peered":"2023-10-30T13:35:07.804990+0000","last_clean":"2023-10-30T13:35:07.804990+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:35:07.804990+0000","last_undegraded":"2023-10-30T13:35:07.804990+0000","last_fullsized":"2023-10-30T13:35:07.804990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:41.177494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808204+0000","last_change":"2023-10-30T13:35:07.776386+0000","last_active":"2023-10-30T13:35:07.808204+0000","last_peered":"2023-10-30T13:35:07.808204+0000","last_clean":"2023-10-30T13:35:07.808204+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:35:07.808204+0000","last_undegraded":"2023-10-30T13:35:07.808204+0000","last_fullsized":"2023-10-30T13:35:07.808204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:03:21.846537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791396+0000","last_change":"2023-10-30T13:35:07.790487+0000","last_active":"2023-10-30T13:35:07.791396+0000","last_peered":"2023-10-30T13:35:07.791396+0000","last_clean":"2023-10-30T13:35:07.791396+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:35:07.791396+0000","last_undegraded":"2023-10-30T13:35:07.791396+0000","last_fullsized":"2023-10-30T13:35:07.791396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:43.765732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.810904+0000","last_change":"2023-10-30T13:35:07.804038+0000","last_active":"2023-10-30T13:35:07.810904+0000","last_peered":"2023-10-30T13:35:07.810904+0000","last_clean":"2023-10-30T13:35:07.810904+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:35:07.810904+0000","last_undegraded":"2023-10-30T13:35:07.810904+0000","last_fullsized":"2023-10-30T13:35:07.810904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:37:23.081000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.802735+0000","last_change":"2023-10-30T13:35:07.800657+0000","last_active":"2023-10-30T13:35:07.802735+0000","last_peered":"2023-10-30T13:35:07.802735+0000","last_clean":"2023-10-30T13:35:07.802735+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:35:07.802735+0000","last_undegraded":"2023-10-30T13:35:07.802735+0000","last_fullsized":"2023-10-30T13:35:07.802735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:15.007172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.805252+0000","last_change":"2023-10-30T13:35:07.802245+0000","last_active":"2023-10-30T13:35:07.805252+0000","last_peered":"2023-10-30T13:35:07.805252+0000","last_clean":"2023-10-30T13:35:07.805252+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:35:07.805252+0000","last_undegraded":"2023-10-30T13:35:07.805252+0000","last_fullsized":"2023-10-30T13:35:07.805252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:59.843727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.787751+0000","last_change":"2023-10-30T13:35:07.783940+0000","last_active":"2023-10-30T13:35:07.787751+0000","last_peered":"2023-10-30T13:35:07.787751+0000","last_clean":"2023-10-30T13:35:07.787751+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:35:07.787751+0000","last_undegraded":"2023-10-30T13:35:07.787751+0000","last_fullsized":"2023-10-30T13:35:07.787751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:36.232521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.781412+0000","last_change":"2023-10-30T13:35:07.780349+0000","last_active":"2023-10-30T13:35:07.781412+0000","last_peered":"2023-10-30T13:35:07.781412+0000","last_clean":"2023-10-30T13:35:07.781412+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:35:07.781412+0000","last_undegraded":"2023-10-30T13:35:07.781412+0000","last_fullsized":"2023-10-30T13:35:07.781412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:18.044765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.755460+0000","last_change":"2023-10-30T13:35:07.750959+0000","last_active":"2023-10-30T13:35:07.755460+0000","last_peered":"2023-10-30T13:35:07.755460+0000","last_clean":"2023-10-30T13:35:07.755460+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:35:07.755460+0000","last_undegraded":"2023-10-30T13:35:07.755460+0000","last_fullsized":"2023-10-30T13:35:07.755460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:59:01.565769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.768466+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:27.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804445+0000","last_change":"2023-10-30T13:35:07.789918+0000","last_active":"2023-10-30T13:35:07.804445+0000","last_peered":"2023-10-30T13:35:07.804445+0000","last_clean":"2023-10-30T13:35:07.804445+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:35:07.804445+0000","last_undegraded":"2023-10-30T13:35:07.804445+0000","last_fullsized":"2023-10-30T13:35:07.804445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:29.496632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.805766+0000","last_change":"2023-10-30T13:35:07.796429+0000","last_active":"2023-10-30T13:35:07.805766+0000","last_peered":"2023-10-30T13:35:07.805766+0000","last_clean":"2023-10-30T13:35:07.805766+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:35:07.805766+0000","last_undegraded":"2023-10-30T13:35:07.805766+0000","last_fullsized":"2023-10-30T13:35:07.805766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:04.970161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.781097+0000","last_change":"2023-10-30T13:35:07.780131+0000","last_active":"2023-10-30T13:35:07.781097+0000","last_peered":"2023-10-30T13:35:07.781097+0000","last_clean":"2023-10-30T13:35:07.781097+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:35:07.781097+0000","last_undegraded":"2023-10-30T13:35:07.781097+0000","last_fullsized":"2023-10-30T13:35:07.781097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:21.748687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808301+0000","last_change":"2023-10-30T13:35:07.791464+0000","last_active":"2023-10-30T13:35:07.808301+0000","last_peered":"2023-10-30T13:35:07.808301+0000","last_clean":"2023-10-30T13:35:07.808301+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:35:07.808301+0000","last_undegraded":"2023-10-30T13:35:07.808301+0000","last_fullsized":"2023-10-30T13:35:07.808301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:29:21.519846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.789106+0000","last_change":"2023-10-30T13:35:07.783079+0000","last_active":"2023-10-30T13:35:07.789106+0000","last_peered":"2023-10-30T13:35:07.789106+0000","last_clean":"2023-10-30T13:35:07.789106+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:35:07.789106+0000","last_undegraded":"2023-10-30T13:35:07.789106+0000","last_fullsized":"2023-10-30T13:35:07.789106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:17:06.190093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.801890+0000","last_change":"2023-10-30T13:35:07.794666+0000","last_active":"2023-10-30T13:35:07.801890+0000","last_peered":"2023-10-30T13:35:07.801890+0000","last_clean":"2023-10-30T13:35:07.801890+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:35:07.801890+0000","last_undegraded":"2023-10-30T13:35:07.801890+0000","last_fullsized":"2023-10-30T13:35:07.801890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:42.312392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.806985+0000","last_change":"2023-10-30T13:35:07.796601+0000","last_active":"2023-10-30T13:35:07.806985+0000","last_peered":"2023-10-30T13:35:07.806985+0000","last_clean":"2023-10-30T13:35:07.806985+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:35:07.806985+0000","last_undegraded":"2023-10-30T13:35:07.806985+0000","last_fullsized":"2023-10-30T13:35:07.806985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:53.943593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.769377+0000","last_change":"2023-10-30T13:35:07.767965+0000","last_active":"2023-10-30T13:35:07.769377+0000","last_peered":"2023-10-30T13:35:07.769377+0000","last_clean":"2023-10-30T13:35:07.769377+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:35:07.769377+0000","last_undegraded":"2023-10-30T13:35:07.769377+0000","last_fullsized":"2023-10-30T13:35:07.769377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:10:00.028378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.788967+0000","last_change":"2023-10-30T13:35:07.772029+0000","last_active":"2023-10-30T13:35:07.788967+0000","last_peered":"2023-10-30T13:35:07.788967+0000","last_clean":"2023-10-30T13:35:07.788967+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:35:07.788967+0000","last_undegraded":"2023-10-30T13:35:07.788967+0000","last_fullsized":"2023-10-30T13:35:07.788967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:13.735624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.759188+0000","last_change":"2023-10-30T13:35:07.758274+0000","last_active":"2023-10-30T13:35:07.759188+0000","last_peered":"2023-10-30T13:35:07.759188+0000","last_clean":"2023-10-30T13:35:07.759188+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:35:07.759188+0000","last_undegraded":"2023-10-30T13:35:07.759188+0000","last_fullsized":"2023-10-30T13:35:07.759188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:04.233120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.807364+0000","last_change":"2023-10-30T13:35:07.803220+0000","last_active":"2023-10-30T13:35:07.807364+0000","last_peered":"2023-10-30T13:35:07.807364+0000","last_clean":"2023-10-30T13:35:07.807364+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:35:07.807364+0000","last_undegraded":"2023-10-30T13:35:07.807364+0000","last_fullsized":"2023-10-30T13:35:07.807364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:50.524288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743680+0000","last_change":"2023-10-30T13:35:07.753288+0000","last_active":"2023-10-30T13:35:08.743680+0000","last_peered":"2023-10-30T13:35:08.743680+0000","last_clean":"2023-10-30T13:35:08.743680+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:35:08.743680+0000","last_undegraded":"2023-10-30T13:35:08.743680+0000","last_fullsized":"2023-10-30T13:35:08.743680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:43.815379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.769395+0000","last_change":"2023-10-30T13:35:07.767931+0000","last_active":"2023-10-30T13:35:07.769395+0000","last_peered":"2023-10-30T13:35:07.769395+0000","last_clean":"2023-10-30T13:35:07.769395+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:35:07.769395+0000","last_undegraded":"2023-10-30T13:35:07.769395+0000","last_fullsized":"2023-10-30T13:35:07.769395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:09.828603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756607+0000","last_change":"2023-10-30T13:35:07.753695+0000","last_active":"2023-10-30T13:35:07.756607+0000","last_peered":"2023-10-30T13:35:07.756607+0000","last_clean":"2023-10-30T13:35:07.756607+0000","last_became_active":"2023-10-30T13:35:07.753358+0000","last_became_peered":"2023-10-30T13:35:07.753358+0000","last_unstale":"2023-10-30T13:35:07.756607+0000","last_undegraded":"2023-10-30T13:35:07.756607+0000","last_fullsized":"2023-10-30T13:35:07.756607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:03:35.697267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804596+0000","last_change":"2023-10-30T13:35:07.790236+0000","last_active":"2023-10-30T13:35:07.804596+0000","last_peered":"2023-10-30T13:35:07.804596+0000","last_clean":"2023-10-30T13:35:07.804596+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:35:07.804596+0000","last_undegraded":"2023-10-30T13:35:07.804596+0000","last_fullsized":"2023-10-30T13:35:07.804596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:48.035840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791500+0000","last_change":"2023-10-30T13:35:07.790720+0000","last_active":"2023-10-30T13:35:07.791500+0000","last_peered":"2023-10-30T13:35:07.791500+0000","last_clean":"2023-10-30T13:35:07.791500+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:35:07.791500+0000","last_undegraded":"2023-10-30T13:35:07.791500+0000","last_fullsized":"2023-10-30T13:35:07.791500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:49:00.265087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.789161+0000","last_change":"2023-10-30T13:35:07.780576+0000","last_active":"2023-10-30T13:35:07.789161+0000","last_peered":"2023-10-30T13:35:07.789161+0000","last_clean":"2023-10-30T13:35:07.789161+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:35:07.789161+0000","last_undegraded":"2023-10-30T13:35:07.789161+0000","last_fullsized":"2023-10-30T13:35:07.789161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:33.677737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.761724+0000","last_change":"2023-10-30T13:35:07.755919+0000","last_active":"2023-10-30T13:35:07.761724+0000","last_peered":"2023-10-30T13:35:07.761724+0000","last_clean":"2023-10-30T13:35:07.761724+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:35:07.761724+0000","last_undegraded":"2023-10-30T13:35:07.761724+0000","last_fullsized":"2023-10-30T13:35:07.761724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:00.465200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.748399+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:35:07.748399+0000","last_peered":"2023-10-30T13:35:07.748399+0000","last_clean":"2023-10-30T13:35:07.748399+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:35:07.748399+0000","last_undegraded":"2023-10-30T13:35:07.748399+0000","last_fullsized":"2023-10-30T13:35:07.748399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.763593+0000","last_change":"2023-10-30T13:35:07.762428+0000","last_active":"2023-10-30T13:35:07.763593+0000","last_peered":"2023-10-30T13:35:07.763593+0000","last_clean":"2023-10-30T13:35:07.763593+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:35:07.763593+0000","last_undegraded":"2023-10-30T13:35:07.763593+0000","last_fullsized":"2023-10-30T13:35:07.763593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:17.453197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.763104+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:35:07.763104+0000","last_peered":"2023-10-30T13:35:07.763104+0000","last_clean":"2023-10-30T13:35:07.763104+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:35:07.763104+0000","last_undegraded":"2023-10-30T13:35:07.763104+0000","last_fullsized":"2023-10-30T13:35:07.763104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.746252+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:35:07.746252+0000","last_peered":"2023-10-30T13:35:07.746252+0000","last_clean":"2023-10-30T13:35:07.746252+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:35:07.746252+0000","last_undegraded":"2023-10-30T13:35:07.746252+0000","last_fullsized":"2023-10-30T13:35:07.746252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.754244+0000","last_change":"2023-10-30T13:35:07.751131+0000","last_active":"2023-10-30T13:35:07.754244+0000","last_peered":"2023-10-30T13:35:07.754244+0000","last_clean":"2023-10-30T13:35:07.754244+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:35:07.754244+0000","last_undegraded":"2023-10-30T13:35:07.754244+0000","last_fullsized":"2023-10-30T13:35:07.754244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:27:03.270510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756967+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:35:07.756967+0000","last_peered":"2023-10-30T13:35:07.756967+0000","last_clean":"2023-10-30T13:35:07.756967+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:35:07.756967+0000","last_undegraded":"2023-10-30T13:35:07.756967+0000","last_fullsized":"2023-10-30T13:35:07.756967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743438+0000","last_change":"2023-10-30T13:35:07.750392+0000","last_active":"2023-10-30T13:35:08.743438+0000","last_peered":"2023-10-30T13:35:08.743438+0000","last_clean":"2023-10-30T13:35:08.743438+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:35:08.743438+0000","last_undegraded":"2023-10-30T13:35:08.743438+0000","last_fullsized":"2023-10-30T13:35:08.743438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:33.554754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.753242+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:35:07.753242+0000","last_peered":"2023-10-30T13:35:07.753242+0000","last_clean":"2023-10-30T13:35:07.753242+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:35:07.753242+0000","last_undegraded":"2023-10-30T13:35:07.753242+0000","last_fullsized":"2023-10-30T13:35:07.753242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.807074+0000","last_change":"2023-10-30T13:35:07.797513+0000","last_active":"2023-10-30T13:35:07.807074+0000","last_peered":"2023-10-30T13:35:07.807074+0000","last_clean":"2023-10-30T13:35:07.807074+0000","last_became_active":"2023-10-30T13:35:07.796748+0000","last_became_peered":"2023-10-30T13:35:07.796748+0000","last_unstale":"2023-10-30T13:35:07.807074+0000","last_undegraded":"2023-10-30T13:35:07.807074+0000","last_fullsized":"2023-10-30T13:35:07.807074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:39.151161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.756154+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:35:07.756154+0000","last_peered":"2023-10-30T13:35:07.756154+0000","last_clean":"2023-10-30T13:35:07.756154+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:35:07.756154+0000","last_undegraded":"2023-10-30T13:35:07.756154+0000","last_fullsized":"2023-10-30T13:35:07.756154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791718+0000","last_change":"2023-10-30T13:35:07.789983+0000","last_active":"2023-10-30T13:35:07.791718+0000","last_peered":"2023-10-30T13:35:07.791718+0000","last_clean":"2023-10-30T13:35:07.791718+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:35:07.791718+0000","last_undegraded":"2023-10-30T13:35:07.791718+0000","last_fullsized":"2023-10-30T13:35:07.791718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:40:26.139549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743813+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:35:08.743813+0000","last_peered":"2023-10-30T13:35:08.743813+0000","last_clean":"2023-10-30T13:35:08.743813+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:35:08.743813+0000","last_undegraded":"2023-10-30T13:35:08.743813+0000","last_fullsized":"2023-10-30T13:35:08.743813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.803657+0000","last_change":"2023-10-30T13:35:07.794930+0000","last_active":"2023-10-30T13:35:07.803657+0000","last_peered":"2023-10-30T13:35:07.803657+0000","last_clean":"2023-10-30T13:35:07.803657+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:35:07.803657+0000","last_undegraded":"2023-10-30T13:35:07.803657+0000","last_fullsized":"2023-10-30T13:35:07.803657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:24.359886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.764306+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:35:07.764306+0000","last_peered":"2023-10-30T13:35:07.764306+0000","last_clean":"2023-10-30T13:35:07.764306+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:35:07.764306+0000","last_undegraded":"2023-10-30T13:35:07.764306+0000","last_fullsized":"2023-10-30T13:35:07.764306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808355+0000","last_change":"2023-10-30T13:35:07.791923+0000","last_active":"2023-10-30T13:35:07.808355+0000","last_peered":"2023-10-30T13:35:07.808355+0000","last_clean":"2023-10-30T13:35:07.808355+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:35:07.808355+0000","last_undegraded":"2023-10-30T13:35:07.808355+0000","last_fullsized":"2023-10-30T13:35:07.808355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:23.544401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.746223+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:35:07.746223+0000","last_peered":"2023-10-30T13:35:07.746223+0000","last_clean":"2023-10-30T13:35:07.746223+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:35:07.746223+0000","last_undegraded":"2023-10-30T13:35:07.746223+0000","last_fullsized":"2023-10-30T13:35:07.746223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.801902+0000","last_change":"2023-10-30T13:35:07.794790+0000","last_active":"2023-10-30T13:35:07.801902+0000","last_peered":"2023-10-30T13:35:07.801902+0000","last_clean":"2023-10-30T13:35:07.801902+0000","last_became_active":"2023-10-30T13:35:07.794477+0000","last_became_peered":"2023-10-30T13:35:07.794477+0000","last_unstale":"2023-10-30T13:35:07.801902+0000","last_undegraded":"2023-10-30T13:35:07.801902+0000","last_fullsized":"2023-10-30T13:35:07.801902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:47:29.816491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.771810+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:35:07.771810+0000","last_peered":"2023-10-30T13:35:07.771810+0000","last_clean":"2023-10-30T13:35:07.771810+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:35:07.771810+0000","last_undegraded":"2023-10-30T13:35:07.771810+0000","last_fullsized":"2023-10-30T13:35:07.771810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808744+0000","last_change":"2023-10-30T13:35:07.797247+0000","last_active":"2023-10-30T13:35:07.808744+0000","last_peered":"2023-10-30T13:35:07.808744+0000","last_clean":"2023-10-30T13:35:07.808744+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:35:07.808744+0000","last_undegraded":"2023-10-30T13:35:07.808744+0000","last_fullsized":"2023-10-30T13:35:07.808744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:46:21.871813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.754609+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:35:07.754609+0000","last_peered":"2023-10-30T13:35:07.754609+0000","last_clean":"2023-10-30T13:35:07.754609+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:35:07.754609+0000","last_undegraded":"2023-10-30T13:35:07.754609+0000","last_fullsized":"2023-10-30T13:35:07.754609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.781957+0000","last_change":"2023-10-30T13:35:07.773683+0000","last_active":"2023-10-30T13:35:07.781957+0000","last_peered":"2023-10-30T13:35:07.781957+0000","last_clean":"2023-10-30T13:35:07.781957+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:35:07.781957+0000","last_undegraded":"2023-10-30T13:35:07.781957+0000","last_fullsized":"2023-10-30T13:35:07.781957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:20.750111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.744464+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:35:07.744464+0000","last_peered":"2023-10-30T13:35:07.744464+0000","last_clean":"2023-10-30T13:35:07.744464+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:35:07.744464+0000","last_undegraded":"2023-10-30T13:35:07.744464+0000","last_fullsized":"2023-10-30T13:35:07.744464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.810692+0000","last_change":"2023-10-30T13:35:07.791266+0000","last_active":"2023-10-30T13:35:07.810692+0000","last_peered":"2023-10-30T13:35:07.810692+0000","last_clean":"2023-10-30T13:35:07.810692+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:35:07.810692+0000","last_undegraded":"2023-10-30T13:35:07.810692+0000","last_fullsized":"2023-10-30T13:35:07.810692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:23:01.624384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768918+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:35:07.768918+0000","last_peered":"2023-10-30T13:35:07.768918+0000","last_clean":"2023-10-30T13:35:07.768918+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:35:07.768918+0000","last_undegraded":"2023-10-30T13:35:07.768918+0000","last_fullsized":"2023-10-30T13:35:07.768918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808484+0000","last_change":"2023-10-30T13:35:07.794747+0000","last_active":"2023-10-30T13:35:07.808484+0000","last_peered":"2023-10-30T13:35:07.808484+0000","last_clean":"2023-10-30T13:35:07.808484+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:35:07.808484+0000","last_undegraded":"2023-10-30T13:35:07.808484+0000","last_fullsized":"2023-10-30T13:35:07.808484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:31.479675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.745728+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:35:07.745728+0000","last_peered":"2023-10-30T13:35:07.745728+0000","last_clean":"2023-10-30T13:35:07.745728+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:35:07.745728+0000","last_undegraded":"2023-10-30T13:35:07.745728+0000","last_fullsized":"2023-10-30T13:35:07.745728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808438+0000","last_change":"2023-10-30T13:35:07.793255+0000","last_active":"2023-10-30T13:35:07.808438+0000","last_peered":"2023-10-30T13:35:07.808438+0000","last_clean":"2023-10-30T13:35:07.808438+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:35:07.808438+0000","last_undegraded":"2023-10-30T13:35:07.808438+0000","last_fullsized":"2023-10-30T13:35:07.808438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:36.854385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743869+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:35:08.743869+0000","last_peered":"2023-10-30T13:35:08.743869+0000","last_clean":"2023-10-30T13:35:08.743869+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:35:08.743869+0000","last_undegraded":"2023-10-30T13:35:08.743869+0000","last_fullsized":"2023-10-30T13:35:08.743869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804649+0000","last_change":"2023-10-30T13:35:07.791920+0000","last_active":"2023-10-30T13:35:07.804649+0000","last_peered":"2023-10-30T13:35:07.804649+0000","last_clean":"2023-10-30T13:35:07.804649+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:35:07.804649+0000","last_undegraded":"2023-10-30T13:35:07.804649+0000","last_fullsized":"2023-10-30T13:35:07.804649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:58.172166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.746443+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:35:07.746443+0000","last_peered":"2023-10-30T13:35:07.746443+0000","last_clean":"2023-10-30T13:35:07.746443+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:35:07.746443+0000","last_undegraded":"2023-10-30T13:35:07.746443+0000","last_fullsized":"2023-10-30T13:35:07.746443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.810851+0000","last_change":"2023-10-30T13:35:07.803211+0000","last_active":"2023-10-30T13:35:07.810851+0000","last_peered":"2023-10-30T13:35:07.810851+0000","last_clean":"2023-10-30T13:35:07.810851+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:35:07.810851+0000","last_undegraded":"2023-10-30T13:35:07.810851+0000","last_fullsized":"2023-10-30T13:35:07.810851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:34:13.535807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.289348+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:35:07.289348+0000","last_peered":"2023-10-30T13:35:07.289348+0000","last_clean":"2023-10-30T13:35:07.289348+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:35:07.289348+0000","last_undegraded":"2023-10-30T13:35:07.289348+0000","last_fullsized":"2023-10-30T13:35:07.289348+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.749763+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:35:07.749763+0000","last_peered":"2023-10-30T13:35:07.749763+0000","last_clean":"2023-10-30T13:35:07.749763+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:35:07.749763+0000","last_undegraded":"2023-10-30T13:35:07.749763+0000","last_fullsized":"2023-10-30T13:35:07.749763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.796651+0000","last_change":"2023-10-30T13:35:07.792082+0000","last_active":"2023-10-30T13:35:07.796651+0000","last_peered":"2023-10-30T13:35:07.796651+0000","last_clean":"2023-10-30T13:35:07.796651+0000","last_became_active":"2023-10-30T13:35:07.791831+0000","last_became_peered":"2023-10-30T13:35:07.791831+0000","last_unstale":"2023-10-30T13:35:07.796651+0000","last_undegraded":"2023-10-30T13:35:07.796651+0000","last_fullsized":"2023-10-30T13:35:07.796651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:58:35.766504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768549+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:35:07.768549+0000","last_peered":"2023-10-30T13:35:07.768549+0000","last_clean":"2023-10-30T13:35:07.768549+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:35:07.768549+0000","last_undegraded":"2023-10-30T13:35:07.768549+0000","last_fullsized":"2023-10-30T13:35:07.768549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.761377+0000","last_change":"2023-10-30T13:35:07.760132+0000","last_active":"2023-10-30T13:35:07.761377+0000","last_peered":"2023-10-30T13:35:07.761377+0000","last_clean":"2023-10-30T13:35:07.761377+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:35:07.761377+0000","last_undegraded":"2023-10-30T13:35:07.761377+0000","last_fullsized":"2023-10-30T13:35:07.761377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:58.458839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.745502+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:35:07.745502+0000","last_peered":"2023-10-30T13:35:07.745502+0000","last_clean":"2023-10-30T13:35:07.745502+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:35:07.745502+0000","last_undegraded":"2023-10-30T13:35:07.745502+0000","last_fullsized":"2023-10-30T13:35:07.745502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.750958+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:53.697186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.772257+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:35:07.772257+0000","last_peered":"2023-10-30T13:35:07.772257+0000","last_clean":"2023-10-30T13:35:07.772257+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:35:07.772257+0000","last_undegraded":"2023-10-30T13:35:07.772257+0000","last_fullsized":"2023-10-30T13:35:07.772257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.782470+0000","last_change":"2023-10-30T13:35:07.768640+0000","last_active":"2023-10-30T13:35:07.782470+0000","last_peered":"2023-10-30T13:35:07.782470+0000","last_clean":"2023-10-30T13:35:07.782470+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:35:07.782470+0000","last_undegraded":"2023-10-30T13:35:07.782470+0000","last_fullsized":"2023-10-30T13:35:07.782470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:57.460587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743612+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:35:08.743612+0000","last_peered":"2023-10-30T13:35:08.743612+0000","last_clean":"2023-10-30T13:35:08.743612+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:35:08.743612+0000","last_undegraded":"2023-10-30T13:35:08.743612+0000","last_fullsized":"2023-10-30T13:35:08.743612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808749+0000","last_change":"2023-10-30T13:35:07.805535+0000","last_active":"2023-10-30T13:35:07.808749+0000","last_peered":"2023-10-30T13:35:07.808749+0000","last_clean":"2023-10-30T13:35:07.808749+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:35:07.808749+0000","last_undegraded":"2023-10-30T13:35:07.808749+0000","last_fullsized":"2023-10-30T13:35:07.808749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:48.167900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.766088+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:35:07.766088+0000","last_peered":"2023-10-30T13:35:07.766088+0000","last_clean":"2023-10-30T13:35:07.766088+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:35:07.766088+0000","last_undegraded":"2023-10-30T13:35:07.766088+0000","last_fullsized":"2023-10-30T13:35:07.766088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743474+0000","last_change":"2023-10-30T13:35:07.754171+0000","last_active":"2023-10-30T13:35:08.743474+0000","last_peered":"2023-10-30T13:35:08.743474+0000","last_clean":"2023-10-30T13:35:08.743474+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:35:08.743474+0000","last_undegraded":"2023-10-30T13:35:08.743474+0000","last_fullsized":"2023-10-30T13:35:08.743474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:50.849695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744041+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:35:08.744041+0000","last_peered":"2023-10-30T13:35:08.744041+0000","last_clean":"2023-10-30T13:35:08.744041+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:35:08.744041+0000","last_undegraded":"2023-10-30T13:35:08.744041+0000","last_fullsized":"2023-10-30T13:35:08.744041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.801917+0000","last_change":"2023-10-30T13:35:07.794804+0000","last_active":"2023-10-30T13:35:07.801917+0000","last_peered":"2023-10-30T13:35:07.801917+0000","last_clean":"2023-10-30T13:35:07.801917+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:35:07.801917+0000","last_undegraded":"2023-10-30T13:35:07.801917+0000","last_fullsized":"2023-10-30T13:35:07.801917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:45:13.550352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.753319+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:35:07.753319+0000","last_peered":"2023-10-30T13:35:07.753319+0000","last_clean":"2023-10-30T13:35:07.753319+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:35:07.753319+0000","last_undegraded":"2023-10-30T13:35:07.753319+0000","last_fullsized":"2023-10-30T13:35:07.753319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.755643+0000","last_change":"2023-10-30T13:35:07.752077+0000","last_active":"2023-10-30T13:35:07.755643+0000","last_peered":"2023-10-30T13:35:07.755643+0000","last_clean":"2023-10-30T13:35:07.755643+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:35:07.755643+0000","last_undegraded":"2023-10-30T13:35:07.755643+0000","last_fullsized":"2023-10-30T13:35:07.755643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:39.134825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.767414+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:35:07.767414+0000","last_peered":"2023-10-30T13:35:07.767414+0000","last_clean":"2023-10-30T13:35:07.767414+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:35:07.767414+0000","last_undegraded":"2023-10-30T13:35:07.767414+0000","last_fullsized":"2023-10-30T13:35:07.767414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.766791+0000","last_change":"2023-10-30T13:35:07.759389+0000","last_active":"2023-10-30T13:35:07.766791+0000","last_peered":"2023-10-30T13:35:07.766791+0000","last_clean":"2023-10-30T13:35:07.766791+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:35:07.766791+0000","last_undegraded":"2023-10-30T13:35:07.766791+0000","last_fullsized":"2023-10-30T13:35:07.766791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:48:06.413218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743540+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:35:08.743540+0000","last_peered":"2023-10-30T13:35:08.743540+0000","last_clean":"2023-10-30T13:35:08.743540+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:35:08.743540+0000","last_undegraded":"2023-10-30T13:35:08.743540+0000","last_fullsized":"2023-10-30T13:35:08.743540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.757489+0000","last_change":"2023-10-30T13:35:07.753673+0000","last_active":"2023-10-30T13:35:07.757489+0000","last_peered":"2023-10-30T13:35:07.757489+0000","last_clean":"2023-10-30T13:35:07.757489+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:35:07.757489+0000","last_undegraded":"2023-10-30T13:35:07.757489+0000","last_fullsized":"2023-10-30T13:35:07.757489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:08:30.823529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743863+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:35:08.743863+0000","last_peered":"2023-10-30T13:35:08.743863+0000","last_clean":"2023-10-30T13:35:08.743863+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:35:08.743863+0000","last_undegraded":"2023-10-30T13:35:08.743863+0000","last_fullsized":"2023-10-30T13:35:08.743863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.757579+0000","last_change":"2023-10-30T13:35:07.755302+0000","last_active":"2023-10-30T13:35:07.757579+0000","last_peered":"2023-10-30T13:35:07.757579+0000","last_clean":"2023-10-30T13:35:07.757579+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:35:07.757579+0000","last_undegraded":"2023-10-30T13:35:07.757579+0000","last_fullsized":"2023-10-30T13:35:07.757579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:06:03.470605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.767574+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:35:07.767574+0000","last_peered":"2023-10-30T13:35:07.767574+0000","last_clean":"2023-10-30T13:35:07.767574+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:35:07.767574+0000","last_undegraded":"2023-10-30T13:35:07.767574+0000","last_fullsized":"2023-10-30T13:35:07.767574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.786227+0000","last_change":"2023-10-30T13:35:07.780297+0000","last_active":"2023-10-30T13:35:07.786227+0000","last_peered":"2023-10-30T13:35:07.786227+0000","last_clean":"2023-10-30T13:35:07.786227+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:35:07.786227+0000","last_undegraded":"2023-10-30T13:35:07.786227+0000","last_fullsized":"2023-10-30T13:35:07.786227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:20:12.682537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.808790+0000","last_change":"2023-10-30T13:35:07.806203+0000","last_active":"2023-10-30T13:35:07.808790+0000","last_peered":"2023-10-30T13:35:07.808790+0000","last_clean":"2023-10-30T13:35:07.808790+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:35:07.808790+0000","last_undegraded":"2023-10-30T13:35:07.808790+0000","last_fullsized":"2023-10-30T13:35:07.808790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:53.855040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.759765+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:35:07.759765+0000","last_peered":"2023-10-30T13:35:07.759765+0000","last_clean":"2023-10-30T13:35:07.759765+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:35:07.759765+0000","last_undegraded":"2023-10-30T13:35:07.759765+0000","last_fullsized":"2023-10-30T13:35:07.759765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.804522+0000","last_change":"2023-10-30T13:35:07.791319+0000","last_active":"2023-10-30T13:35:07.804522+0000","last_peered":"2023-10-30T13:35:07.804522+0000","last_clean":"2023-10-30T13:35:07.804522+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:35:07.804522+0000","last_undegraded":"2023-10-30T13:35:07.804522+0000","last_fullsized":"2023-10-30T13:35:07.804522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:57:23.365203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.764665+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:35:07.764665+0000","last_peered":"2023-10-30T13:35:07.764665+0000","last_clean":"2023-10-30T13:35:07.764665+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:35:07.764665+0000","last_undegraded":"2023-10-30T13:35:07.764665+0000","last_fullsized":"2023-10-30T13:35:07.764665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.791738+0000","last_change":"2023-10-30T13:35:07.790084+0000","last_active":"2023-10-30T13:35:07.791738+0000","last_peered":"2023-10-30T13:35:07.791738+0000","last_clean":"2023-10-30T13:35:07.791738+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:35:07.791738+0000","last_undegraded":"2023-10-30T13:35:07.791738+0000","last_fullsized":"2023-10-30T13:35:07.791738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:32:22.844026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.748210+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:35:07.748210+0000","last_peered":"2023-10-30T13:35:07.748210+0000","last_clean":"2023-10-30T13:35:07.748210+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:35:07.748210+0000","last_undegraded":"2023-10-30T13:35:07.748210+0000","last_fullsized":"2023-10-30T13:35:07.748210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.784143+0000","last_change":"2023-10-30T13:35:07.782341+0000","last_active":"2023-10-30T13:35:07.784143+0000","last_peered":"2023-10-30T13:35:07.784143+0000","last_clean":"2023-10-30T13:35:07.784143+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:35:07.784143+0000","last_undegraded":"2023-10-30T13:35:07.784143+0000","last_fullsized":"2023-10-30T13:35:07.784143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:08.779946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.744301+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:35:07.744301+0000","last_peered":"2023-10-30T13:35:07.744301+0000","last_clean":"2023-10-30T13:35:07.744301+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:35:07.744301+0000","last_undegraded":"2023-10-30T13:35:07.744301+0000","last_fullsized":"2023-10-30T13:35:07.744301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":656919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":656919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:35:14.835 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-30T13:35:14.836 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T13:35:14.836 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T13:35:14.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T13:35:14.837 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T13:35:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd7326ea640 1 Processor -- start 2023-10-30T13:35:15.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd7326ea640 1 -- start start 2023-10-30T13:35:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd7326ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd72c06c420 0x7fd72c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd7326ea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72c0c5f80 con 0x7fd72c153bb0 2023-10-30T13:35:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd7326ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72c0c5690 con 0x7fd72c0c7170 2023-10-30T13:35:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd7326ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd72c0c7580 con 0x7fd72c06c420 2023-10-30T13:35:15.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd72bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fd72c153bb0 0x7fd72c0c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51776/0 (socket says 172.21.15.31:51776) 2023-10-30T13:35:15.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd72bfff640 1 -- 172.21.15.31:0/3232068296 learned_addr learned my addr 172.21.15.31:0/3232068296 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:15.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.012+0000 7fd72b7fe640 1 --2- 172.21.15.31:0/3232068296 >> v2:172.21.15.33:3300/0 conn(0x7fd72c06c420 0x7fd72c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292875945 0 0) 0x7fd72c0c5f80 con 0x7fd72c153bb0 2023-10-30T13:35:15.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd714003530 con 0x7fd72c153bb0 2023-10-30T13:35:15.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 227161467 0 0) 0x7fd714003530 con 0x7fd72c153bb0 2023-10-30T13:35:15.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 >> v2:172.21.15.33:3300/0 conn(0x7fd72c06c420 msgr2=0x7fd72c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:15.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 --2- 172.21.15.31:0/3232068296 >> v2:172.21.15.33:3300/0 conn(0x7fd72c06c420 0x7fd72c0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 >> v1:172.21.15.33:6789/0 conn(0x7fd72c0c7170 legacy=0x7fd72c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c15a580 con 0x7fd72c153bb0 2023-10-30T13:35:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fd71c001680 con 0x7fd72c153bb0 2023-10-30T13:35:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 >> v1:172.21.15.31:6789/0 conn(0x7fd72c153bb0 legacy=0x7fd72c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd714003980 0x7fd714003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714004380 0x7fd714004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd714005050 0x7fd71400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c0c7580 con 0x7fd714003980 2023-10-30T13:35:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71400bf90 con 0x7fd714004380 2023-10-30T13:35:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71400c150 con 0x7fd714005050 2023-10-30T13:35:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72bfff640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd714003980 0x7fd714003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72b7fe640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714004380 0x7fd714004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd730c87640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd714005050 0x7fd71400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72bfff640 1 -- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd714005050 msgr2=0x7fd71400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72bfff640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd714005050 0x7fd71400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72bfff640 1 -- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714004380 msgr2=0x7fd714004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72bfff640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714004380 0x7fd714004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72bfff640 1 -- 172.21.15.31:0/3232068296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd714003530 con 0x7fd714003980 2023-10-30T13:35:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd72b7fe640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714004380 0x7fd714004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:15.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd730c87640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd714005050 0x7fd71400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:15.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.016+0000 7fd730c87640 1 -- 172.21.15.31:0/3232068296 reap_dead start 2023-10-30T13:35:15.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd72bfff640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd714003980 0x7fd714003d60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0050a0 tx=0x7fd71c01f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1532e9932e77c7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c005ae0 con 0x7fd714003980 2023-10-30T13:35:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71c005c40 con 0x7fd714003980 2023-10-30T13:35:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd72affd640 1 -- 172.21.15.31:0/3232068296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c008110 con 0x7fd714003980 2023-10-30T13:35:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- 172.21.15.31:0/3232068296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd714003980 msgr2=0x7fd714003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd714003980 0x7fd714003d60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0050a0 tx=0x7fd71c01f9d0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- 172.21.15.31:0/3232068296 shutdown_connections 2023-10-30T13:35:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 --2- 172.21.15.31:0/3232068296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd714003980 0x7fd714003d60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- 172.21.15.31:0/3232068296 >> 172.21.15.31:0/3232068296 conn(0x7fd72c0cac70 msgr2=0x7fd72c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:15.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- 172.21.15.31:0/3232068296 shutdown_connections 2023-10-30T13:35:15.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- 172.21.15.31:0/3232068296 wait complete. 2023-10-30T13:35:15.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 Processor -- start 2023-10-30T13:35:15.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- start start 2023-10-30T13:35:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714003980 0x7fd72c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd72c1eeb00 0x7fd72c156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd72c158840 0x7fd72c158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c1502a0 con 0x7fd72c1eeb00 2023-10-30T13:35:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c0bcd90 con 0x7fd714003980 2023-10-30T13:35:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.020+0000 7fd7326ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c1573d0 con 0x7fd72c158840 2023-10-30T13:35:15.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd72c1eeb00 0x7fd72c156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd72c1eeb00 0x7fd72c156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35844/0 (socket says 172.21.15.31:35844) 2023-10-30T13:35:15.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 -- 172.21.15.31:0/1846202109 learned_addr learned my addr 172.21.15.31:0/1846202109 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:15.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72bfff640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714003980 0x7fd72c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd730c87640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd72c158840 0x7fd72c158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 -- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd72c158840 msgr2=0x7fd72c158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd72c158840 0x7fd72c158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 -- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714003980 msgr2=0x7fd72c153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714003980 0x7fd72c153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 -- 172.21.15.31:0/1846202109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c154580 con 0x7fd72c1eeb00 2023-10-30T13:35:15.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd72b7fe640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd72c1eeb00 0x7fd72c156e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd720011900 tx=0x7fd720011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd728ff9640 1 -- 172.21.15.31:0/1846202109 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720004220 con 0x7fd72c1eeb00 2023-10-30T13:35:15.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd728ff9640 1 -- 172.21.15.31:0/1846202109 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd720004380 con 0x7fd72c1eeb00 2023-10-30T13:35:15.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd728ff9640 1 -- 172.21.15.31:0/1846202109 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72000fb70 con 0x7fd72c1eeb00 2023-10-30T13:35:15.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72c154840 con 0x7fd72c1eeb00 2023-10-30T13:35:15.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72c154e00 con 0x7fd72c1eeb00 2023-10-30T13:35:15.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72c0bcf40 con 0x7fd72c1eeb00 2023-10-30T13:35:15.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.024+0000 7fd728ff9640 1 -- 172.21.15.31:0/1846202109 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd72000fcd0 con 0x7fd72c1eeb00 2023-10-30T13:35:15.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.028+0000 7fd728ff9640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd7180664a0 0x7fd718068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.028+0000 7fd728ff9640 1 -- 172.21.15.31:0/1846202109 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd720084520 con 0x7fd72c1eeb00 2023-10-30T13:35:15.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.028+0000 7fd72bfff640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd7180664a0 0x7fd718068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.028+0000 7fd72bfff640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd7180664a0 0x7fd718068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00a000 tx=0x7fd71c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.032+0000 7fd728ff9640 1 -- 172.21.15.31:0/1846202109 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd72c0bcf40 con 0x7fd72c1eeb00 2023-10-30T13:35:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.224+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd72c0bcf40 con 0x7fd72c1eeb00 2023-10-30T13:35:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.228+0000 7fd728ff9640 1 -- 172.21.15.31:0/1846202109 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164429 (secure 0 0 0) 0x7fd72c0bcf40 con 0x7fd72c1eeb00 2023-10-30T13:35:15.235 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:15.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd7180664a0 msgr2=0x7fd718068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd7180664a0 0x7fd718068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00a000 tx=0x7fd71c008040 comp rx=0 tx=0).stop 2023-10-30T13:35:15.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd72c1eeb00 msgr2=0x7fd72c156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd72c1eeb00 0x7fd72c156e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd720011900 tx=0x7fd720011d00 comp rx=0 tx=0).stop 2023-10-30T13:35:15.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 shutdown_connections 2023-10-30T13:35:15.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd7180664a0 0x7fd718068960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd72c158840 0x7fd72c158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd72c1eeb00 0x7fd72c156e90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 --2- 172.21.15.31:0/1846202109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714003980 0x7fd72c153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 >> 172.21.15.31:0/1846202109 conn(0x7fd72c0cac70 msgr2=0x7fd72c0c8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:15.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 shutdown_connections 2023-10-30T13:35:15.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.232+0000 7fd7326ea640 1 -- 172.21.15.31:0/1846202109 wait complete. 2023-10-30T13:35:15.255 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2139658642},{"type":"v1","addr":"172.21.15.33:6833","nonce":2139658642}]},"active_addr":"172.21.15.33:6833/2139658642","active_change":"2023-10-30T13:34:47.138577+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":300955228}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1588005012}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3555313353}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3437089534}]}]} 2023-10-30T13:35:15.259 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T13:35:15.259 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T13:35:15.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 Processor -- start 2023-10-30T13:35:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 -- start start 2023-10-30T13:35:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb63406b760 0x7fb63415e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406bb40 0x7fb634157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb63406c420 0x7fb634157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb634071e00 con 0x7fb63406b760 2023-10-30T13:35:15.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb634070020 con 0x7fb63406c420 2023-10-30T13:35:15.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63bf1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63415eb90 con 0x7fb63406bb40 2023-10-30T13:35:15.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb639cb9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb63406b760 0x7fb63415e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb639cb9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb63406b760 0x7fb63415e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35858/0 (socket says 172.21.15.31:35858) 2023-10-30T13:35:15.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb639cb9640 1 -- 172.21.15.31:0/2201708249 learned_addr learned my addr 172.21.15.31:0/2201708249 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:15.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb6394b8640 1 --2- 172.21.15.31:0/2201708249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406bb40 0x7fb634157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb63a4ba640 1 --2- 172.21.15.31:0/2201708249 >> v2:172.21.15.33:3300/0 conn(0x7fb63406c420 0x7fb634157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb639cb9640 1 -- 172.21.15.31:0/2201708249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406bb40 msgr2=0x7fb634157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb639cb9640 1 --2- 172.21.15.31:0/2201708249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406bb40 0x7fb634157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb639cb9640 1 -- 172.21.15.31:0/2201708249 >> v2:172.21.15.33:3300/0 conn(0x7fb63406c420 msgr2=0x7fb634157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.400+0000 7fb639cb9640 1 --2- 172.21.15.31:0/2201708249 >> v2:172.21.15.33:3300/0 conn(0x7fb63406c420 0x7fb634157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb639cb9640 1 -- 172.21.15.31:0/2201708249 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb634070f10 con 0x7fb63406b760 2023-10-30T13:35:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb6394b8640 1 --2- 172.21.15.31:0/2201708249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406bb40 0x7fb634157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:15.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb639cb9640 1 --2- 172.21.15.31:0/2201708249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb63415e650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb63415b7a0 tx=0x7fb63001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91de1e7353cc490e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb638cb7640 1 -- 172.21.15.31:0/2201708249 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6300aa5f0 con 0x7fb63406b760 2023-10-30T13:35:15.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb638cb7640 1 -- 172.21.15.31:0/2201708249 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6300aa750 con 0x7fb63406b760 2023-10-30T13:35:15.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb638cb7640 1 -- 172.21.15.31:0/2201708249 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6300aaae0 con 0x7fb63406b760 2023-10-30T13:35:15.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 -- 172.21.15.31:0/2201708249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 msgr2=0x7fb63415e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/2201708249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb63415e650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb63415b7a0 tx=0x7fb63001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 -- 172.21.15.31:0/2201708249 shutdown_connections 2023-10-30T13:35:15.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/2201708249 >> v2:172.21.15.33:3300/0 conn(0x7fb63406c420 0x7fb634157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/2201708249 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406bb40 0x7fb634157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/2201708249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb63415e650 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 -- 172.21.15.31:0/2201708249 >> 172.21.15.31:0/2201708249 conn(0x7fb6340cac70 msgr2=0x7fb6340cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:15.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 -- 172.21.15.31:0/2201708249 shutdown_connections 2023-10-30T13:35:15.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.404+0000 7fb63bf1d640 1 -- 172.21.15.31:0/2201708249 wait complete. 2023-10-30T13:35:15.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 Processor -- start 2023-10-30T13:35:15.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 -- start start 2023-10-30T13:35:15.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb634159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb63406bb40 0x7fb63415a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406c420 0x7fb634158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340c4c80 con 0x7fb63406b760 2023-10-30T13:35:15.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341f0400 con 0x7fb63406bb40 2023-10-30T13:35:15.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340c4730 con 0x7fb63406c420 2023-10-30T13:35:15.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb634159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb634159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35864/0 (socket says 172.21.15.31:35864) 2023-10-30T13:35:15.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 -- 172.21.15.31:0/1737341490 learned_addr learned my addr 172.21.15.31:0/1737341490 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:15.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63a4ba640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406c420 0x7fb634158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb6394b8640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb63406bb40 0x7fb63415a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 -- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406c420 msgr2=0x7fb634158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406c420 0x7fb634158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 -- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb63406bb40 msgr2=0x7fb63415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb63406bb40 0x7fb63415a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 -- 172.21.15.31:0/1737341490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300094d0 con 0x7fb63406b760 2023-10-30T13:35:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb639cb9640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb634159d20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb63001ab80 tx=0x7fb63001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb62a7fc640 1 -- 172.21.15.31:0/1737341490 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6300aa5f0 con 0x7fb63406b760 2023-10-30T13:35:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb62a7fc640 1 -- 172.21.15.31:0/1737341490 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6300b3800 con 0x7fb63406b760 2023-10-30T13:35:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6340c49a0 con 0x7fb63406b760 2023-10-30T13:35:15.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb62a7fc640 1 -- 172.21.15.31:0/1737341490 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6300b3b90 con 0x7fb63406b760 2023-10-30T13:35:15.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.408+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6341f0090 con 0x7fb63406b760 2023-10-30T13:35:15.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.412+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb634066510 con 0x7fb63406b760 2023-10-30T13:35:15.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.412+0000 7fb62a7fc640 1 -- 172.21.15.31:0/1737341490 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb6300b3cf0 con 0x7fb63406b760 2023-10-30T13:35:15.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.412+0000 7fb62a7fc640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6180664a0 0x7fb618068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.412+0000 7fb62a7fc640 1 -- 172.21.15.31:0/1737341490 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6300a7070 con 0x7fb63406b760 2023-10-30T13:35:15.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.412+0000 7fb6394b8640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6180664a0 0x7fb618068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.412+0000 7fb6394b8640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6180664a0 0x7fb618068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb6340ca880 tx=0x7fb624008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.416+0000 7fb62a7fc640 1 -- 172.21.15.31:0/1737341490 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6300ebc90 con 0x7fb63406b760 2023-10-30T13:35:15.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.576+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb63404e170 con 0x7fb63406b760 2023-10-30T13:35:15.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.576+0000 7fb62a7fc640 1 -- 172.21.15.31:0/1737341490 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fb6300ef2b0 con 0x7fb63406b760 2023-10-30T13:35:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:15.581 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:15.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6180664a0 msgr2=0x7fb618068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6180664a0 0x7fb618068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb6340ca880 tx=0x7fb624008040 comp rx=0 tx=0).stop 2023-10-30T13:35:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 msgr2=0x7fb634159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb634159d20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb63001ab80 tx=0x7fb63001ad10 comp rx=0 tx=0).stop 2023-10-30T13:35:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 shutdown_connections 2023-10-30T13:35:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6180664a0 0x7fb618068960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb63406c420 0x7fb634158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb63406bb40 0x7fb63415a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 --2- 172.21.15.31:0/1737341490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63406b760 0x7fb634159d20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 >> 172.21.15.31:0/1737341490 conn(0x7fb6340cac70 msgr2=0x7fb6340736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 shutdown_connections 2023-10-30T13:35:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.580+0000 7fb63bf1d640 1 -- 172.21.15.31:0/1737341490 wait complete. 2023-10-30T13:35:15.602 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T13:35:15.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.740+0000 7ff147165640 1 Processor -- start 2023-10-30T13:35:15.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.740+0000 7ff147165640 1 -- start start 2023-10-30T13:35:15.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.740+0000 7ff147165640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c4a10 0x7ff14015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.740+0000 7ff147165640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff140159f20 con 0x7ff1400c42a0 2023-10-30T13:35:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.740+0000 7ff147165640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14015d620 con 0x7ff140153c80 2023-10-30T13:35:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.740+0000 7ff147165640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14015dec0 con 0x7ff1400c4a10 2023-10-30T13:35:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff145702640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff1400c42a0 0x7ff1400ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51784/0 (socket says 172.21.15.31:51784) 2023-10-30T13:35:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff145702640 1 -- 172.21.15.31:0/3592609668 learned_addr learned my addr 172.21.15.31:0/3592609668 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13ffff640 1 --2- 172.21.15.31:0/3592609668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c4a10 0x7ff14015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13f7fe640 1 -- 172.21.15.31:0/3592609668 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692723784 0 0) 0x7ff140159f20 con 0x7ff1400c42a0 2023-10-30T13:35:15.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13f7fe640 1 -- 172.21.15.31:0/3592609668 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff130003530 con 0x7ff1400c42a0 2023-10-30T13:35:15.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13f7fe640 1 -- 172.21.15.31:0/3592609668 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978603839 0 0) 0x7ff14015d620 con 0x7ff140153c80 2023-10-30T13:35:15.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13ffff640 1 -- 172.21.15.31:0/3592609668 >> v1:172.21.15.33:6789/0 conn(0x7ff140153c80 legacy=0x7ff14015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13ffff640 1 -- 172.21.15.31:0/3592609668 >> v1:172.21.15.31:6789/0 conn(0x7ff1400c42a0 legacy=0x7ff1400ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13ffff640 1 -- 172.21.15.31:0/3592609668 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14015d900 con 0x7ff1400c4a10 2023-10-30T13:35:15.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.744+0000 7ff13ffff640 1 --2- 172.21.15.31:0/3592609668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c4a10 0x7ff14015cf90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0097f0 tx=0x7ff12c01ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92351ad770186a4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff13f7fe640 1 -- 172.21.15.31:0/3592609668 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0a89b0 con 0x7ff1400c4a10 2023-10-30T13:35:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff13f7fe640 1 -- 172.21.15.31:0/3592609668 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c0a8b10 con 0x7ff1400c4a10 2023-10-30T13:35:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff13f7fe640 1 -- 172.21.15.31:0/3592609668 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0a8ea0 con 0x7ff1400c4a10 2023-10-30T13:35:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 -- 172.21.15.31:0/3592609668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c4a10 msgr2=0x7ff14015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 --2- 172.21.15.31:0/3592609668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c4a10 0x7ff14015cf90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0097f0 tx=0x7ff12c01ecb0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 -- 172.21.15.31:0/3592609668 shutdown_connections 2023-10-30T13:35:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 --2- 172.21.15.31:0/3592609668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c4a10 0x7ff14015cf90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 -- 172.21.15.31:0/3592609668 >> 172.21.15.31:0/3592609668 conn(0x7ff1400715c0 msgr2=0x7ff1400719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 -- 172.21.15.31:0/3592609668 shutdown_connections 2023-10-30T13:35:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 -- 172.21.15.31:0/3592609668 wait complete. 2023-10-30T13:35:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 Processor -- start 2023-10-30T13:35:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.748+0000 7ff147165640 1 -- start start 2023-10-30T13:35:15.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c42a0 0x7ff1400cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1400c4a10 0x7ff1401f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff140153c80 0x7ff1401f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14015bc80 con 0x7ff140153c80 2023-10-30T13:35:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400c5bf0 con 0x7ff1400c4a10 2023-10-30T13:35:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400cdbb0 con 0x7ff1400c42a0 2023-10-30T13:35:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff144f01640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c42a0 0x7ff1400cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff145702640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff140153c80 0x7ff1401f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff144f01640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c42a0 0x7ff1400cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53312/0 (socket says 172.21.15.31:53312) 2023-10-30T13:35:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff144f01640 1 -- 172.21.15.31:0/2558167963 learned_addr learned my addr 172.21.15.31:0/2558167963 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff13ffff640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1400c4a10 0x7ff1401f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff145702640 1 -- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c42a0 msgr2=0x7ff1400cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff145702640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c42a0 0x7ff1400cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff145702640 1 -- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1400c4a10 msgr2=0x7ff1401f0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff145702640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1400c4a10 0x7ff1401f0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff145702640 1 -- 172.21.15.31:0/2558167963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c009420 con 0x7ff140153c80 2023-10-30T13:35:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff145702640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff140153c80 0x7ff1401f08c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff13800ca80 tx=0x7ff13800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff13d7fa640 1 -- 172.21.15.31:0/2558167963 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13800ac30 con 0x7ff140153c80 2023-10-30T13:35:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff13d7fa640 1 -- 172.21.15.31:0/2558167963 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13800ad90 con 0x7ff140153c80 2023-10-30T13:35:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff13d7fa640 1 -- 172.21.15.31:0/2558167963 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138012600 con 0x7ff140153c80 2023-10-30T13:35:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1400cde50 con 0x7ff140153c80 2023-10-30T13:35:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1400ceef0 con 0x7ff140153c80 2023-10-30T13:35:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.752+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14004e170 con 0x7ff140153c80 2023-10-30T13:35:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.756+0000 7ff13d7fa640 1 -- 172.21.15.31:0/2558167963 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff138007070 con 0x7ff140153c80 2023-10-30T13:35:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.756+0000 7ff13d7fa640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1240664d0 0x7ff124068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.756+0000 7ff13d7fa640 1 -- 172.21.15.31:0/2558167963 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff138083bd0 con 0x7ff140153c80 2023-10-30T13:35:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.756+0000 7ff144f01640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1240664d0 0x7ff124068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.756+0000 7ff144f01640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1240664d0 0x7ff124068990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff134000f70 tx=0x7ff134009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.760+0000 7ff13d7fa640 1 -- 172.21.15.31:0/2558167963 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff138051190 con 0x7ff140153c80 2023-10-30T13:35:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.916+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff140151420 con 0x7ff140153c80 2023-10-30T13:35:15.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff13d7fa640 1 -- 172.21.15.31:0/2558167963 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7ff138017090 con 0x7ff140153c80 2023-10-30T13:35:15.924 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:15.924 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:15.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1240664d0 msgr2=0x7ff124068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1240664d0 0x7ff124068990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff134000f70 tx=0x7ff134009040 comp rx=0 tx=0).stop 2023-10-30T13:35:15.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff140153c80 msgr2=0x7ff1401f08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:15.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff140153c80 0x7ff1401f08c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff13800ca80 tx=0x7ff13800cf40 comp rx=0 tx=0).stop 2023-10-30T13:35:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 shutdown_connections 2023-10-30T13:35:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1240664d0 0x7ff124068990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff140153c80 0x7ff1401f08c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1400c4a10 0x7ff1401f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 --2- 172.21.15.31:0/2558167963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1400c42a0 0x7ff1400cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 >> 172.21.15.31:0/2558167963 conn(0x7ff1400715c0 msgr2=0x7ff1400c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 shutdown_connections 2023-10-30T13:35:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:15.920+0000 7ff147165640 1 -- 172.21.15.31:0/2558167963 wait complete. 2023-10-30T13:35:15.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T13:35:15.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T13:35:15.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T13:35:15.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T13:35:15.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T13:35:15.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T13:35:15.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T13:35:15.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T13:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.188+0000 7f256a074640 1 Processor -- start 2023-10-30T13:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.188+0000 7f256a074640 1 -- start start 2023-10-30T13:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f87658ea640 1 Processor -- start 2023-10-30T13:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f87658ea640 1 -- start start 2023-10-30T13:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f256a074640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f256406d830 0x7f25641562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f256a074640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f256406e3f0 0x7f256414f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f256a074640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25640c24c0 con 0x7f256406dc10 2023-10-30T13:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f256a074640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25640c49f0 con 0x7f256406e3f0 2023-10-30T13:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f256a074640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25641567e0 con 0x7f256406d830 2023-10-30T13:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f2569072640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f256406d830 0x7f25641562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f2568871640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f256406dc10 0x7f25640c6310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51796/0 (socket says 172.21.15.31:51796) 2023-10-30T13:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f2568871640 1 -- 172.21.15.31:0/1990460607 learned_addr learned my addr 172.21.15.31:0/1990460607 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f2569873640 1 --2- 172.21.15.31:0/1990460607 >> v2:172.21.15.31:3300/0 conn(0x7f256406e3f0 0x7f256414f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f87658ea640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f87601538a0 0x7f876006ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.192+0000 7f87658ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f876015d880 0x7f87601597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f255bfff640 1 -- 172.21.15.31:0/1990460607 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261456778 0 0) 0x7f25640c24c0 con 0x7f256406dc10 2023-10-30T13:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f255bfff640 1 -- 172.21.15.31:0/1990460607 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2550005740 con 0x7f256406dc10 2023-10-30T13:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f2569072640 1 -- 172.21.15.31:0/1990460607 >> v2:172.21.15.31:3300/0 conn(0x7f256406e3f0 msgr2=0x7f256414f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f2569072640 1 --2- 172.21.15.31:0/1990460607 >> v2:172.21.15.31:3300/0 conn(0x7f256406e3f0 0x7f256414f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f2569072640 1 -- 172.21.15.31:0/1990460607 >> v1:172.21.15.31:6789/0 conn(0x7f256406dc10 legacy=0x7f25640c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875e7fc640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f876015d880 0x7f87601597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f87658ea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8760153c80 con 0x7f87600c6cb0 2023-10-30T13:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f87658ea640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876015d460 con 0x7f87601538a0 2023-10-30T13:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f87658ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876015dcc0 con 0x7f876015d880 2023-10-30T13:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875f7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f87600c6cb0 0x7f87600c7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51808/0 (socket says 172.21.15.31:51808) 2023-10-30T13:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875f7fe640 1 -- 172.21.15.31:0/3029694821 learned_addr learned my addr 172.21.15.31:0/3029694821 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 --2- 172.21.15.31:0/3029694821 >> v2:172.21.15.31:3300/0 conn(0x7f87601538a0 0x7f876006ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739506569 0 0) 0x7f8760153c80 con 0x7f87600c6cb0 2023-10-30T13:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8748003530 con 0x7f87600c6cb0 2023-10-30T13:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3513549240 0 0) 0x7f8748003530 con 0x7f87600c6cb0 2023-10-30T13:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 >> v2:172.21.15.33:3300/0 conn(0x7f876015d880 msgr2=0x7f87601597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 --2- 172.21.15.31:0/3029694821 >> v2:172.21.15.33:3300/0 conn(0x7f876015d880 0x7f87601597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 >> v2:172.21.15.31:3300/0 conn(0x7f87601538a0 msgr2=0x7f876006ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 --2- 172.21.15.31:0/3029694821 >> v2:172.21.15.31:3300/0 conn(0x7f87601538a0 0x7f876006ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876015a060 con 0x7f87600c6cb0 2023-10-30T13:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f8750001680 con 0x7f87600c6cb0 2023-10-30T13:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 >> v1:172.21.15.31:6789/0 conn(0x7f87600c6cb0 legacy=0x7f87600c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8748003980 0x7f8748003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87480047d0 0x7f8748004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8748005190 0x7f874800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876015d460 con 0x7f87480047d0 2023-10-30T13:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876015dcc0 con 0x7f8748003980 2023-10-30T13:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874800c140 con 0x7f8748005190 2023-10-30T13:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 --2- 172.21.15.31:0/3029694821 >> v2:172.21.15.31:3300/0 conn(0x7f87601538a0 0x7f876006ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875f7fe640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8748005190 0x7f874800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8748003980 0x7f8748003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f2569072640 1 -- 172.21.15.31:0/1990460607 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2564072d00 con 0x7f256406d830 2023-10-30T13:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875e7fc640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87480047d0 0x7f8748004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 -- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8748005190 msgr2=0x7f874800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8748005190 0x7f874800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 -- 172.21.15.31:0/3029694821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87480047d0 msgr2=0x7f8748004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87480047d0 0x7f8748004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.196+0000 7f875effd640 1 -- 172.21.15.31:0/3029694821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876015a060 con 0x7f8748003980 2023-10-30T13:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f875f7fe640 1 -- 172.21.15.31:0/3029694821 reap_dead start 2023-10-30T13:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f2569072640 1 --2- 172.21.15.31:0/1990460607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 0x7f25641562a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f256406af60 tx=0x7f256001c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0665d5a5f601463 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f255bfff640 1 -- 172.21.15.31:0/1990460607 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25600a5880 con 0x7f256406d830 2023-10-30T13:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f255bfff640 1 -- 172.21.15.31:0/1990460607 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25600a59e0 con 0x7f256406d830 2023-10-30T13:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7fb392d0f640 1 Processor -- start 2023-10-30T13:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7fb392d0f640 1 -- start start 2023-10-30T13:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f256a074640 1 -- 172.21.15.31:0/1990460607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 msgr2=0x7f25641562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f256a074640 1 --2- 172.21.15.31:0/1990460607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 0x7f25641562a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f256406af60 tx=0x7f256001c8d0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f256a074640 1 -- 172.21.15.31:0/1990460607 shutdown_connections 2023-10-30T13:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f256a074640 1 --2- 172.21.15.31:0/1990460607 >> v2:172.21.15.31:3300/0 conn(0x7f256406e3f0 0x7f256414f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f256a074640 1 --2- 172.21.15.31:0/1990460607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 0x7f25641562a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.200+0000 7f256a074640 1 -- 172.21.15.31:0/1990460607 >> 172.21.15.31:0/1990460607 conn(0x7f25640c7560 msgr2=0x7f25640c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f256a074640 1 -- 172.21.15.31:0/1990460607 shutdown_connections 2023-10-30T13:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f256a074640 1 -- 172.21.15.31:0/1990460607 wait complete. 2023-10-30T13:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f875effd640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8748003980 0x7f8748003da0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f875400aaa0 tx=0x7f875401c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e54fc6ad9bbdf401 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb392d0f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb38c070520 0x7fb38c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb392d0f640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c0c2310 con 0x7fb38c150830 2023-10-30T13:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb392d0f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c156710 con 0x7fb38c0cb7d0 2023-10-30T13:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb392d0f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38c06bc10 con 0x7fb38c070520 2023-10-30T13:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f256a074640 1 Processor -- start 2023-10-30T13:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f256a074640 1 -- start start 2023-10-30T13:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 0x7f25641ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f256406dc10 0x7f25641538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f256406e3f0 0x7f2564153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256414a1c0 con 0x7f256406e3f0 2023-10-30T13:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25641f9d30 con 0x7f256406d830 2023-10-30T13:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25641fa260 con 0x7f256406dc10 2023-10-30T13:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87540a9620 con 0x7f8748003980 2023-10-30T13:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87540a9780 con 0x7f8748003980 2023-10-30T13:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f875dffb640 1 -- 172.21.15.31:0/3029694821 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87540a9b30 con 0x7f8748003980 2023-10-30T13:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb390aab640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb38c150830 0x7fb38c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51824/0 (socket says 172.21.15.31:51824) 2023-10-30T13:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb390aab640 1 -- 172.21.15.31:0/511549816 learned_addr learned my addr 172.21.15.31:0/511549816 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb3912ac640 1 --2- 172.21.15.31:0/511549816 >> v2:172.21.15.31:3300/0 conn(0x7fb38c070520 0x7fb38c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb38b7fe640 1 -- 172.21.15.31:0/511549816 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544908283 0 0) 0x7fb38c0c2310 con 0x7fb38c150830 2023-10-30T13:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb38b7fe640 1 -- 172.21.15.31:0/511549816 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb37c003530 con 0x7fb38c150830 2023-10-30T13:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb3912ac640 1 -- 172.21.15.31:0/511549816 >> v1:172.21.15.33:6789/0 conn(0x7fb38c0cb7d0 legacy=0x7fb38c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb3912ac640 1 -- 172.21.15.31:0/511549816 >> v1:172.21.15.31:6789/0 conn(0x7fb38c150830 legacy=0x7fb38c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7fb3912ac640 1 -- 172.21.15.31:0/511549816 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0c4360 con 0x7fb38c070520 2023-10-30T13:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569072640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 0x7f25641ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569072640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 0x7f25641ebe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60566/0 (socket says 172.21.15.31:60566) 2023-10-30T13:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569072640 1 -- 172.21.15.31:0/2151650727 learned_addr learned my addr 172.21.15.31:0/2151650727 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569873640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f256406e3f0 0x7f2564153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f87658ea640 1 -- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8748003980 msgr2=0x7f8748003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.204+0000 7f87658ea640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8748003980 0x7f8748003da0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f875400aaa0 tx=0x7f875401c6b0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f87658ea640 1 -- 172.21.15.31:0/3029694821 shutdown_connections 2023-10-30T13:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f87658ea640 1 --2- 172.21.15.31:0/3029694821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8748003980 0x7f8748003da0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f87658ea640 1 -- 172.21.15.31:0/3029694821 >> 172.21.15.31:0/3029694821 conn(0x7f876006d810 msgr2=0x7f876006dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569873640 1 -- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f256406dc10 msgr2=0x7f25641538e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T13:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569873640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f256406dc10 0x7f25641538e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569873640 1 -- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 msgr2=0x7f25641ebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569873640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256406d830 0x7f25641ebe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569873640 1 -- 172.21.15.31:0/2151650727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2560009a70 con 0x7f256406e3f0 2023-10-30T13:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2569873640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f256406e3f0 0x7f2564153e20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f256406cd20 tx=0x7f255c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f87658ea640 1 -- 172.21.15.31:0/3029694821 shutdown_connections 2023-10-30T13:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f87658ea640 1 -- 172.21.15.31:0/3029694821 wait complete. 2023-10-30T13:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f8144bc5640 1 Processor -- start 2023-10-30T13:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 Processor -- start 2023-10-30T13:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c011070 con 0x7f256406e3f0 2023-10-30T13:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c00ce70 con 0x7f256406e3f0 2023-10-30T13:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c009740 con 0x7f256406e3f0 2023-10-30T13:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25640ca920 con 0x7f256406e3f0 2023-10-30T13:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25640ca360 con 0x7f256406e3f0 2023-10-30T13:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25640c5df0 con 0x7f256406e3f0 2023-10-30T13:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 -- start start 2023-10-30T13:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.208+0000 7f8144bc5640 1 -- start start 2023-10-30T13:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f8144bc5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380dd4e0 0x7f8138046590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f8144bc5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f81380e38d0 0x7f813804bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f8144bc5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8138045580 con 0x7f813804bb90 2023-10-30T13:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f8144bc5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8138046100 con 0x7f81380e38d0 2023-10-30T13:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f8144bc5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81380e3d60 con 0x7f81380dd4e0 2023-10-30T13:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f813dd9b640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f813804bb90 0x7f81380e31c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45714/0 (socket says 172.21.15.31:45714) 2023-10-30T13:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f813dd9b640 1 -- 172.21.15.31:0/2223149724 learned_addr learned my addr 172.21.15.31:0/2223149724 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7fbdeffff640 1 Processor -- start 2023-10-30T13:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7fbdeffff640 1 -- start start 2023-10-30T13:35:16.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7fb3912ac640 1 --2- 172.21.15.31:0/511549816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb38c070520 0x7fb38c070990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb384009d90 tx=0x7fb38401f100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2cdb98b95a47f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7fb38b7fe640 1 -- 172.21.15.31:0/511549816 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840af600 con 0x7fb38c070520 2023-10-30T13:35:16.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f3385594640 1 Processor -- start 2023-10-30T13:35:16.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7fb38b7fe640 1 -- 172.21.15.31:0/511549816 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3840ab050 con 0x7fb38c070520 2023-10-30T13:35:16.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7fb38b7fe640 1 -- 172.21.15.31:0/511549816 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840af980 con 0x7fb38c070520 2023-10-30T13:35:16.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f3385594640 1 -- start start 2023-10-30T13:35:16.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8748003980 0x7f87600ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87600cb780 0x7f87601eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f87600c8a80 0x7f87601ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f875e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87600cb780 0x7f87601eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f875e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87600cb780 0x7f87601eebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60592/0 (socket says 172.21.15.31:60592) 2023-10-30T13:35:16.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f875effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8748003980 0x7f87600ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f875e7fc640 1 -- 172.21.15.31:0/134582758 learned_addr learned my addr 172.21.15.31:0/134582758 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87600cb420 con 0x7f8748003980 2023-10-30T13:35:16.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87600c9c60 con 0x7f87600cb780 2023-10-30T13:35:16.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f87658ea640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87601ee720 con 0x7f87600c8a80 2023-10-30T13:35:16.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f813e59c640 1 --2- 172.21.15.31:0/2223149724 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380dd4e0 0x7f8138046590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f813ed9d640 1 --2- 172.21.15.31:0/2223149724 >> v2:172.21.15.33:3300/0 conn(0x7f81380e38d0 0x7f813804bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f255c0098a0 con 0x7f256406e3f0 2023-10-30T13:35:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.212+0000 7f875f7fe640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f87600c8a80 0x7f87601ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdeffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbdf006da70 0x7fbdf006de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fb392d0f640 1 -- 172.21.15.31:0/511549816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb38c070520 msgr2=0x7fb38c070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f813d59a640 1 -- 172.21.15.31:0/2223149724 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807821286 0 0) 0x7f8138045580 con 0x7f813804bb90 2023-10-30T13:35:16.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f813d59a640 1 -- 172.21.15.31:0/2223149724 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8128003530 con 0x7f813804bb90 2023-10-30T13:35:16.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdeffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f875e7fc640 1 -- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f87600c8a80 msgr2=0x7f87601ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f875e7fc640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f87600c8a80 0x7f87601ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f875e7fc640 1 -- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8748003980 msgr2=0x7f87600ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fb392d0f640 1 --2- 172.21.15.31:0/511549816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb38c070520 0x7fb38c070990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb384009d90 tx=0x7fb38401f100 comp rx=0 tx=0).stop 2023-10-30T13:35:16.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f813e59c640 1 -- 172.21.15.31:0/2223149724 >> v2:172.21.15.33:3300/0 conn(0x7f81380e38d0 msgr2=0x7f813804bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f813e59c640 1 --2- 172.21.15.31:0/2223149724 >> v2:172.21.15.33:3300/0 conn(0x7f81380e38d0 0x7f813804bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f813e59c640 1 -- 172.21.15.31:0/2223149724 >> v1:172.21.15.33:6789/0 conn(0x7f813804bb90 legacy=0x7f81380e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fb392d0f640 1 -- 172.21.15.31:0/511549816 shutdown_connections 2023-10-30T13:35:16.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fb392d0f640 1 --2- 172.21.15.31:0/511549816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb38c070520 0x7fb38c070990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fb392d0f640 1 -- 172.21.15.31:0/511549816 >> 172.21.15.31:0/511549816 conn(0x7fb38c0cac70 msgr2=0x7fb38c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f875e7fc640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8748003980 0x7f87600ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f875e7fc640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876014df00 con 0x7f87600cb780 2023-10-30T13:35:16.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdeffff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf00c7180 con 0x7fbdf006fc50 2023-10-30T13:35:16.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdeffff640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf0161f30 con 0x7fbdf006da70 2023-10-30T13:35:16.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdeffff640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf006e390 con 0x7fbdf01615c0 2023-10-30T13:35:16.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbde67fc640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fbdf006fc50 0x7fbdf00c6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45726/0 (socket says 172.21.15.31:45726) 2023-10-30T13:35:16.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbde67fc640 1 -- 172.21.15.31:0/154276806 learned_addr learned my addr 172.21.15.31:0/154276806 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f813e59c640 1 -- 172.21.15.31:0/2223149724 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380dd8c0 con 0x7f81380dd4e0 2023-10-30T13:35:16.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f3385594640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3380151840 0x7f33800c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdef7fe640 1 --2- 172.21.15.31:0/154276806 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f875effd640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8748003980 0x7f87600ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:16.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f3385594640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33800c5030 con 0x7f33800cb8b0 2023-10-30T13:35:16.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f3385594640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3380157830 con 0x7f3380151c20 2023-10-30T13:35:16.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f3385594640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3380157b10 con 0x7f3380151840 2023-10-30T13:35:16.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f875e7fc640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87600cb780 0x7f87601eebb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8758007fa0 tx=0x7f8758011660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdee7fc640 1 -- 172.21.15.31:0/154276806 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213867289 0 0) 0x7fbdf00c7180 con 0x7fbdf006fc50 2023-10-30T13:35:16.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdeeffd640 1 --2- 172.21.15.31:0/154276806 >> v2:172.21.15.31:3300/0 conn(0x7fbdf006da70 0x7fbdf006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdef7fe640 1 -- 172.21.15.31:0/154276806 >> v2:172.21.15.31:3300/0 conn(0x7fbdf006da70 msgr2=0x7fbdf006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdef7fe640 1 --2- 172.21.15.31:0/154276806 >> v2:172.21.15.31:3300/0 conn(0x7fbdf006da70 0x7fbdf006de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758016470 con 0x7f87600cb780 2023-10-30T13:35:16.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758016610 con 0x7f87600cb780 2023-10-30T13:35:16.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87580169c0 con 0x7f87600cb780 2023-10-30T13:35:16.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdef7fe640 1 -- 172.21.15.31:0/154276806 >> v1:172.21.15.33:6789/0 conn(0x7fbdf006fc50 legacy=0x7fbdf00c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdef7fe640 1 -- 172.21.15.31:0/154276806 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf015dd20 con 0x7fbdf01615c0 2023-10-30T13:35:16.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fbdeeffd640 1 --2- 172.21.15.31:0/154276806 >> v2:172.21.15.31:3300/0 conn(0x7fbdf006da70 0x7fbdf006de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:16.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f337e7fc640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f3380151c20 0x7f3380072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45736/0 (socket says 172.21.15.31:45736) 2023-10-30T13:35:16.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f337e7fc640 1 -- 172.21.15.31:0/2144731570 learned_addr learned my addr 172.21.15.31:0/2144731570 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f337effd640 1 --2- 172.21.15.31:0/2144731570 >> v2:172.21.15.31:3300/0 conn(0x7f3380151840 0x7f33800c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7fb392d0f640 1 -- 172.21.15.31:0/511549816 shutdown_connections 2023-10-30T13:35:16.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f2559ffb640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2548066570 0x7f2548068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f255c082dd0 con 0x7f256406e3f0 2023-10-30T13:35:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f2559ffb640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f254806e790 0x7f2548070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f2569072640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2548066570 0x7f2548068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 --> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2548071260 con 0x7f254806e790 2023-10-30T13:35:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.216+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f255c0853f0 con 0x7f256406e3f0 2023-10-30T13:35:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f87658ea640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87600cb140 con 0x7f87600cb780 2023-10-30T13:35:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc713fff640 1 Processor -- start 2023-10-30T13:35:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f87658ea640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876014e180 con 0x7f87600cb780 2023-10-30T13:35:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc713fff640 1 -- start start 2023-10-30T13:35:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f2568871640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f254806e790 0x7f2548070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f2569072640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2548066570 0x7f2548068a30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f25600194e0 tx=0x7f25600a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fb392d0f640 1 -- 172.21.15.31:0/511549816 wait complete. 2023-10-30T13:35:16.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8738000fd0 con 0x7f87600cb780 2023-10-30T13:35:16.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f337dffb640 1 -- 172.21.15.31:0/2144731570 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692740326 0 0) 0x7f3380157830 con 0x7f3380151c20 2023-10-30T13:35:16.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f337dffb640 1 -- 172.21.15.31:0/2144731570 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3368003530 con 0x7f3380151c20 2023-10-30T13:35:16.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f337dffb640 1 -- 172.21.15.31:0/2144731570 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994094941 0 0) 0x7f33800c5030 con 0x7f33800cb8b0 2023-10-30T13:35:16.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f337dffb640 1 -- 172.21.15.31:0/2144731570 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380157830 con 0x7f33800cb8b0 2023-10-30T13:35:16.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f337effd640 1 -- 172.21.15.31:0/2144731570 >> v1:172.21.15.33:6789/0 conn(0x7f3380151c20 legacy=0x7f3380072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f337effd640 1 -- 172.21.15.31:0/2144731570 >> v1:172.21.15.31:6789/0 conn(0x7f33800cb8b0 legacy=0x7f338006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f337effd640 1 -- 172.21.15.31:0/2144731570 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3380157df0 con 0x7f3380151840 2023-10-30T13:35:16.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f813e59c640 1 --2- 172.21.15.31:0/2223149724 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380dd4e0 0x7f8138046590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8140069480 tx=0x7f81400973d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=544f76180867c804 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc713fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc7141539d0 0x7fc7140bc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc713fff640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7140bcf10 con 0x7fc7141535f0 2023-10-30T13:35:16.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f2568871640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f254806e790 0x7f2548070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc713fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7140c5880 con 0x7fc7140bcaf0 2023-10-30T13:35:16.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc713fff640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7140c5b60 con 0x7fc7141539d0 2023-10-30T13:35:16.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f813d59a640 1 -- 172.21.15.31:0/2223149724 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140126600 con 0x7f81380dd4e0 2023-10-30T13:35:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f813d59a640 1 -- 172.21.15.31:0/2223149724 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8140122050 con 0x7f81380dd4e0 2023-10-30T13:35:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7f813d59a640 1 -- 172.21.15.31:0/2223149724 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140126a20 con 0x7f81380dd4e0 2023-10-30T13:35:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fbdef7fe640 1 --2- 172.21.15.31:0/154276806 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01619f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009d90 tx=0x7fbde801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4cf655fa4eada server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc7137fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fc7140bcaf0 0x7fc71406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45738/0 (socket says 172.21.15.31:45738) 2023-10-30T13:35:16.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc7137fe640 1 -- 172.21.15.31:0/1618054631 learned_addr learned my addr 172.21.15.31:0/1618054631 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fb392d0f640 1 Processor -- start 2023-10-30T13:35:16.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fb392d0f640 1 -- start start 2023-10-30T13:35:16.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc70a7fc640 1 --2- 172.21.15.31:0/1618054631 >> v2:172.21.15.33:3300/0 conn(0x7fc7141539d0 0x7fc7140bc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc7127fc640 1 -- 172.21.15.31:0/1618054631 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659287217 0 0) 0x7fc7140c5880 con 0x7fc7140bcaf0 2023-10-30T13:35:16.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8758016b20 con 0x7f87600cb780 2023-10-30T13:35:16.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 -- 172.21.15.31:0/2223149724 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380dd4e0 msgr2=0x7f8138046590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 --2- 172.21.15.31:0/2223149724 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380dd4e0 0x7f8138046590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8140069480 tx=0x7f81400973d0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 -- 172.21.15.31:0/2223149724 shutdown_connections 2023-10-30T13:35:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 --2- 172.21.15.31:0/2223149724 >> v2:172.21.15.33:3300/0 conn(0x7f81380e38d0 0x7f813804bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 --2- 172.21.15.31:0/2223149724 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380dd4e0 0x7f8138046590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 -- 172.21.15.31:0/2223149724 >> 172.21.15.31:0/2223149724 conn(0x7f81380502b0 msgr2=0x7f81380506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdee7fc640 1 -- 172.21.15.31:0/154276806 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80a59e0 con 0x7fbdf01615c0 2023-10-30T13:35:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdee7fc640 1 -- 172.21.15.31:0/154276806 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde80a5b40 con 0x7fbdf01615c0 2023-10-30T13:35:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdee7fc640 1 -- 172.21.15.31:0/154276806 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80ad440 con 0x7fbdf01615c0 2023-10-30T13:35:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.220+0000 7fc7127fc640 1 -- 172.21.15.31:0/1618054631 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc700003530 con 0x7fc7140bcaf0 2023-10-30T13:35:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fc70a7fc640 1 -- 172.21.15.31:0/1618054631 >> v1:172.21.15.33:6789/0 conn(0x7fc7140bcaf0 legacy=0x7fc71406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fc70a7fc640 1 -- 172.21.15.31:0/1618054631 >> v1:172.21.15.31:6789/0 conn(0x7fc7141535f0 legacy=0x7fc71415a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fc70a7fc640 1 -- 172.21.15.31:0/1618054631 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140c5e40 con 0x7fc7141539d0 2023-10-30T13:35:16.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f1bac794640 1 Processor -- start 2023-10-30T13:35:16.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb392d0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb38c070520 0x7fb38c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 -- 172.21.15.31:0/2223149724 shutdown_connections 2023-10-30T13:35:16.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 -- 172.21.15.31:0/154276806 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 msgr2=0x7fbdf01619f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 --2- 172.21.15.31:0/154276806 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01619f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009d90 tx=0x7fbde801c920 comp rx=0 tx=0).stop 2023-10-30T13:35:16.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8144bc5640 1 -- 172.21.15.31:0/2223149724 wait complete. 2023-10-30T13:35:16.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb392d0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb38c0cb7d0 0x7fb38c157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb392d0f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb38c150830 0x7fb38c157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb392d0f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c15a0a0 con 0x7fb38c070520 2023-10-30T13:35:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb392d0f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c159490 con 0x7fb38c0cb7d0 2023-10-30T13:35:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb392d0f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c159770 con 0x7fb38c150830 2023-10-30T13:35:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 -- 172.21.15.31:0/154276806 shutdown_connections 2023-10-30T13:35:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 --2- 172.21.15.31:0/154276806 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01619f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 --2- 172.21.15.31:0/154276806 >> v2:172.21.15.31:3300/0 conn(0x7fbdf006da70 0x7fbdf006de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 -- 172.21.15.31:0/154276806 >> 172.21.15.31:0/154276806 conn(0x7fbdf00cab50 msgr2=0x7fbdf00caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f1bac794640 1 -- start start 2023-10-30T13:35:16.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb3912ac640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb38c150830 0x7fb38c157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb38bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb38c0cb7d0 0x7fb38c157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb38bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb38c0cb7d0 0x7fb38c157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60618/0 (socket says 172.21.15.31:60618) 2023-10-30T13:35:16.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb38bfff640 1 -- 172.21.15.31:0/2107619355 learned_addr learned my addr 172.21.15.31:0/2107619355 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb3912ac640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb38c150830 0x7fb38c157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53362/0 (socket says 172.21.15.31:53362) 2023-10-30T13:35:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb3912ac640 1 -- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb38c0cb7d0 msgr2=0x7fb38c157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb3912ac640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb38c0cb7d0 0x7fb38c157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb3912ac640 1 -- 172.21.15.31:0/2107619355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb38c070520 msgr2=0x7fb38c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb3912ac640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb38c070520 0x7fb38c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fb3912ac640 1 -- 172.21.15.31:0/2107619355 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb384009a70 con 0x7fb38c150830 2023-10-30T13:35:16.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 -- 172.21.15.31:0/154276806 shutdown_connections 2023-10-30T13:35:16.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8747fff640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f874c066570 0x7f874c068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8758084b50 con 0x7f87600cb780 2023-10-30T13:35:16.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8747fff640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f874c06e790 0x7f874c070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f874c071260 con 0x7f874c06e790 2023-10-30T13:35:16.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87580892a0 con 0x7f87600cb780 2023-10-30T13:35:16.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f875effd640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f874c066570 0x7f874c068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f875f7fe640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f874c06e790 0x7f874c070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7fbdeffff640 1 -- 172.21.15.31:0/154276806 wait complete. 2023-10-30T13:35:16.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1bac794640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f1ba406b760 0x7f1ba415e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.224+0000 7f337effd640 1 --2- 172.21.15.31:0/2144731570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3380151840 0x7f33800c49a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f33700097c0 tx=0x7f337001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6467b9961a26118 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fb3912ac640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb38c150830 0x7fb38c157fe0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb3840af000 tx=0x7fb38401bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== osd.0 v2:172.21.15.31:6800/1469638853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f2548071260 con 0x7f254806e790 2023-10-30T13:35:16.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1bac794640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1ba406bb40 0x7f1ba4157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1bac794640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 0x7f1ba4157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1bac794640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba4071e00 con 0x7f1ba406bb40 2023-10-30T13:35:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1bac794640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba4070020 con 0x7f1ba406b760 2023-10-30T13:35:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1bac794640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba415eb90 con 0x7f1ba406c420 2023-10-30T13:35:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f1ba406b760 0x7f1ba415e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f1ba406b760 0x7f1ba415e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60620/0 (socket says 172.21.15.31:60620) 2023-10-30T13:35:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 -- 172.21.15.31:0/3456610085 learned_addr learned my addr 172.21.15.31:0/3456610085 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f875effd640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f874c066570 0x7f874c068a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87540ae000 tx=0x7f87540046d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f875f7fe640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f874c06e790 0x7f874c070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f337dffb640 1 -- 172.21.15.31:0/2144731570 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33700a8970 con 0x7f3380151840 2023-10-30T13:35:16.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f337dffb640 1 -- 172.21.15.31:0/2144731570 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33700a8ad0 con 0x7f3380151840 2023-10-30T13:35:16.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f337dffb640 1 -- 172.21.15.31:0/2144731570 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33700a8e60 con 0x7f3380151840 2023-10-30T13:35:16.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840ba420 con 0x7fb38c150830 2023-10-30T13:35:16.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3840ba580 con 0x7fb38c150830 2023-10-30T13:35:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840ba930 con 0x7fb38c150830 2023-10-30T13:35:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fb392d0f640 1 -- 172.21.15.31:0/2107619355 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38c159a50 con 0x7fb38c150830 2023-10-30T13:35:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1ba9d2f640 1 --2- 172.21.15.31:0/3456610085 >> v2:172.21.15.31:3300/0 conn(0x7f1ba406bb40 0x7f1ba4157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 -- 172.21.15.31:0/3456610085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 msgr2=0x7f1ba4157f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:35:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 --2- 172.21.15.31:0/3456610085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 0x7f1ba4157f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 -- 172.21.15.31:0/3456610085 >> v2:172.21.15.31:3300/0 conn(0x7f1ba406bb40 msgr2=0x7f1ba4157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 --2- 172.21.15.31:0/3456610085 >> v2:172.21.15.31:3300/0 conn(0x7f1ba406bb40 0x7f1ba4157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f1baa530640 1 -- 172.21.15.31:0/3456610085 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba4070f10 con 0x7f1ba406b760 2023-10-30T13:35:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fb392d0f640 1 -- 172.21.15.31:0/2107619355 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38c1f4060 con 0x7fb38c150830 2023-10-30T13:35:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8144bc5640 1 Processor -- start 2023-10-30T13:35:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8144bc5640 1 -- start start 2023-10-30T13:35:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8144bc5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f813804bb90 0x7f81380c3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fbdeffff640 1 Processor -- start 2023-10-30T13:35:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f3385594640 1 -- 172.21.15.31:0/2144731570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3380151840 msgr2=0x7f33800c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f3385594640 1 --2- 172.21.15.31:0/2144731570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3380151840 0x7f33800c49a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f33700097c0 tx=0x7f337001ec70 comp rx=0 tx=0).stop 2023-10-30T13:35:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fbdeffff640 1 -- start start 2023-10-30T13:35:16.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8144bc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81380dd4e0 0x7f81380dedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f813e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f813804bb90 0x7f81380c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f813e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f813804bb90 0x7f81380c3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60636/0 (socket says 172.21.15.31:60636) 2023-10-30T13:35:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb38c156c90 con 0x7fb38c150830 2023-10-30T13:35:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f813e59c640 1 -- 172.21.15.31:0/2103693346 learned_addr learned my addr 172.21.15.31:0/2103693346 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8144bc5640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380e38d0 0x7f8138049cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f3385594640 1 -- 172.21.15.31:0/2144731570 shutdown_connections 2023-10-30T13:35:16.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f3385594640 1 --2- 172.21.15.31:0/2144731570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3380151840 0x7f33800c49a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f3385594640 1 -- 172.21.15.31:0/2144731570 >> 172.21.15.31:0/2144731570 conn(0x7f33800cac70 msgr2=0x7f33800cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81380de110 con 0x7f81380dd4e0 2023-10-30T13:35:16.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== osd.1 v2:172.21.15.31:6802/1190064951 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f874c071260 con 0x7f874c06e790 2023-10-30T13:35:16.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81380df300 con 0x7f813804bb90 2023-10-30T13:35:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.228+0000 7f813ed9d640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380e38d0 0x7f8138049cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81380de280 con 0x7f81380e38d0 2023-10-30T13:35:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f813dd9b640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81380dd4e0 0x7f81380dedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf006da70 0x7fbdf015cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdf006fc50 0x7fbdf01f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1baa530640 1 --2- 172.21.15.31:0/3456610085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba415e650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40cb840 tx=0x7f1ba001c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76c5cbc98c76c6d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f3385594640 1 -- 172.21.15.31:0/2144731570 shutdown_connections 2023-10-30T13:35:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf006da70 0x7fbdf015cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeffff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf01fa2f0 con 0x7fbdf006fc50 2023-10-30T13:35:16.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeffff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf01f9400 con 0x7fbdf006da70 2023-10-30T13:35:16.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeffff640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf01f96e0 con 0x7fbdf01615c0 2023-10-30T13:35:16.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf006da70 0x7fbdf015cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60642/0 (socket says 172.21.15.31:60642) 2023-10-30T13:35:16.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdeeffd640 1 -- 172.21.15.31:0/2563322364 learned_addr learned my addr 172.21.15.31:0/2563322364 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f813ed9d640 1 -- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f813804bb90 msgr2=0x7f81380c3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f813ed9d640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f813804bb90 0x7f81380c3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f813ed9d640 1 -- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81380dd4e0 msgr2=0x7f81380dedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdef7fe640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f813ed9d640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81380dd4e0 0x7f81380dedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f813ed9d640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380ddb50 con 0x7f81380e38d0 2023-10-30T13:35:16.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3840baa90 con 0x7fb38c150830 2023-10-30T13:35:16.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1ba952e640 1 -- 172.21.15.31:0/3456610085 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba00aa610 con 0x7f1ba406b760 2023-10-30T13:35:16.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f3385594640 1 -- 172.21.15.31:0/2144731570 wait complete. 2023-10-30T13:35:16.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fc70a7fc640 1 --2- 172.21.15.31:0/1618054631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141539d0 0x7fc7140bc5b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0097c0 tx=0x7fc6fc01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a54340aeaf8cb4a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1ba952e640 1 -- 172.21.15.31:0/3456610085 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba00a6040 con 0x7f1ba406b760 2023-10-30T13:35:16.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1ba952e640 1 -- 172.21.15.31:0/3456610085 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba00aa990 con 0x7f1ba406b760 2023-10-30T13:35:16.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdef7fe640 1 -- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf006da70 msgr2=0x7fbdf015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdef7fe640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf006da70 0x7fbdf015cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdef7fe640 1 -- 172.21.15.31:0/2563322364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdf006fc50 msgr2=0x7fbdf01f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdef7fe640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdf006fc50 0x7fbdf01f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdef7fe640 1 -- 172.21.15.31:0/2563322364 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8009a70 con 0x7fbdf01615c0 2023-10-30T13:35:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f813ed9d640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380e38d0 0x7f8138049cc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f813000e820 tx=0x7f813000ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fc7127fc640 1 -- 172.21.15.31:0/1618054631 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0a8970 con 0x7fc7141539d0 2023-10-30T13:35:16.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fc7127fc640 1 -- 172.21.15.31:0/1618054631 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc0a8ad0 con 0x7fc7141539d0 2023-10-30T13:35:16.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fc7127fc640 1 -- 172.21.15.31:0/1618054631 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0a8970 con 0x7fc7141539d0 2023-10-30T13:35:16.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813000c730 con 0x7f81380e38d0 2023-10-30T13:35:16.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813000c890 con 0x7f81380e38d0 2023-10-30T13:35:16.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1bac794640 1 -- 172.21.15.31:0/3456610085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 msgr2=0x7f1ba415e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1bac794640 1 --2- 172.21.15.31:0/3456610085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba415e650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40cb840 tx=0x7f1ba001c4f0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813000cc40 con 0x7f81380e38d0 2023-10-30T13:35:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fb3897fa640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3700665a0 0x7fb370068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb384122e30 con 0x7fb38c150830 2023-10-30T13:35:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fb390aab640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3700665a0 0x7fb370068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fb3897fa640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7fb37006e7c0 0x7fb370070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 --> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb370071290 con 0x7fb37006e7c0 2023-10-30T13:35:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81380dde30 con 0x7f81380e38d0 2023-10-30T13:35:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81380d1010 con 0x7f81380e38d0 2023-10-30T13:35:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1bac794640 1 -- 172.21.15.31:0/3456610085 shutdown_connections 2023-10-30T13:35:16.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3841231e0 con 0x7fb38c150830 2023-10-30T13:35:16.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7fbdef7fe640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01f66d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbde80ac000 tx=0x7fbde80196c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1bac794640 1 --2- 172.21.15.31:0/3456610085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 0x7f1ba4157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1bac794640 1 --2- 172.21.15.31:0/3456610085 >> v2:172.21.15.31:3300/0 conn(0x7f1ba406bb40 0x7f1ba4157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.232+0000 7f1bac794640 1 --2- 172.21.15.31:0/3456610085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba415e650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f1bac794640 1 -- 172.21.15.31:0/3456610085 >> 172.21.15.31:0/3456610085 conn(0x7f1ba40cac70 msgr2=0x7f1ba40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fb38bfff640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7fb37006e7c0 0x7fb370070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fb390aab640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3700665a0 0x7fb370068a60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb380007900 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80bf070 con 0x7fbdf01615c0 2023-10-30T13:35:16.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde80aeb60 con 0x7fbdf01615c0 2023-10-30T13:35:16.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fb38bfff640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7fb37006e7c0 0x7fb370070be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf01f99c0 con 0x7fbdf01615c0 2023-10-30T13:35:16.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf01f9f80 con 0x7fbdf01615c0 2023-10-30T13:35:16.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 Processor -- start 2023-10-30T13:35:16.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fc713fff640 1 -- 172.21.15.31:0/1618054631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141539d0 msgr2=0x7fc7140bc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fc713fff640 1 --2- 172.21.15.31:0/1618054631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141539d0 0x7fc7140bc5b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0097c0 tx=0x7fc6fc01ec90 comp rx=0 tx=0).stop 2023-10-30T13:35:16.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8138045d90 con 0x7f81380e38d0 2023-10-30T13:35:16.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 -- start start 2023-10-30T13:35:16.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fc713fff640 1 -- 172.21.15.31:0/1618054631 shutdown_connections 2023-10-30T13:35:16.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fc713fff640 1 --2- 172.21.15.31:0/1618054631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141539d0 0x7fc7140bc5b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fc713fff640 1 -- 172.21.15.31:0/1618054631 >> 172.21.15.31:0/1618054631 conn(0x7fc7140caae0 msgr2=0x7fc7140caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f1bac794640 1 -- 172.21.15.31:0/3456610085 shutdown_connections 2023-10-30T13:35:16.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f1bac794640 1 -- 172.21.15.31:0/3456610085 wait complete. 2023-10-30T13:35:16.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f1bac794640 1 Processor -- start 2023-10-30T13:35:16.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== osd.2 v2:172.21.15.31:6824/3739146666 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb370071290 con 0x7fb37006e7c0 2023-10-30T13:35:16.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80b7410 con 0x7fbdf01615c0 2023-10-30T13:35:16.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbdf01610a0 con 0x7fbdf01615c0 2023-10-30T13:35:16.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33800cb8b0 0x7f3380152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3380151840 0x7f3380146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3380151c20 0x7f33801554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33801ec6a0 con 0x7f33800cb8b0 2023-10-30T13:35:16.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33801ec920 con 0x7f3380151840 2023-10-30T13:35:16.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f3385594640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33801ecc00 con 0x7f3380151c20 2023-10-30T13:35:16.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fc713fff640 1 -- 172.21.15.31:0/1618054631 shutdown_connections 2023-10-30T13:35:16.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7f1bac794640 1 -- start start 2023-10-30T13:35:16.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.236+0000 7fc713fff640 1 -- 172.21.15.31:0/1618054631 wait complete. 2023-10-30T13:35:16.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33800cb8b0 0x7f3380152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337f7fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3380151c20 0x7f33801554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3380151840 0x7f3380146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33800cb8b0 0x7f3380152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35984/0 (socket says 172.21.15.31:35984) 2023-10-30T13:35:16.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 -- 172.21.15.31:0/2991949279 learned_addr learned my addr 172.21.15.31:0/2991949279 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1bac794640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba414dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1bac794640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba406bb40 0x7f1ba41f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba414dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1bac794640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 0x7f1ba41f8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 -- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3380151c20 msgr2=0x7f33801554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3380151c20 0x7f33801554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 -- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3380151840 msgr2=0x7f3380146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3380151840 0x7f3380146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 -- 172.21.15.31:0/2991949279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370009420 con 0x7f33800cb8b0 2023-10-30T13:35:16.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba414dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60660/0 (socket says 172.21.15.31:60660) 2023-10-30T13:35:16.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 -- 172.21.15.31:0/4105478338 learned_addr learned my addr 172.21.15.31:0/4105478338 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baad31640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 0x7f1ba41f8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4160000 con 0x7f1ba406bb40 2023-10-30T13:35:16.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4157340 con 0x7f1ba406b760 2023-10-30T13:35:16.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba414e010 con 0x7f1ba406c420 2023-10-30T13:35:16.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f337effd640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33800cb8b0 0x7f3380152700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f337001b880 tx=0x7f337001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7fc713fff640 1 Processor -- start 2023-10-30T13:35:16.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8130013470 con 0x7f81380e38d0 2023-10-30T13:35:16.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f812effd640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81200664a0 0x7f8120068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 -- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 msgr2=0x7f1ba41f8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 0x7f1ba41f8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 -- 172.21.15.31:0/4105478338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba406bb40 msgr2=0x7f1ba41f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba406bb40 0x7f1ba41f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7fc713fff640 1 -- start start 2023-10-30T13:35:16.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00094d0 con 0x7f1ba406b760 2023-10-30T13:35:16.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8130054f90 con 0x7f81380e38d0 2023-10-30T13:35:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baad31640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1ba406c420 0x7f1ba41f8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f813e59c640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81200664a0 0x7f8120068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbde80b4050 con 0x7fbdf01615c0 2023-10-30T13:35:16.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7fbdecff9640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdd00664d0 0x7fbdd0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f1baa530640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba414dad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ba001aba0 tx=0x7f1ba001ad30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.240+0000 7f812effd640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7f812006f250 0x7f81200716f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8120071dc0 con 0x7f812006f250 2023-10-30T13:35:16.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8130054d60 con 0x7f81380e38d0 2023-10-30T13:35:16.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba00bd5f0 con 0x7f1ba406b760 2023-10-30T13:35:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33700b1cb0 con 0x7f33800cb8b0 2023-10-30T13:35:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33700b1e30 con 0x7f33800cb8b0 2023-10-30T13:35:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33700b9650 con 0x7f33800cb8b0 2023-10-30T13:35:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc713fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7140bcaf0 0x7fc714150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc713fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141535f0 0x7fc7141e3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba00bd770 con 0x7f1ba406b760 2023-10-30T13:35:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f813dd9b640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7f812006f250 0x7f81200716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f813e59c640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81200664a0 0x7f8120068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8140069360 tx=0x7f8140121000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba41f9c80 con 0x7f1ba406b760 2023-10-30T13:35:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba41604f0 con 0x7f1ba406b760 2023-10-30T13:35:16.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba00bdb20 con 0x7f1ba406b760 2023-10-30T13:35:16.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbde811f000 con 0x7fbdf01615c0 2023-10-30T13:35:16.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc713fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7141539d0 0x7fc7141e4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc713fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141e0ab0 con 0x7fc7140bcaf0 2023-10-30T13:35:16.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc713fff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71414ea80 con 0x7fc7141535f0 2023-10-30T13:35:16.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc713fff640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71414ed60 con 0x7fc7141539d0 2023-10-30T13:35:16.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc712ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7140bcaf0 0x7fc714150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc70a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141535f0 0x7fc7141e3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc70a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141535f0 0x7fc7141e3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60668/0 (socket says 172.21.15.31:60668) 2023-10-30T13:35:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc70a7fc640 1 -- 172.21.15.31:0/2116286507 learned_addr learned my addr 172.21.15.31:0/2116286507 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc7137fe640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7141539d0 0x7fc7141e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fbdeeffd640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdd00664d0 0x7fbdd0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc7137fe640 1 -- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141535f0 msgr2=0x7fc7141e3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc7137fe640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7141535f0 0x7fc7141e3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc7137fe640 1 -- 172.21.15.31:0/2116286507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7140bcaf0 msgr2=0x7fc714150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1b8c000fd0 con 0x7f1ba406b760 2023-10-30T13:35:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fbdecff9640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7fbdd006e670 0x7fbdd0070a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 --> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdd0071140 con 0x7fbdd006e670 2023-10-30T13:35:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbde811f3b0 con 0x7fbdf01615c0 2023-10-30T13:35:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fbde67fc640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7fbdd006e670 0x7fbdd0070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fbdeeffd640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdd00664d0 0x7fbdd0068990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8006fd0 tx=0x7fbdd8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f813dd9b640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7f812006f250 0x7f81200716f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33801ecee0 con 0x7f33800cb8b0 2023-10-30T13:35:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33801e0650 con 0x7f33800cb8b0 2023-10-30T13:35:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc7137fe640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7140bcaf0 0x7fc714150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc7137fe640 1 -- 172.21.15.31:0/2116286507 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc009420 con 0x7fc7141539d0 2023-10-30T13:35:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.244+0000 7fc712ffd640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7140bcaf0 0x7fc714150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33801ed390 con 0x7f33800cb8b0 2023-10-30T13:35:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fbde67fc640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7fbdd006e670 0x7fbdd0070a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fc7137fe640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7141539d0 0x7fc7141e4350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00c3c0 tx=0x7fc70c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ba00c8470 con 0x7f1ba406b760 2023-10-30T13:35:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== osd.3 v2:172.21.15.31:6816/2725375815 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8120071dc0 con 0x7f812006f250 2023-10-30T13:35:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c005cf0 con 0x7fc7141539d0 2023-10-30T13:35:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c005e50 con 0x7fc7141539d0 2023-10-30T13:35:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fc713fff640 1 -- 172.21.15.31:0/2116286507 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7141e1140 con 0x7fc7141539d0 2023-10-30T13:35:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fc713fff640 1 -- 172.21.15.31:0/2116286507 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc714157230 con 0x7fc7141539d0 2023-10-30T13:35:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f1b9affd640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b880664f0 0x7f1b880689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6f8000fd0 con 0x7fc7141539d0 2023-10-30T13:35:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ba011ed90 con 0x7f1ba406b760 2023-10-30T13:35:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f1b9affd640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f1b8806e710 0x7f1b88070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b880711e0 con 0x7f1b8806e710 2023-10-30T13:35:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ba011f100 con 0x7f1ba406b760 2023-10-30T13:35:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0072d0 con 0x7fc7141539d0 2023-10-30T13:35:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f1baad31640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f1b8806e710 0x7f1b88070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.248+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33700b7040 con 0x7f33800cb8b0 2023-10-30T13:35:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== osd.4 v2:172.21.15.33:6800/465067075 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fbdd0071140 con 0x7fbdd006e670 2023-10-30T13:35:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f1ba9d2f640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b880664f0 0x7f1b880689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f1baad31640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f1b8806e710 0x7f1b88070b30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f33677fe640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f336c0664f0 0x7f336c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3370122160 con 0x7f33800cb8b0 2023-10-30T13:35:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f33677fe640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f336c06e710 0x7f336c070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 --> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f336c0711e0 con 0x7f336c06e710 2023-10-30T13:35:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3370122600 con 0x7f33800cb8b0 2023-10-30T13:35:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== osd.7 v2:172.21.15.33:6824/3463894958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1b880711e0 con 0x7f1b8806e710 2023-10-30T13:35:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc70c007430 con 0x7fc7141539d0 2023-10-30T13:35:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f337e7fc640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f336c0664f0 0x7f336c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f1ba9d2f640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b880664f0 0x7f1b880689b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ba41f35e0 tx=0x7f1b94008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.252+0000 7f337e7fc640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f336c0664f0 0x7f336c0689b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33801e2d80 tx=0x7f3378009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7f337f7fe640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f336c06e710 0x7f336c070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc70bfff640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6f40664a0 0x7fc6f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7f337f7fe640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f336c06e710 0x7f336c070b30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc70c083c40 con 0x7fc7141539d0 2023-10-30T13:35:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc712ffd640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6f40664a0 0x7fc6f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc70bfff640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fc6f406e6c0 0x7fc6f4070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 --> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f4071190 con 0x7fc6f406e6c0 2023-10-30T13:35:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc70c083ff0 con 0x7fc7141539d0 2023-10-30T13:35:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc712ffd640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6f40664a0 0x7fc6f4068960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc704007fd0 tx=0x7fc704009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc70a7fc640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fc6f406e6c0 0x7fc6f4070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== osd.5 v2:172.21.15.33:6816/3902046602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f336c0711e0 con 0x7f336c06e710 2023-10-30T13:35:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.256+0000 7fc70a7fc640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fc6f406e6c0 0x7fc6f4070ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.260+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 --> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25640c5b10 con 0x7f254806e790 2023-10-30T13:35:16.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.260+0000 7f2559ffb640 1 -- 172.21.15.31:0/2151650727 <== osd.0 v2:172.21.15.31:6800/1469638853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f25540058a0 con 0x7f254806e790 2023-10-30T13:35:16.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== osd.6 v2:172.21.15.33:6808/931136877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc6f4071190 con 0x7fc6f406e6c0 2023-10-30T13:35:16.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 --> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb38c1569b0 con 0x7fb37006e7c0 2023-10-30T13:35:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f254806e790 msgr2=0x7f2548070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f254806e790 0x7f2548070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2548066570 msgr2=0x7f2548068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2548066570 0x7f2548068a30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f25600194e0 tx=0x7f25600a7070 comp rx=0 tx=0).stop 2023-10-30T13:35:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f256406e3f0 msgr2=0x7f2564153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 --2- 172.21.15.31:0/2151650727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f256406e3f0 0x7f2564153e20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f256406cd20 tx=0x7f255c00cb30 comp rx=0 tx=0).stop 2023-10-30T13:35:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f2569873640 1 -- 172.21.15.31:0/2151650727 reap_dead start 2023-10-30T13:35:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 shutdown_connections 2023-10-30T13:35:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.264+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 >> 172.21.15.31:0/2151650727 conn(0x7f25640c7560 msgr2=0x7f25640c0710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 shutdown_connections 2023-10-30T13:35:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7f256a074640 1 -- 172.21.15.31:0/2151650727 wait complete. 2023-10-30T13:35:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 --> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8738002cc0 con 0x7f874c06e790 2023-10-30T13:35:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb3897fa640 1 -- 172.21.15.31:0/2107619355 <== osd.2 v2:172.21.15.31:6824/3739146666 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb3780051e0 con 0x7fb37006e7c0 2023-10-30T13:35:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7fb37006e7c0 msgr2=0x7fb370070be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7fb37006e7c0 0x7fb370070be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3700665a0 msgr2=0x7fb370068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3700665a0 0x7fb370068a60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb380007900 comp rx=0 tx=0).stop 2023-10-30T13:35:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb38c150830 msgr2=0x7fb38c157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 --2- 172.21.15.31:0/2107619355 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb38c150830 0x7fb38c157fe0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb3840af000 tx=0x7fb38401bea0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 --> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbdf015e5a0 con 0x7fbdd006e670 2023-10-30T13:35:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 --> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8138045ab0 con 0x7f812006f250 2023-10-30T13:35:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb3912ac640 1 -- 172.21.15.31:0/2107619355 reap_dead start 2023-10-30T13:35:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7f8747fff640 1 -- 172.21.15.31:0/134582758 <== osd.1 v2:172.21.15.31:6802/1190064951 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8738002cc0 con 0x7f874c06e790 2023-10-30T13:35:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 shutdown_connections 2023-10-30T13:35:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 >> 172.21.15.31:0/2107619355 conn(0x7fb38c0cac70 msgr2=0x7fb38c06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 shutdown_connections 2023-10-30T13:35:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7fb376ffd640 1 -- 172.21.15.31:0/2107619355 wait complete. 2023-10-30T13:35:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f874c06e790 msgr2=0x7f874c070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.268+0000 7f8745ffb640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f874c06e790 0x7f874c070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f874c066570 msgr2=0x7f874c068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f874c066570 0x7f874c068a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87540ae000 tx=0x7f87540046d0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87600cb780 msgr2=0x7f87601eebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 --2- 172.21.15.31:0/134582758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87600cb780 0x7f87601eebb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8758007fa0 tx=0x7f8758011660 comp rx=0 tx=0).stop 2023-10-30T13:35:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f875f7fe640 1 -- 172.21.15.31:0/134582758 reap_dead start 2023-10-30T13:35:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 shutdown_connections 2023-10-30T13:35:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 >> 172.21.15.31:0/134582758 conn(0x7f876006d810 msgr2=0x7f87600c79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 shutdown_connections 2023-10-30T13:35:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8745ffb640 1 -- 172.21.15.31:0/134582758 wait complete. 2023-10-30T13:35:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f812effd640 1 -- 172.21.15.31:0/2103693346 <== osd.3 v2:172.21.15.31:6816/2725375815 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8134001360 con 0x7f812006f250 2023-10-30T13:35:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7fbdecff9640 1 -- 172.21.15.31:0/2563322364 <== osd.4 v2:172.21.15.33:6800/465067075 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbde00011d0 con 0x7fbdd006e670 2023-10-30T13:35:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7f812006f250 msgr2=0x7f81200716f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7f812006f250 0x7f81200716f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81200664a0 msgr2=0x7f8120068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81200664a0 0x7f8120068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8140069360 tx=0x7f8140121000 comp rx=0 tx=0).stop 2023-10-30T13:35:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380e38d0 msgr2=0x7f8138049cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 --2- 172.21.15.31:0/2103693346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81380e38d0 0x7f8138049cc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f813000e820 tx=0x7f813000ece0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f813ed9d640 1 -- 172.21.15.31:0/2103693346 reap_dead start 2023-10-30T13:35:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7fbdd006e670 msgr2=0x7fbdd0070a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7fbdeffff640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7fbdd006e670 0x7fbdd0070a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdd00664d0 msgr2=0x7fbdd0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7fbdeffff640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdd00664d0 0x7fbdd0068990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8006fd0 tx=0x7fbdd8008040 comp rx=0 tx=0).stop 2023-10-30T13:35:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 msgr2=0x7fbdf01f66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7fbdeffff640 1 --2- 172.21.15.31:0/2563322364 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdf01615c0 0x7fbdf01f66d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbde80ac000 tx=0x7fbde80196c0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 shutdown_connections 2023-10-30T13:35:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.272+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 >> 172.21.15.31:0/2103693346 conn(0x7f81380502b0 msgr2=0x7f8138006b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.276+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 shutdown_connections 2023-10-30T13:35:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.276+0000 7fbdef7fe640 1 -- 172.21.15.31:0/2563322364 reap_dead start 2023-10-30T13:35:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.276+0000 7f8144bc5640 1 -- 172.21.15.31:0/2103693346 wait complete. 2023-10-30T13:35:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.276+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 shutdown_connections 2023-10-30T13:35:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.276+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 >> 172.21.15.31:0/2563322364 conn(0x7fbdf00cab50 msgr2=0x7fbdf00c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.280+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 shutdown_connections 2023-10-30T13:35:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.280+0000 7fbdeffff640 1 -- 172.21.15.31:0/2563322364 wait complete. 2023-10-30T13:35:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.280+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 --> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1b8c002cc0 con 0x7f1b8806e710 2023-10-30T13:35:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1b9affd640 1 -- 172.21.15.31:0/4105478338 <== osd.7 v2:172.21.15.33:6824/3463894958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1b8c002cc0 con 0x7f1b8806e710 2023-10-30T13:35:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f1b8806e710 msgr2=0x7f1b88070b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f1b8806e710 0x7f1b88070b30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b880664f0 msgr2=0x7f1b880689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b880664f0 0x7f1b880689b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ba41f35e0 tx=0x7f1b94008040 comp rx=0 tx=0).stop 2023-10-30T13:35:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 msgr2=0x7f1ba414dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 --2- 172.21.15.31:0/4105478338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba406b760 0x7f1ba414dad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ba001aba0 tx=0x7f1ba001ad30 comp rx=0 tx=0).stop 2023-10-30T13:35:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1baad31640 1 -- 172.21.15.31:0/4105478338 reap_dead start 2023-10-30T13:35:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 shutdown_connections 2023-10-30T13:35:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 >> 172.21.15.31:0/4105478338 conn(0x7f1ba40cac70 msgr2=0x7f1ba4153040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 shutdown_connections 2023-10-30T13:35:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.284+0000 7f1bac794640 1 -- 172.21.15.31:0/4105478338 wait complete. 2023-10-30T13:35:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.288+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 --> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33801583b0 con 0x7f336c06e710 2023-10-30T13:35:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.288+0000 7f33677fe640 1 -- 172.21.15.31:0/2991949279 <== osd.5 v2:172.21.15.33:6816/3902046602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3374001470 con 0x7f336c06e710 2023-10-30T13:35:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.288+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f336c06e710 msgr2=0x7f336c070b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.288+0000 7f3385594640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f336c06e710 0x7f336c070b30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 --> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc6f8002cc0 con 0x7fc6f406e6c0 2023-10-30T13:35:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc70bfff640 1 -- 172.21.15.31:0/2116286507 <== osd.6 v2:172.21.15.33:6808/931136877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc6f8002cc0 con 0x7fc6f406e6c0 2023-10-30T13:35:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fc6f406e6c0 msgr2=0x7fc6f4070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fc6f406e6c0 0x7fc6f4070ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6f40664a0 msgr2=0x7fc6f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6f40664a0 0x7fc6f4068960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc704007fd0 tx=0x7fc704009040 comp rx=0 tx=0).stop 2023-10-30T13:35:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7141539d0 msgr2=0x7fc7141e4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 --2- 172.21.15.31:0/2116286507 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7141539d0 0x7fc7141e4350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00c3c0 tx=0x7fc70c00c880 comp rx=0 tx=0).stop 2023-10-30T13:35:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7137fe640 1 -- 172.21.15.31:0/2116286507 reap_dead start 2023-10-30T13:35:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 shutdown_connections 2023-10-30T13:35:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.292+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 >> 172.21.15.31:0/2116286507 conn(0x7fc7140caae0 msgr2=0x7fc714157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 shutdown_connections 2023-10-30T13:35:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f336c0664f0 msgr2=0x7f336c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7fc7097fa640 1 -- 172.21.15.31:0/2116286507 wait complete. 2023-10-30T13:35:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7f3385594640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f336c0664f0 0x7f336c0689b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33801e2d80 tx=0x7f3378009040 comp rx=0 tx=0).stop 2023-10-30T13:35:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33800cb8b0 msgr2=0x7f3380152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7f3385594640 1 --2- 172.21.15.31:0/2991949279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33800cb8b0 0x7f3380152700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f337001b880 tx=0x7f337001ba10 comp rx=0 tx=0).stop 2023-10-30T13:35:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7f337f7fe640 1 -- 172.21.15.31:0/2991949279 reap_dead start 2023-10-30T13:35:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 shutdown_connections 2023-10-30T13:35:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.296+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 >> 172.21.15.31:0/2991949279 conn(0x7f33800cac70 msgr2=0x7f33800c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.300+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 shutdown_connections 2023-10-30T13:35:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.300+0000 7f3385594640 1 -- 172.21.15.31:0/2991949279 wait complete. 2023-10-30T13:35:16.382 INFO:teuthology.orchestra.run.smithi031.stdout:60129542153 2023-10-30T13:35:16.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T13:35:16.383 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2023-10-30T13:35:16.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T13:35:16.384 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2023-10-30T13:35:16.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T13:35:16.385 INFO:teuthology.orchestra.run.smithi031.stdout:60129542151 2023-10-30T13:35:16.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T13:35:16.387 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2023-10-30T13:35:16.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T13:35:16.388 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2023-10-30T13:35:16.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T13:35:16.389 INFO:teuthology.orchestra.run.smithi031.stdout:60129542151 2023-10-30T13:35:16.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T13:35:16.390 INFO:teuthology.orchestra.run.smithi031.stdout:60129542153 2023-10-30T13:35:16.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T13:35:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.660+0000 7f95a7ff5640 1 Processor -- start 2023-10-30T13:35:16.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a7ff5640 1 -- start start 2023-10-30T13:35:16.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a7ff5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f95980d8f90 0x7f95980df7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a7ff5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f959804c5f0 0x7f9598005dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a7ff5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95980dfd20 con 0x7f9598046d60 2023-10-30T13:35:16.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a7ff5640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f959804b480 con 0x7f95980d8f90 2023-10-30T13:35:16.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a7ff5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9598045fa0 con 0x7f959804c5f0 2023-10-30T13:35:16.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a6592640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f9598046d60 0x7f9598045320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51860/0 (socket says 172.21.15.31:51860) 2023-10-30T13:35:16.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a6592640 1 -- 172.21.15.31:0/1761497933 learned_addr learned my addr 172.21.15.31:0/1761497933 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a5590640 1 --2- 172.21.15.31:0/1761497933 >> v2:172.21.15.33:3300/0 conn(0x7f959804c5f0 0x7f9598005dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a5590640 1 -- 172.21.15.31:0/1761497933 >> v2:172.21.15.31:3300/0 conn(0x7f95980d8f90 msgr2=0x7f95980df7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.664+0000 7f95a5590640 1 --2- 172.21.15.31:0/1761497933 >> v2:172.21.15.31:3300/0 conn(0x7f95980d8f90 0x7f95980df7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.668+0000 7f95a5590640 1 -- 172.21.15.31:0/1761497933 >> v1:172.21.15.31:6789/0 conn(0x7f9598046d60 legacy=0x7f9598045320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.668+0000 7f95a5590640 1 -- 172.21.15.31:0/1761497933 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598046280 con 0x7f959804c5f0 2023-10-30T13:35:16.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.672+0000 7f95a5590640 1 --2- 172.21.15.31:0/1761497933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598005dc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9598033060 tx=0x7f9590024c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ea89abf2408cd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.672+0000 7f95a4d8f640 1 -- 172.21.15.31:0/1761497933 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900adc50 con 0x7f959804c5f0 2023-10-30T13:35:16.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.672+0000 7f95a4d8f640 1 -- 172.21.15.31:0/1761497933 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95900addb0 con 0x7f959804c5f0 2023-10-30T13:35:16.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.672+0000 7f95a4d8f640 1 -- 172.21.15.31:0/1761497933 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900b5660 con 0x7f959804c5f0 2023-10-30T13:35:16.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f8c734c9640 1 Processor -- start 2023-10-30T13:35:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.672+0000 7f95a7ff5640 1 -- 172.21.15.31:0/1761497933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 msgr2=0x7f9598005dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.672+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/1761497933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598005dc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9598033060 tx=0x7f9590024c90 comp rx=0 tx=0).stop 2023-10-30T13:35:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f95a7ff5640 1 -- 172.21.15.31:0/1761497933 shutdown_connections 2023-10-30T13:35:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f8c734c9640 1 -- start start 2023-10-30T13:35:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/1761497933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598005dc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/1761497933 >> v2:172.21.15.31:3300/0 conn(0x7f95980d8f90 0x7f95980df7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f95a7ff5640 1 -- 172.21.15.31:0/1761497933 >> 172.21.15.31:0/1761497933 conn(0x7f9598050170 msgr2=0x7f9598050580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f3a3bfff640 1 Processor -- start 2023-10-30T13:35:16.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f95a7ff5640 1 -- 172.21.15.31:0/1761497933 shutdown_connections 2023-10-30T13:35:16.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f3a3bfff640 1 -- start start 2023-10-30T13:35:16.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f95a7ff5640 1 -- 172.21.15.31:0/1761497933 wait complete. 2023-10-30T13:35:16.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.676+0000 7f62a9d65640 1 Processor -- start 2023-10-30T13:35:16.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c734c9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8c6c153b10 0x7f8c6c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c734c9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8c6c0ca8e0 0x7f8c6c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c734c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 0x7f8c6c159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c734c9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c159900 con 0x7f8c6c153b10 2023-10-30T13:35:16.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c734c9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c15d070 con 0x7f8c6c0ca8e0 2023-10-30T13:35:16.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c734c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c15d350 con 0x7f8c6c06c480 2023-10-30T13:35:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c71265640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8c6c153b10 0x7f8c6c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c71265640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8c6c153b10 0x7f8c6c0c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35998/0 (socket says 172.21.15.31:35998) 2023-10-30T13:35:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c71265640 1 -- 172.21.15.31:0/2691341852 learned_addr learned my addr 172.21.15.31:0/2691341852 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c70a64640 1 --2- 172.21.15.31:0/2691341852 >> v2:172.21.15.33:3300/0 conn(0x7f8c6c0ca8e0 0x7f8c6c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a3bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3a340dd350 0x7f3a34046670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a3bfff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3a3404c760 0x7f3a3404acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a3bfff640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a340461e0 con 0x7f3a34005f90 2023-10-30T13:35:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a3bfff640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a34046bb0 con 0x7f3a340dd350 2023-10-30T13:35:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a3bfff640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a340e3760 con 0x7f3a3404c760 2023-10-30T13:35:16.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c71a66640 1 --2- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 0x7f8c6c159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f95a7ff5640 1 Processor -- start 2023-10-30T13:35:16.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a417d3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3a3404c760 0x7f3a3404acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a417d3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3a3404c760 0x7f3a3404acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60692/0 (socket says 172.21.15.31:60692) 2023-10-30T13:35:16.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a417d3640 1 -- 172.21.15.31:0/4214070911 learned_addr learned my addr 172.21.15.31:0/4214070911 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f95a7ff5640 1 -- start start 2023-10-30T13:35:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f62a9d65640 1 -- start start 2023-10-30T13:35:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f95a7ff5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598046d60 0x7f95980e1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a9d65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a4153b10 0x7f62a40c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a9d65640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f62a40ca8e0 0x7f62a406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a9d65640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415d070 con 0x7f62a406c480 2023-10-30T13:35:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a9d65640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a4159900 con 0x7f62a40ca8e0 2023-10-30T13:35:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a9d65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a415d350 con 0x7f62a4153b10 2023-10-30T13:35:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598046d60 0x7f95980e1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a3fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f62a406c480 0x7f62a4159d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45752/0 (socket says 172.21.15.31:45752) 2023-10-30T13:35:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598046d60 0x7f95980e1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36016/0 (socket says 172.21.15.31:36016) 2023-10-30T13:35:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 -- 172.21.15.31:0/4029259629 learned_addr learned my addr 172.21.15.31:0/4029259629 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a3fff640 1 -- 172.21.15.31:0/2703219694 learned_addr learned my addr 172.21.15.31:0/2703219694 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598176350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a37fe640 1 --2- 172.21.15.31:0/2703219694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a4153b10 0x7f62a40c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95980d8f90 0x7f9598047760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a417d3640 1 -- 172.21.15.31:0/4214070911 >> v2:172.21.15.31:3300/0 conn(0x7f3a340dd350 msgr2=0x7f3a34046670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a417d3640 1 --2- 172.21.15.31:0/4214070911 >> v2:172.21.15.31:3300/0 conn(0x7f3a340dd350 0x7f3a34046670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a417d3640 1 -- 172.21.15.31:0/4214070911 >> v1:172.21.15.31:6789/0 conn(0x7f3a34005f90 legacy=0x7f3a340e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f3a417d3640 1 -- 172.21.15.31:0/4214070911 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a34045c00 con 0x7f3a3404c760 2023-10-30T13:35:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95981735d0 con 0x7f9598046d60 2023-10-30T13:35:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95980e16b0 con 0x7f959804c5f0 2023-10-30T13:35:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9598177230 con 0x7f95980d8f90 2023-10-30T13:35:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c70a64640 1 -- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 msgr2=0x7f8c6c159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c70a64640 1 --2- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 0x7f8c6c159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c70a64640 1 -- 172.21.15.31:0/2691341852 >> v2:172.21.15.31:3300/0 conn(0x7f8c6c153b10 msgr2=0x7f8c6c0c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c70a64640 1 --2- 172.21.15.31:0/2691341852 >> v2:172.21.15.31:3300/0 conn(0x7f8c6c153b10 0x7f8c6c0c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.680+0000 7f8c70a64640 1 -- 172.21.15.31:0/2691341852 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c15d630 con 0x7f8c6c0ca8e0 2023-10-30T13:35:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a37fe640 1 -- 172.21.15.31:0/2703219694 >> v2:172.21.15.31:3300/0 conn(0x7f62a40ca8e0 msgr2=0x7f62a406bf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:35:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a37fe640 1 --2- 172.21.15.31:0/2703219694 >> v2:172.21.15.31:3300/0 conn(0x7f62a40ca8e0 0x7f62a406bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a37fe640 1 -- 172.21.15.31:0/2703219694 >> v1:172.21.15.33:6789/0 conn(0x7f62a406c480 legacy=0x7f62a4159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f62a37fe640 1 -- 172.21.15.31:0/2703219694 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a415d630 con 0x7f62a4153b10 2023-10-30T13:35:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a6592640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95980d8f90 0x7f9598047760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5590640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598176350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f8c70a64640 1 --2- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c0ca8e0 0x7f8c6c06bf40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c0b1dd0 tx=0x7f8c5c01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eb5c6a665350df8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 -- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95980d8f90 msgr2=0x7f9598047760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95980d8f90 0x7f9598047760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 -- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 msgr2=0x7f9598176350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f8c63fff640 1 -- 172.21.15.31:0/2691341852 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0ab600 con 0x7f8c6c0ca8e0 2023-10-30T13:35:16.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598176350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f95a5d91640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590009420 con 0x7f9598046d60 2023-10-30T13:35:16.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95a6592640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95980d8f90 0x7f9598047760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:16.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95a5590640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598176350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:16.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f8c63fff640 1 -- 172.21.15.31:0/2691341852 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c0a7050 con 0x7f8c6c0ca8e0 2023-10-30T13:35:16.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.684+0000 7f8c63fff640 1 -- 172.21.15.31:0/2691341852 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0ab980 con 0x7f8c6c0ca8e0 2023-10-30T13:35:16.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95a5d91640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598046d60 0x7f95980e1170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95a0000b70 tx=0x7f95a00674c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f3a417d3640 1 --2- 172.21.15.31:0/4214070911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a3404c760 0x7f3a3404acc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c074c50 tx=0x7f3a3c0974c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dfa87f6be00cdee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95967fc640 1 -- 172.21.15.31:0/4029259629 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0066d80 con 0x7f9598046d60 2023-10-30T13:35:16.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9598176c70 con 0x7f9598046d60 2023-10-30T13:35:16.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 -- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c0ca8e0 msgr2=0x7f8c6c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c0ca8e0 0x7f8c6c06bf40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c0b1dd0 tx=0x7f8c5c01c460 comp rx=0 tx=0).stop 2023-10-30T13:35:16.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9598172f80 con 0x7f9598046d60 2023-10-30T13:35:16.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95967fc640 1 -- 172.21.15.31:0/4029259629 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a00675f0 con 0x7f9598046d60 2023-10-30T13:35:16.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95967fc640 1 -- 172.21.15.31:0/4029259629 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0072b10 con 0x7f9598046d60 2023-10-30T13:35:16.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 -- 172.21.15.31:0/2691341852 shutdown_connections 2023-10-30T13:35:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 0x7f8c6c159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2691341852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c0ca8e0 0x7f8c6c06bf40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2691341852 >> v2:172.21.15.31:3300/0 conn(0x7f8c6c153b10 0x7f8c6c0c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 -- 172.21.15.31:0/2691341852 >> 172.21.15.31:0/2691341852 conn(0x7f8c6c06a240 msgr2=0x7f8c6c06a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f3a39ffb640 1 -- 172.21.15.31:0/4214070911 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c072040 con 0x7f3a3404c760 2023-10-30T13:35:16.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f3a39ffb640 1 -- 172.21.15.31:0/4214070911 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c122050 con 0x7f3a3404c760 2023-10-30T13:35:16.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f3a39ffb640 1 -- 172.21.15.31:0/4214070911 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c126900 con 0x7f3a3404c760 2023-10-30T13:35:16.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f7a43fff640 1 Processor -- start 2023-10-30T13:35:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f7a43fff640 1 -- start start 2023-10-30T13:35:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f3a3bfff640 1 -- 172.21.15.31:0/4214070911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a3404c760 msgr2=0x7f3a3404acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/4214070911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a3404c760 0x7f3a3404acc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c074c50 tx=0x7f3a3c0974c0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 -- 172.21.15.31:0/2691341852 shutdown_connections 2023-10-30T13:35:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f8c734c9640 1 -- 172.21.15.31:0/2691341852 wait complete. 2023-10-30T13:35:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9588005260 con 0x7f9598046d60 2023-10-30T13:35:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.688+0000 7f7a43fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a3c0dd500 0x7f7a3c0e71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a43fff640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c04a6c0 con 0x7f7a3c04f680 2023-10-30T13:35:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a43fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c0e76e0 con 0x7f7a3c049440 2023-10-30T13:35:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a43fff640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3c050dc0 con 0x7f7a3c0dd500 2023-10-30T13:35:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a42ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a3c0dd500 0x7f7a3c0e71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f3a3bfff640 1 -- 172.21.15.31:0/4214070911 shutdown_connections 2023-10-30T13:35:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/4214070911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a3404c760 0x7f3a3404acc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/4214070911 >> v2:172.21.15.31:3300/0 conn(0x7f3a340dd350 0x7f3a34046670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f3a3bfff640 1 -- 172.21.15.31:0/4214070911 >> 172.21.15.31:0/4214070911 conn(0x7f3a34050340 msgr2=0x7f3a34050750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a42ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a3c0dd500 0x7f7a3c0e71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36018/0 (socket says 172.21.15.31:36018) 2023-10-30T13:35:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a42ffd640 1 -- 172.21.15.31:0/2947570573 learned_addr learned my addr 172.21.15.31:0/2947570573 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f62a37fe640 1 --2- 172.21.15.31:0/2703219694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a4153b10 0x7f62a40c9b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f628c0097c0 tx=0x7f628c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59eb109b47b18f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7fe01d091640 1 Processor -- start 2023-10-30T13:35:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f3a3bfff640 1 -- 172.21.15.31:0/4214070911 shutdown_connections 2023-10-30T13:35:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7fe01d091640 1 -- start start 2023-10-30T13:35:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f3a3bfff640 1 -- 172.21.15.31:0/4214070911 wait complete. 2023-10-30T13:35:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a42ffd640 1 -- 172.21.15.31:0/2947570573 >> v1:172.21.15.33:6789/0 conn(0x7f7a3c049440 legacy=0x7f7a3c049870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a42ffd640 1 -- 172.21.15.31:0/2947570573 >> v1:172.21.15.31:6789/0 conn(0x7f7a3c04f680 legacy=0x7f7a3c04ab50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:35:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f7a42ffd640 1 -- 172.21.15.31:0/2947570573 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c050800 con 0x7f7a3c0dd500 2023-10-30T13:35:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f5813aea640 1 Processor -- start 2023-10-30T13:35:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f5813aea640 1 -- start start 2023-10-30T13:35:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8af9ed6640 1 Processor -- start 2023-10-30T13:35:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8c734c9640 1 Processor -- start 2023-10-30T13:35:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f95967fc640 1 -- 172.21.15.31:0/4029259629 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95a0091020 con 0x7f9598046d60 2023-10-30T13:35:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f95967fc640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95840664a0 0x7f9584068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f95967fc640 1 -- 172.21.15.31:0/4029259629 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95a00fe2d0 con 0x7f9598046d60 2023-10-30T13:35:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f95a5590640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95840664a0 0x7f9584068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f3a3bfff640 1 Processor -- start 2023-10-30T13:35:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f3a3bfff640 1 -- start start 2023-10-30T13:35:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8c734c9640 1 -- start start 2023-10-30T13:35:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8c734c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 0x7f8c6c146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8c734c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c0ca8e0 0x7f8c6c1f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8c734c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c153b10 0x7f8c6c142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8c734c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c1414c0 con 0x7f8c6c0ca8e0 2023-10-30T13:35:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c734c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c151c40 con 0x7f8c6c153b10 2023-10-30T13:35:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c734c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c151ee0 con 0x7f8c6c06c480 2023-10-30T13:35:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c153b10 0x7f8c6c142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c153b10 0x7f8c6c142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60728/0 (socket says 172.21.15.31:60728) 2023-10-30T13:35:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 -- 172.21.15.31:0/2490336496 learned_addr learned my addr 172.21.15.31:0/2490336496 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 -- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 msgr2=0x7f8c6c146f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T13:35:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 0x7f8c6c146f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 -- 172.21.15.31:0/2490336496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c0ca8e0 msgr2=0x7f8c6c1f6260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T13:35:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c0ca8e0 0x7f8c6c1f6260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f62a2ffd640 1 -- 172.21.15.31:0/2703219694 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c0ac410 con 0x7f62a4153b10 2023-10-30T13:35:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f62a2ffd640 1 -- 172.21.15.31:0/2703219694 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c0ac570 con 0x7f62a4153b10 2023-10-30T13:35:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f62a2ffd640 1 -- 172.21.15.31:0/2703219694 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c0ac900 con 0x7f62a4153b10 2023-10-30T13:35:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7fe01d091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe0180c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7fe01d091640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe018071e00 con 0x7fe01806bb40 2023-10-30T13:35:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7fe01d091640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe018070020 con 0x7fe01806b760 2023-10-30T13:35:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7fe01d091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe018070f10 con 0x7fe01806c420 2023-10-30T13:35:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7fe01759e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe0180c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7fe01759e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe0180c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53440/0 (socket says 172.21.15.31:53440) 2023-10-30T13:35:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7fe01759e640 1 -- 172.21.15.31:0/1248708390 learned_addr learned my addr 172.21.15.31:0/1248708390 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7fe01759e640 1 -- 172.21.15.31:0/1248708390 >> v1:172.21.15.33:6789/0 conn(0x7fe01806b760 legacy=0x7fe01815e650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:35:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7fe01759e640 1 -- 172.21.15.31:0/1248708390 >> v1:172.21.15.31:6789/0 conn(0x7fe01806bb40 legacy=0x7fe018157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f5813aea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f580c153bb0 0x7f580c0c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f5813aea640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f580c159980 0x7f580c159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f5813aea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580c0c5f80 con 0x7f580c06c420 2023-10-30T13:35:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f5813aea640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c0c5690 con 0x7f580c159980 2023-10-30T13:35:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f5813aea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c15a2d0 con 0x7f580c153bb0 2023-10-30T13:35:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811085640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f580c06c420 0x7f580c0c6c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45776/0 (socket says 172.21.15.31:45776) 2023-10-30T13:35:16.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.692+0000 7f8af9ed6640 1 -- start start 2023-10-30T13:35:16.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af9ed6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8af40cb790 0x7f8af4155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 -- 172.21.15.31:0/2490336496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c009420 con 0x7f8c6c153b10 2023-10-30T13:35:16.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af9ed6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8af40c6ff0 0x7f8af414f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c71a66640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c153b10 0x7f8c6c142b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400ccf0 tx=0x7f8c6400a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 -- 172.21.15.31:0/2703219694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a4153b10 msgr2=0x7f62a40c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2703219694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a4153b10 0x7f62a40c9b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f628c0097c0 tx=0x7f628c01c4a0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 -- 172.21.15.31:0/2703219694 shutdown_connections 2023-10-30T13:35:16.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2703219694 >> v2:172.21.15.31:3300/0 conn(0x7f62a40ca8e0 0x7f62a406bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2703219694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a4153b10 0x7f62a40c9b90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 -- 172.21.15.31:0/2703219694 >> 172.21.15.31:0/2703219694 conn(0x7f62a406a240 msgr2=0x7f62a406a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 -- 172.21.15.31:0/2703219694 shutdown_connections 2023-10-30T13:35:16.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7fe01759e640 1 -- 172.21.15.31:0/1248708390 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180711f0 con 0x7fe01806c420 2023-10-30T13:35:16.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811886640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f580c153bb0 0x7f580c0c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811886640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f580c153bb0 0x7f580c0c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60712/0 (socket says 172.21.15.31:60712) 2023-10-30T13:35:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811886640 1 -- 172.21.15.31:0/1162540947 learned_addr learned my addr 172.21.15.31:0/1162540947 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5812087640 1 --2- 172.21.15.31:0/1162540947 >> v2:172.21.15.31:3300/0 conn(0x7f580c159980 0x7f580c159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af9ed6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40c50d0 con 0x7f8af40c6c10 2023-10-30T13:35:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af9ed6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af40c5a70 con 0x7f8af40cb790 2023-10-30T13:35:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af9ed6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af41567b0 con 0x7f8af40c6ff0 2023-10-30T13:35:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af2ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8af40cb790 0x7f8af4155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c61ffb640 1 -- 172.21.15.31:0/2490336496 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64003ba0 con 0x7f8c6c153b10 2023-10-30T13:35:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f62a9d65640 1 -- 172.21.15.31:0/2703219694 wait complete. 2023-10-30T13:35:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811886640 1 -- 172.21.15.31:0/1162540947 >> v2:172.21.15.31:3300/0 conn(0x7f580c159980 msgr2=0x7f580c159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811886640 1 --2- 172.21.15.31:0/1162540947 >> v2:172.21.15.31:3300/0 conn(0x7f580c159980 0x7f580c159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811886640 1 -- 172.21.15.31:0/1162540947 >> v1:172.21.15.33:6789/0 conn(0x7f580c06c420 legacy=0x7f580c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f5811886640 1 -- 172.21.15.31:0/1162540947 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c15a580 con 0x7f580c153bb0 2023-10-30T13:35:16.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af37fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8af40c6c10 0x7f8af40c5e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51904/0 (socket says 172.21.15.31:51904) 2023-10-30T13:35:16.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af37fe640 1 -- 172.21.15.31:0/2076695400 learned_addr learned my addr 172.21.15.31:0/2076695400 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:16.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8af3fff640 1 --2- 172.21.15.31:0/2076695400 >> v2:172.21.15.33:3300/0 conn(0x7f8af40c6ff0 0x7f8af414f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c1521a0 con 0x7f8c6c153b10 2023-10-30T13:35:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c1f9ab0 con 0x7f8c6c153b10 2023-10-30T13:35:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f3a3bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a34005f90 0x7f3a34049e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f3a3bfff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a3404c760 0x7f3a3417bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f3a3bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a340dd350 0x7f3a3404eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f8c61ffb640 1 -- 172.21.15.31:0/2490336496 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64003d00 con 0x7f8c6c153b10 2023-10-30T13:35:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.696+0000 7f7a42ffd640 1 --2- 172.21.15.31:0/2947570573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c0dd500 0x7f7a3c0e71a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7a34009870 tx=0x7f7a3401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a78b7c28d2221a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f8af3fff640 1 -- 172.21.15.31:0/2076695400 >> v2:172.21.15.31:3300/0 conn(0x7f8af40cb790 msgr2=0x7f8af4155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f8af3fff640 1 --2- 172.21.15.31:0/2076695400 >> v2:172.21.15.31:3300/0 conn(0x7f8af40cb790 0x7f8af4155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f95a5590640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95840664a0 0x7f9584068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f95900b7000 tx=0x7f95900af070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a34005f90 0x7f3a34049e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f8af3fff640 1 -- 172.21.15.31:0/2076695400 >> v1:172.21.15.31:6789/0 conn(0x7f8af40c6c10 legacy=0x7f8af40c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3bfff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a340d9380 con 0x7f3a340dd350 2023-10-30T13:35:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3bfff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3417eaa0 con 0x7f3a34005f90 2023-10-30T13:35:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f8af3fff640 1 -- 172.21.15.31:0/2076695400 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af4155f10 con 0x7f8af40c6ff0 2023-10-30T13:35:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3bfff640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3417f620 con 0x7f3a3404c760 2023-10-30T13:35:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a34005f90 0x7f3a34049e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60752/0 (socket says 172.21.15.31:60752) 2023-10-30T13:35:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 -- 172.21.15.31:0/2403563302 learned_addr learned my addr 172.21.15.31:0/2403563302 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f8c61ffb640 1 -- 172.21.15.31:0/2490336496 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64003ba0 con 0x7f8c6c153b10 2023-10-30T13:35:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3a7fc640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a3404c760 0x7f3a3417bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 -- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a3404c760 msgr2=0x7f3a3417bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a3404c760 0x7f3a3417bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a417d3640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a340dd350 0x7f3a3404eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 -- 172.21.15.31:0/2403563302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a340dd350 msgr2=0x7f3a3404eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a340dd350 0x7f3a3404eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 -- 172.21.15.31:0/2403563302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3417c4c0 con 0x7f3a34005f90 2023-10-30T13:35:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a417d3640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a340dd350 0x7f3a3404eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4c005260 con 0x7f8c6c153b10 2023-10-30T13:35:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3affd640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a34005f90 0x7f3a34049e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00b470 tx=0x7f3a2c00b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f7a41ffb640 1 -- 172.21.15.31:0/2947570573 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a340aa5f0 con 0x7f7a3c0dd500 2023-10-30T13:35:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7fe01759e640 1 --2- 172.21.15.31:0/1248708390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe0180c57a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0100098a0 tx=0x7fe01001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac00c1bf9b5fbd7a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a237fe640 1 -- 172.21.15.31:0/2403563302 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c011890 con 0x7f3a34005f90 2023-10-30T13:35:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a237fe640 1 -- 172.21.15.31:0/2403563302 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c0119f0 con 0x7f3a34005f90 2023-10-30T13:35:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3404a500 con 0x7f3a34005f90 2023-10-30T13:35:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f7a41ffb640 1 -- 172.21.15.31:0/2947570573 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a340aa750 con 0x7f7a3c0dd500 2023-10-30T13:35:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f62a9d65640 1 Processor -- start 2023-10-30T13:35:16.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f62a9d65640 1 -- start start 2023-10-30T13:35:16.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f3a237fe640 1 -- 172.21.15.31:0/2403563302 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c011da0 con 0x7f3a34005f90 2023-10-30T13:35:16.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f7a41ffb640 1 -- 172.21.15.31:0/2947570573 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a340aa5f0 con 0x7f7a3c0dd500 2023-10-30T13:35:16.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f95967fc640 1 -- 172.21.15.31:0/4029259629 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95a00cf370 con 0x7f9598046d60 2023-10-30T13:35:16.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7fe015d9b640 1 -- 172.21.15.31:0/1248708390 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100aa650 con 0x7fe01806c420 2023-10-30T13:35:16.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7fe015d9b640 1 -- 172.21.15.31:0/1248708390 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0100aa7b0 con 0x7fe01806c420 2023-10-30T13:35:16.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7fe015d9b640 1 -- 172.21.15.31:0/1248708390 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100aa650 con 0x7fe01806c420 2023-10-30T13:35:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3417efd0 con 0x7f3a34005f90 2023-10-30T13:35:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f7a43fff640 1 -- 172.21.15.31:0/2947570573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c0dd500 msgr2=0x7f7a3c0e71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f7a43fff640 1 --2- 172.21.15.31:0/2947570573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c0dd500 0x7f7a3c0e71a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7a34009870 tx=0x7f7a3401c490 comp rx=0 tx=0).stop 2023-10-30T13:35:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f7a43fff640 1 -- 172.21.15.31:0/2947570573 shutdown_connections 2023-10-30T13:35:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f7a43fff640 1 --2- 172.21.15.31:0/2947570573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c0dd500 0x7f7a3c0e71a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f7a43fff640 1 -- 172.21.15.31:0/2947570573 >> 172.21.15.31:0/2947570573 conn(0x7f7a3c0545a0 msgr2=0x7f7a3c0549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f62a9d65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a406c480 0x7f62a40b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f62a9d65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a40ca8e0 0x7f62a413fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a9d65640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62a4153b10 0x7f62a41401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a9d65640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cf270 con 0x7f62a40ca8e0 2023-10-30T13:35:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a9d65640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40ce470 con 0x7f62a4153b10 2023-10-30T13:35:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a9d65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a414d990 con 0x7f62a406c480 2023-10-30T13:35:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a37fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a406c480 0x7f62a40b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a37fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a406c480 0x7f62a40b2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53446/0 (socket says 172.21.15.31:53446) 2023-10-30T13:35:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a37fe640 1 -- 172.21.15.31:0/2872513828 learned_addr learned my addr 172.21.15.31:0/2872513828 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f629affd640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a40ca8e0 0x7f62a413fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a3fff640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62a4153b10 0x7f62a41401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f8c61ffb640 1 -- 172.21.15.31:0/2490336496 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c64003e70 con 0x7f8c6c153b10 2023-10-30T13:35:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.700+0000 7f5811886640 1 --2- 172.21.15.31:0/1162540947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c153bb0 0x7f580c0c6410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f580c0cb840 tx=0x7f580801ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb1520cfe3e8b937 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7fe01d091640 1 -- 172.21.15.31:0/1248708390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 msgr2=0x7fe0180c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7fe01d091640 1 --2- 172.21.15.31:0/1248708390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe0180c57a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0100098a0 tx=0x7fe01001c510 comp rx=0 tx=0).stop 2023-10-30T13:35:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a3fff640 1 -- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a406c480 msgr2=0x7f62a40b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a3fff640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a406c480 0x7f62a40b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a3fff640 1 -- 172.21.15.31:0/2872513828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a40ca8e0 msgr2=0x7f62a413fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a3fff640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a40ca8e0 0x7f62a413fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a3fff640 1 -- 172.21.15.31:0/2872513828 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c009420 con 0x7f62a4153b10 2023-10-30T13:35:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7fe01d091640 1 -- 172.21.15.31:0/1248708390 shutdown_connections 2023-10-30T13:35:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7fe01d091640 1 --2- 172.21.15.31:0/1248708390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe0180c57a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7fe01d091640 1 -- 172.21.15.31:0/1248708390 >> 172.21.15.31:0/1248708390 conn(0x7fe0180cac70 msgr2=0x7fe0180cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f7a43fff640 1 -- 172.21.15.31:0/2947570573 shutdown_connections 2023-10-30T13:35:16.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f62a3fff640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62a4153b10 0x7f62a41401a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f629c00c3c0 tx=0x7f629c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a34005450 con 0x7f3a34005f90 2023-10-30T13:35:16.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f7a43fff640 1 -- 172.21.15.31:0/2947570573 wait complete. 2023-10-30T13:35:16.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f8c61ffb640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c50066560 0x7f8c50068a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7fe01d091640 1 -- 172.21.15.31:0/1248708390 shutdown_connections 2023-10-30T13:35:16.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f8c61ffb640 1 -- 172.21.15.31:0/2490336496 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c64012070 con 0x7f8c6c153b10 2023-10-30T13:35:16.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f8c71265640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c50066560 0x7f8c50068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7fe01d091640 1 -- 172.21.15.31:0/1248708390 wait complete. 2023-10-30T13:35:16.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f8af3fff640 1 --2- 172.21.15.31:0/2076695400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6ff0 0x7f8af414f450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8af40c5970 tx=0x7f8aec01f110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afb2bc0abfe67bf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f8c71265640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c50066560 0x7f8c50068a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009940 tx=0x7f8c68009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.704+0000 7f7a43fff640 1 Processor -- start 2023-10-30T13:35:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 Processor -- start 2023-10-30T13:35:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 -- start start 2023-10-30T13:35:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 -- start start 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c049440 0x7f7a3c0e06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a3c04f680 0x7f7a3c0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a3c0dd500 0x7f7a3c0e4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c052bb0 con 0x7f7a3c049440 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0e3a80 con 0x7f7a3c0dd500 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0e2480 con 0x7f7a3c04f680 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c049440 0x7f7a3c0e06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c049440 0x7f7a3c0e06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36046/0 (socket says 172.21.15.31:36046) 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 -- 172.21.15.31:0/4231899524 learned_addr learned my addr 172.21.15.31:0/4231899524 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a427fc640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a3c04f680 0x7f7a3c0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a437fe640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a3c0dd500 0x7f7a3c0e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe01806b760 0x7fe01814c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe01806bb40 0x7fe0181489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe018154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 -- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a3c04f680 msgr2=0x7f7a3c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a3c04f680 0x7f7a3c0d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 -- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a3c0dd500 msgr2=0x7f7a3c0e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a3c0dd500 0x7f7a3c0e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 -- 172.21.15.31:0/4231899524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340094d0 con 0x7f7a3c049440 2023-10-30T13:35:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a427fc640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a3c04f680 0x7f7a3c0d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0181f5c50 con 0x7fe01806b760 2023-10-30T13:35:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe018148f30 con 0x7fe01806bb40 2023-10-30T13:35:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01d091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0181f9eb0 con 0x7fe01806c420 2023-10-30T13:35:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe01806b760 0x7fe01814c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe01806bb40 0x7fe0181489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe01806b760 0x7fe01814c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36062/0 (socket says 172.21.15.31:36062) 2023-10-30T13:35:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 -- 172.21.15.31:0/138199069 learned_addr learned my addr 172.21.15.31:0/138199069 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01759e640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe018154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a42ffd640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c049440 0x7f7a3c0e06d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a34000c10 tx=0x7f7a34019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 -- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 msgr2=0x7fe018154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe018154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 -- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe01806bb40 msgr2=0x7fe0181489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe01806bb40 0x7fe0181489f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 -- 172.21.15.31:0/138199069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0100094d0 con 0x7fe01806b760 2023-10-30T13:35:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe01659c640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe01806bb40 0x7fe0181489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a2bfff640 1 -- 172.21.15.31:0/4231899524 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a340aad50 con 0x7f7a3c049440 2023-10-30T13:35:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a2bfff640 1 -- 172.21.15.31:0/4231899524 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a34003620 con 0x7f7a3c049440 2023-10-30T13:35:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a2bfff640 1 -- 172.21.15.31:0/4231899524 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a340039b0 con 0x7f7a3c049440 2023-10-30T13:35:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c052de0 con 0x7f7a3c049440 2023-10-30T13:35:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fe016d9d640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe01806b760 0x7fe01814c530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00b4a0 tx=0x7fe00c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c0e1b50 con 0x7f7a3c049440 2023-10-30T13:35:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.708+0000 7fdfff7fe640 1 -- 172.21.15.31:0/138199069 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c010610 con 0x7fe01806b760 2023-10-30T13:35:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0181f5e40 con 0x7fe01806b760 2023-10-30T13:35:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0181f6400 con 0x7fe01806b760 2023-10-30T13:35:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7fdfff7fe640 1 -- 172.21.15.31:0/138199069 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c010770 con 0x7fe01806b760 2023-10-30T13:35:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7fdfff7fe640 1 -- 172.21.15.31:0/138199069 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c010610 con 0x7fe01806b760 2023-10-30T13:35:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c0056d0 con 0x7f7a3c049440 2023-10-30T13:35:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff8005260 con 0x7fe01806b760 2023-10-30T13:35:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7f7a2bfff640 1 -- 172.21.15.31:0/4231899524 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a34003b10 con 0x7f7a3c049440 2023-10-30T13:35:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.712+0000 7fdfff7fe640 1 -- 172.21.15.31:0/138199069 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe00c0108e0 con 0x7fe01806b760 2023-10-30T13:35:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f7a2bfff640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a300664a0 0x7f7a30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5810884640 1 -- 172.21.15.31:0/1162540947 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58080a89f0 con 0x7f580c153bb0 2023-10-30T13:35:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af27fc640 1 -- 172.21.15.31:0/2076695400 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0b0440 con 0x7f8af40c6ff0 2023-10-30T13:35:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8c61ffb640 1 -- 172.21.15.31:0/2490336496 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c64051cb0 con 0x7f8c6c153b10 2023-10-30T13:35:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f7a2bfff640 1 -- 172.21.15.31:0/4231899524 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a340a7070 con 0x7f7a3c049440 2023-10-30T13:35:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7fdfff7fe640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe004066500 0x7fe0040689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af27fc640 1 -- 172.21.15.31:0/2076695400 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec0b05a0 con 0x7f8af40c6ff0 2023-10-30T13:35:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af27fc640 1 -- 172.21.15.31:0/2076695400 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0b0930 con 0x7f8af40c6ff0 2023-10-30T13:35:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7fdfff7fe640 1 -- 172.21.15.31:0/138199069 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe00c083cd0 con 0x7fe01806b760 2023-10-30T13:35:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f7a427fc640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a300664a0 0x7f7a30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f62a0ff9640 1 -- 172.21.15.31:0/2872513828 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c005cf0 con 0x7f62a4153b10 2023-10-30T13:35:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a40ce030 con 0x7f62a4153b10 2023-10-30T13:35:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a40ce8d0 con 0x7f62a4153b10 2023-10-30T13:35:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7fe01659c640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe004066500 0x7fe0040689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f62a0ff9640 1 -- 172.21.15.31:0/2872513828 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c005e50 con 0x7f62a4153b10 2023-10-30T13:35:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f62a0ff9640 1 -- 172.21.15.31:0/2872513828 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c0072d0 con 0x7f62a4153b10 2023-10-30T13:35:16.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5810884640 1 -- 172.21.15.31:0/1162540947 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58080a8b50 con 0x7f580c153bb0 2023-10-30T13:35:16.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5810884640 1 -- 172.21.15.31:0/1162540947 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58080b0440 con 0x7f580c153bb0 2023-10-30T13:35:16.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2076695400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6ff0 msgr2=0x7f8af414f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2076695400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6ff0 0x7f8af414f450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8af40c5970 tx=0x7f8aec01f110 comp rx=0 tx=0).stop 2023-10-30T13:35:16.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2076695400 shutdown_connections 2023-10-30T13:35:16.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2076695400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6ff0 0x7f8af414f450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2076695400 >> v2:172.21.15.31:3300/0 conn(0x7f8af40cb790 0x7f8af4155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2076695400 >> 172.21.15.31:0/2076695400 conn(0x7f8af40cac70 msgr2=0x7f8af40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5813aea640 1 -- 172.21.15.31:0/1162540947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c153bb0 msgr2=0x7f580c0c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5813aea640 1 --2- 172.21.15.31:0/1162540947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c153bb0 0x7f580c0c6410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f580c0cb840 tx=0x7f580801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6288005260 con 0x7f62a4153b10 2023-10-30T13:35:16.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5813aea640 1 -- 172.21.15.31:0/1162540947 shutdown_connections 2023-10-30T13:35:16.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5813aea640 1 --2- 172.21.15.31:0/1162540947 >> v2:172.21.15.31:3300/0 conn(0x7f580c159980 0x7f580c159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5813aea640 1 --2- 172.21.15.31:0/1162540947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c153bb0 0x7f580c0c6410 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f580c0cb840 tx=0x7f580801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f5813aea640 1 -- 172.21.15.31:0/1162540947 >> 172.21.15.31:0/1162540947 conn(0x7f580c0cac70 msgr2=0x7f580c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2076695400 shutdown_connections 2023-10-30T13:35:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f7a427fc640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a300664a0 0x7f7a30068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c005cf0 tx=0x7f7a38008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f7a2bfff640 1 -- 172.21.15.31:0/4231899524 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a340ec1b0 con 0x7f7a3c049440 2023-10-30T13:35:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7fe01659c640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe004066500 0x7fe0040689c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe0080012f0 tx=0x7fe008009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.716+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2076695400 wait complete. 2023-10-30T13:35:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f5813aea640 1 -- 172.21.15.31:0/1162540947 shutdown_connections 2023-10-30T13:35:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f3a237fe640 1 -- 172.21.15.31:0/2403563302 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a2c01a400 con 0x7f3a34005f90 2023-10-30T13:35:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f8af9ed6640 1 Processor -- start 2023-10-30T13:35:16.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f5813aea640 1 -- 172.21.15.31:0/1162540947 wait complete. 2023-10-30T13:35:16.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f3a237fe640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a280664f0 0x7f3a280689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f3a237fe640 1 -- 172.21.15.31:0/2403563302 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a2c084780 con 0x7f3a34005f90 2023-10-30T13:35:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f3a3a7fc640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a280664f0 0x7f3a280689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f5813aea640 1 Processor -- start 2023-10-30T13:35:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f5813aea640 1 -- start start 2023-10-30T13:35:16.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f3a3a7fc640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a280664f0 0x7f3a280689b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a3404f5b0 tx=0x7f3a30007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.720+0000 7f8af9ed6640 1 -- start start 2023-10-30T13:35:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af9ed6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6c10 0x7f8af41df810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af9ed6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8af40c6ff0 0x7f8af41e00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af9ed6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8af40cb790 0x7f8af41f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af9ed6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41f3eb0 con 0x7f8af40c6ff0 2023-10-30T13:35:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af9ed6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41472a0 con 0x7f8af40c6c10 2023-10-30T13:35:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af9ed6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af4147820 con 0x7f8af40cb790 2023-10-30T13:35:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6c10 0x7f8af41df810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6c10 0x7f8af41df810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60794/0 (socket says 172.21.15.31:60794) 2023-10-30T13:35:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 -- 172.21.15.31:0/2709888536 learned_addr learned my addr 172.21.15.31:0/2709888536 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af3fff640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8af40cb790 0x7f8af41f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5813aea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c06c420 0x7f580c1fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5813aea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f580c159980 0x7f580c1583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811886640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c06c420 0x7f580c1fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af2ffd640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8af40c6ff0 0x7f8af41e00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f580c159980 0x7f580c1583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f580c159980 0x7f580c1583c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36086/0 (socket says 172.21.15.31:36086) 2023-10-30T13:35:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 -- 172.21.15.31:0/2672284719 learned_addr learned my addr 172.21.15.31:0/2672284719 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5813aea640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f580c150600 0x7f580c150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0bb3b0 con 0x7f580c159980 2023-10-30T13:35:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0b84d0 con 0x7f580c06c420 2023-10-30T13:35:16.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 -- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8af40cb790 msgr2=0x7f8af41f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c8e50 con 0x7f580c150600 2023-10-30T13:35:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8af40cb790 0x7f8af41f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 -- 172.21.15.31:0/2709888536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8af40c6ff0 msgr2=0x7f8af41e00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8af40c6ff0 0x7f8af41e00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 -- 172.21.15.31:0/2709888536 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec009a70 con 0x7f8af40c6c10 2023-10-30T13:35:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af2ffd640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8af40c6ff0 0x7f8af41e00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:16.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5812087640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f580c150600 0x7f580c150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f8af37fe640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6c10 0x7f8af41df810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400c500 tx=0x7f8ae400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7fdfff7fe640 1 -- 172.21.15.31:0/138199069 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe00c0511e0 con 0x7fe01806b760 2023-10-30T13:35:16.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 -- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f580c150600 msgr2=0x7f580c150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f580c150600 0x7f580c150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 -- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c06c420 msgr2=0x7f580c1fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c06c420 0x7f580c1fbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.724+0000 7f5811085640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58080094d0 con 0x7f580c159980 2023-10-30T13:35:16.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f5812087640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f580c150600 0x7f580c150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:16.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f5811085640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f580c159980 0x7f580c1583c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f580000c430 tx=0x7f580000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f57fe7fc640 1 -- 172.21.15.31:0/2672284719 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58000074e0 con 0x7f580c159980 2023-10-30T13:35:16.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f57fe7fc640 1 -- 172.21.15.31:0/2672284719 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800007640 con 0x7f580c159980 2023-10-30T13:35:16.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f57fe7fc640 1 -- 172.21.15.31:0/2672284719 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58000049f0 con 0x7f580c159980 2023-10-30T13:35:16.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c0b9d20 con 0x7f580c159980 2023-10-30T13:35:16.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c0ba2e0 con 0x7f580c159980 2023-10-30T13:35:16.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c0b8640 con 0x7f580c159980 2023-10-30T13:35:16.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f3a237fe640 1 -- 172.21.15.31:0/2403563302 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a2c051be0 con 0x7f3a34005f90 2023-10-30T13:35:16.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f8adbfff640 1 -- 172.21.15.31:0/2709888536 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4005cf0 con 0x7f8af40c6c10 2023-10-30T13:35:16.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f8adbfff640 1 -- 172.21.15.31:0/2709888536 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae4005e50 con 0x7f8af40c6c10 2023-10-30T13:35:16.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.728+0000 7f8adbfff640 1 -- 172.21.15.31:0/2709888536 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae40072f0 con 0x7f8af40c6c10 2023-10-30T13:35:16.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.732+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af4147540 con 0x7f8af40c6c10 2023-10-30T13:35:16.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.732+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af41df010 con 0x7f8af40c6c10 2023-10-30T13:35:16.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.732+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af41437f0 con 0x7f8af40c6c10 2023-10-30T13:35:16.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.732+0000 7f57fe7fc640 1 -- 172.21.15.31:0/2672284719 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5800004b50 con 0x7f580c159980 2023-10-30T13:35:16.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.732+0000 7f57fe7fc640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f57f00664a0 0x7f57f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.732+0000 7f57fe7fc640 1 -- 172.21.15.31:0/2672284719 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5800083890 con 0x7f580c159980 2023-10-30T13:35:16.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.732+0000 7f5811886640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f57f00664a0 0x7f57f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.736+0000 7f5811886640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f57f00664a0 0x7f57f0068960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f58080b2000 tx=0x7f5808008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.736+0000 7f62a0ff9640 1 -- 172.21.15.31:0/2872513828 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f629c007430 con 0x7f62a4153b10 2023-10-30T13:35:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.736+0000 7f62a0ff9640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f62840664d0 0x7f6284068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.736+0000 7f62a0ff9640 1 -- 172.21.15.31:0/2872513828 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f629c0837f0 con 0x7f62a4153b10 2023-10-30T13:35:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.736+0000 7f62a37fe640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f62840664d0 0x7f6284068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.740+0000 7f62a37fe640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f62840664d0 0x7f6284068990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f628c0af040 tx=0x7f628c0a6040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.744+0000 7f62a0ff9640 1 -- 172.21.15.31:0/2872513828 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f629c050d00 con 0x7f62a4153b10 2023-10-30T13:35:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.744+0000 7f57fe7fc640 1 -- 172.21.15.31:0/2672284719 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5800050d00 con 0x7f580c159980 2023-10-30T13:35:16.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.744+0000 7f8adbfff640 1 -- 172.21.15.31:0/2709888536 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ae4007450 con 0x7f8af40c6c10 2023-10-30T13:35:16.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.748+0000 7f8adbfff640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ae00664d0 0x7f8ae0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.748+0000 7f8adbfff640 1 -- 172.21.15.31:0/2709888536 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ae4083910 con 0x7f8af40c6c10 2023-10-30T13:35:16.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.748+0000 7f8af2ffd640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ae00664d0 0x7f8ae0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.752+0000 7f8af2ffd640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ae00664d0 0x7f8ae0068990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8af41f7520 tx=0x7f8ae8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:16.756+0000 7f8adbfff640 1 -- 172.21.15.31:0/2709888536 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ae4050e20 con 0x7f8af40c6c10 2023-10-30T13:35:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.020+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f95880057e0 con 0x7f9598046d60 2023-10-30T13:35:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.024+0000 7f95967fc640 1 -- 172.21.15.31:0/4029259629 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f95a0099020 con 0x7f9598046d60 2023-10-30T13:35:17.034 INFO:teuthology.orchestra.run.smithi031.stdout:60129542152 2023-10-30T13:35:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95840664a0 msgr2=0x7f9584068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95840664a0 0x7f9584068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f95900b7000 tx=0x7f95900af070 comp rx=0 tx=0).stop 2023-10-30T13:35:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598046d60 msgr2=0x7f95980e1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598046d60 0x7f95980e1170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95a0000b70 tx=0x7f95a00674c0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 shutdown_connections 2023-10-30T13:35:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95840664a0 0x7f9584068960 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95980d8f90 0x7f9598047760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f959804c5f0 0x7f9598176350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 --2- 172.21.15.31:0/4029259629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598046d60 0x7f95980e1170 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 >> 172.21.15.31:0/4029259629 conn(0x7f9598050170 msgr2=0x7f9598034e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.032+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 shutdown_connections 2023-10-30T13:35:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.036+0000 7f95a7ff5640 1 -- 172.21.15.31:0/4029259629 wait complete. 2023-10-30T13:35:17.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.040+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8c4c0057e0 con 0x7f8c6c153b10 2023-10-30T13:35:17.050 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2023-10-30T13:35:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.044+0000 7f8c61ffb640 1 -- 172.21.15.31:0/2490336496 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c640552d0 con 0x7f8c6c153b10 2023-10-30T13:35:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c50066560 msgr2=0x7f8c50068a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c50066560 0x7f8c50068a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009940 tx=0x7f8c68009240 comp rx=0 tx=0).stop 2023-10-30T13:35:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c153b10 msgr2=0x7f8c6c142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c153b10 0x7f8c6c142b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400ccf0 tx=0x7f8c6400a660 comp rx=0 tx=0).stop 2023-10-30T13:35:17.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 shutdown_connections 2023-10-30T13:35:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c50066560 0x7f8c50068a20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c6c153b10 0x7f8c6c142b80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c0ca8e0 0x7f8c6c1f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 --2- 172.21.15.31:0/2490336496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c6c06c480 0x7f8c6c146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 >> 172.21.15.31:0/2490336496 conn(0x7f8c6c06a240 msgr2=0x7f8c6c158260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 shutdown_connections 2023-10-30T13:35:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.048+0000 7f8c734c9640 1 -- 172.21.15.31:0/2490336496 wait complete. 2023-10-30T13:35:17.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.060+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3a3404b870 con 0x7f3a34005f90 2023-10-30T13:35:17.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.060+0000 7f3a237fe640 1 -- 172.21.15.31:0/2403563302 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a2c055200 con 0x7f3a34005f90 2023-10-30T13:35:17.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.060+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7a3c053530 con 0x7f7a3c049440 2023-10-30T13:35:17.066 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2023-10-30T13:35:17.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.060+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdff80057e0 con 0x7fe01806b760 2023-10-30T13:35:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.060+0000 7f7a2bfff640 1 -- 172.21.15.31:0/4231899524 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a340c3020 con 0x7f7a3c049440 2023-10-30T13:35:17.067 INFO:teuthology.orchestra.run.smithi031.stdout:60129542150 2023-10-30T13:35:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7fdfff7fe640 1 -- 172.21.15.31:0/138199069 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe00c014020 con 0x7fe01806b760 2023-10-30T13:35:17.068 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2023-10-30T13:35:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a280664f0 msgr2=0x7f3a280689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a280664f0 0x7f3a280689b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a3404f5b0 tx=0x7f3a30007e40 comp rx=0 tx=0).stop 2023-10-30T13:35:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a34005f90 msgr2=0x7f3a34049e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a34005f90 0x7f3a34049e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00b470 tx=0x7f3a2c00b930 comp rx=0 tx=0).stop 2023-10-30T13:35:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 shutdown_connections 2023-10-30T13:35:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a280664f0 0x7f3a280689b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a340dd350 0x7f3a3404eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a3404c760 0x7f3a3417bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 --2- 172.21.15.31:0/2403563302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a34005f90 0x7f3a34049e00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 >> 172.21.15.31:0/2403563302 conn(0x7f3a34050340 msgr2=0x7f3a340e2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 shutdown_connections 2023-10-30T13:35:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f3a3bfff640 1 -- 172.21.15.31:0/2403563302 wait complete. 2023-10-30T13:35:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a300664a0 msgr2=0x7f7a30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a300664a0 0x7f7a30068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c005cf0 tx=0x7f7a38008040 comp rx=0 tx=0).stop 2023-10-30T13:35:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c049440 msgr2=0x7f7a3c0e06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c049440 0x7f7a3c0e06d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a34000c10 tx=0x7f7a34019230 comp rx=0 tx=0).stop 2023-10-30T13:35:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 shutdown_connections 2023-10-30T13:35:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a300664a0 0x7f7a30068960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a3c0dd500 0x7f7a3c0e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a3c04f680 0x7f7a3c0d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 --2- 172.21.15.31:0/4231899524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c049440 0x7f7a3c0e06d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 >> 172.21.15.31:0/4231899524 conn(0x7f7a3c0545a0 msgr2=0x7f7a3c0dc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 shutdown_connections 2023-10-30T13:35:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.064+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe004066500 msgr2=0x7fe0040689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe004066500 0x7fe0040689c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe0080012f0 tx=0x7fe008009040 comp rx=0 tx=0).stop 2023-10-30T13:35:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7f7a43fff640 1 -- 172.21.15.31:0/4231899524 wait complete. 2023-10-30T13:35:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe01806b760 msgr2=0x7fe01814c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe01806b760 0x7fe01814c530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00b4a0 tx=0x7fe00c00b960 comp rx=0 tx=0).stop 2023-10-30T13:35:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 shutdown_connections 2023-10-30T13:35:17.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe004066500 0x7fe0040689c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe01806c420 0x7fe018154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe01806bb40 0x7fe0181489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 --2- 172.21.15.31:0/138199069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe01806b760 0x7fe01814c530 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 >> 172.21.15.31:0/138199069 conn(0x7fe0180cac70 msgr2=0x7fe01814cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.0 2023-10-30T13:35:17.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 shutdown_connections 2023-10-30T13:35:17.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.068+0000 7fe01d091640 1 -- 172.21.15.31:0/138199069 wait complete. 2023-10-30T13:35:17.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.072+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f62880057e0 con 0x7f62a4153b10 2023-10-30T13:35:17.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.076+0000 7f62a0ff9640 1 -- 172.21.15.31:0/2872513828 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f629c021020 con 0x7f62a4153b10 2023-10-30T13:35:17.082 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2023-10-30T13:35:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f62840664d0 msgr2=0x7f6284068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f62840664d0 0x7f6284068990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f628c0af040 tx=0x7f628c0a6040 comp rx=0 tx=0).stop 2023-10-30T13:35:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62a4153b10 msgr2=0x7f62a41401a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62a4153b10 0x7f62a41401a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f629c00c3c0 tx=0x7f629c00c880 comp rx=0 tx=0).stop 2023-10-30T13:35:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 shutdown_connections 2023-10-30T13:35:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f62840664d0 0x7f6284068990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62a4153b10 0x7f62a41401a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a40ca8e0 0x7f62a413fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 --2- 172.21.15.31:0/2872513828 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f62a406c480 0x7f62a40b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 >> 172.21.15.31:0/2872513828 conn(0x7f62a406a240 msgr2=0x7f62a4159190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 shutdown_connections 2023-10-30T13:35:17.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f62a9d65640 1 -- 172.21.15.31:0/2872513828 wait complete. 2023-10-30T13:35:17.086 INFO:teuthology.orchestra.run.smithi031.stdout:60129542150 2023-10-30T13:35:17.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f580c04e170 con 0x7f580c159980 2023-10-30T13:35:17.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.080+0000 7f57fe7fc640 1 -- 172.21.15.31:0/2672284719 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5800054320 con 0x7f580c159980 2023-10-30T13:35:17.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f57f00664a0 msgr2=0x7f57f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f57f00664a0 0x7f57f0068960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f58080b2000 tx=0x7f5808008040 comp rx=0 tx=0).stop 2023-10-30T13:35:17.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f580c159980 msgr2=0x7f580c1583c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f580c159980 0x7f580c1583c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f580000c430 tx=0x7f580000c8f0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.2 2023-10-30T13:35:17.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 shutdown_connections 2023-10-30T13:35:17.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f57f00664a0 0x7f57f0068960 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f58080b2000 tx=0x7f5808008040 comp rx=0 tx=0).stop 2023-10-30T13:35:17.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f580c150600 0x7f580c150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f580c159980 0x7f580c1583c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 --2- 172.21.15.31:0/2672284719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f580c06c420 0x7f580c1fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 >> 172.21.15.31:0/2672284719 conn(0x7f580c0cac70 msgr2=0x7f580c0c9650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8af404e170 con 0x7f8af40c6c10 2023-10-30T13:35:17.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 shutdown_connections 2023-10-30T13:35:17.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f5813aea640 1 -- 172.21.15.31:0/2672284719 wait complete. 2023-10-30T13:35:17.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.084+0000 7f8adbfff640 1 -- 172.21.15.31:0/2709888536 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ae4021090 con 0x7f8af40c6c10 2023-10-30T13:35:17.092 INFO:teuthology.orchestra.run.smithi031.stdout:60129542152 2023-10-30T13:35:17.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ae00664d0 msgr2=0x7f8ae0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ae00664d0 0x7f8ae0068990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8af41f7520 tx=0x7f8ae8008040 comp rx=0 tx=0).stop 2023-10-30T13:35:17.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6c10 msgr2=0x7f8af41df810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:17.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6c10 0x7f8af41df810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400c500 tx=0x7f8ae400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 shutdown_connections 2023-10-30T13:35:17.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ae00664d0 0x7f8ae0068990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8af40cb790 0x7f8af41f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8af40c6ff0 0x7f8af41e00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 --2- 172.21.15.31:0/2709888536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af40c6c10 0x7f8af41df810 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:17.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 >> 172.21.15.31:0/2709888536 conn(0x7f8af40cac70 msgr2=0x7f8af40c3460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:17.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 shutdown_connections 2023-10-30T13:35:17.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:17.088+0000 7f8af9ed6640 1 -- 172.21.15.31:0/2709888536 wait complete. 2023-10-30T13:35:17.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.4 2023-10-30T13:35:17.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.1 2023-10-30T13:35:17.114 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.7 2023-10-30T13:35:17.120 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.3 2023-10-30T13:35:17.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.5 2023-10-30T13:35:17.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.6 2023-10-30T13:35:18.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T13:35:18.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T13:35:18.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T13:35:18.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T13:35:18.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T13:35:18.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T13:35:18.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T13:35:18.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T13:35:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.296+0000 7fc345bb8640 1 Processor -- start 2023-10-30T13:35:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.296+0000 7fc345bb8640 1 -- start start 2023-10-30T13:35:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc345bb8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc340071d50 0x7fc34006fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc345bb8640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc3400c7070 0x7fc340156240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc345bb8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc340159490 con 0x7fc340159920 2023-10-30T13:35:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc33ffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fc340159920 0x7fc3400c7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45790/0 (socket says 172.21.15.31:45790) 2023-10-30T13:35:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc33ffff640 1 -- 172.21.15.31:0/19674222 learned_addr learned my addr 172.21.15.31:0/19674222 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc345bb8640 1 -- 172.21.15.31:0/19674222 --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3400c67f0 con 0x7fc3400c7070 2023-10-30T13:35:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc345bb8640 1 -- 172.21.15.31:0/19674222 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3400c6230 con 0x7fc340071d50 2023-10-30T13:35:18.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.300+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093898970 0 0) 0x7fc340159490 con 0x7fc340159920 2023-10-30T13:35:18.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc328004640 con 0x7fc340159920 2023-10-30T13:35:18.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33f7fe640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc340071d50 0x7fc34006fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33effd640 1 --2- 172.21.15.31:0/19674222 >> v2:172.21.15.33:3300/0 conn(0x7fc3400c7070 0x7fc340156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 64653131 0 0) 0x7fc328004640 con 0x7fc340159920 2023-10-30T13:35:18.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc340071d50 msgr2=0x7fc34006fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc340071d50 0x7fc34006fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 >> v2:172.21.15.33:3300/0 conn(0x7fc3400c7070 msgr2=0x7fc340156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 --2- 172.21.15.31:0/19674222 >> v2:172.21.15.33:3300/0 conn(0x7fc3400c7070 0x7fc340156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3400c6510 con 0x7fc340159920 2023-10-30T13:35:18.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fc334004ce0 con 0x7fc340159920 2023-10-30T13:35:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 >> v1:172.21.15.33:6789/0 conn(0x7fc340159920 legacy=0x7fc3400c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328004a90 0x7fc328004eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc3280053f0 0x7fc3280058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc328006050 0x7fc328003450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3400c67f0 con 0x7fc328004a90 2023-10-30T13:35:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3400c6230 con 0x7fc328006050 2023-10-30T13:35:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc328003990 con 0x7fc3280053f0 2023-10-30T13:35:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33ffff640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc328006050 0x7fc328003450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33effd640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc3280053f0 0x7fc3280058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33f7fe640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328004a90 0x7fc328004eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33ffff640 1 -- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc3280053f0 msgr2=0x7fc3280058e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33ffff640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc3280053f0 0x7fc3280058e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33ffff640 1 -- 172.21.15.31:0/19674222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328004a90 msgr2=0x7fc328004eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33ffff640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328004a90 0x7fc328004eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.304+0000 7fc33ffff640 1 -- 172.21.15.31:0/19674222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc328004640 con 0x7fc328006050 2023-10-30T13:35:18.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc33ffff640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc328006050 0x7fc328003450 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc334001e20 tx=0x7fc33401f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13ef23aaa260e4e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3340aa620 con 0x7fc328006050 2023-10-30T13:35:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc33e7fc640 1 -- 172.21.15.31:0/19674222 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3340aa7a0 con 0x7fc328006050 2023-10-30T13:35:18.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc345bb8640 1 -- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc328006050 msgr2=0x7fc328003450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc345bb8640 1 --2- 172.21.15.31:0/19674222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc328006050 0x7fc328003450 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc334001e20 tx=0x7fc33401f8d0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc33ffff640 1 -- 172.21.15.31:0/19674222 reap_dead start 2023-10-30T13:35:18.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc345bb8640 1 -- 172.21.15.31:0/19674222 shutdown_connections 2023-10-30T13:35:18.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc345bb8640 1 -- 172.21.15.31:0/19674222 >> 172.21.15.31:0/19674222 conn(0x7fc34006d810 msgr2=0x7fc34006dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc345bb8640 1 -- 172.21.15.31:0/19674222 shutdown_connections 2023-10-30T13:35:18.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.312+0000 7fc345bb8640 1 -- 172.21.15.31:0/19674222 wait complete. 2023-10-30T13:35:18.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.316+0000 7fc345bb8640 1 Processor -- start 2023-10-30T13:35:18.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.316+0000 7fc345bb8640 1 -- start start 2023-10-30T13:35:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc345bb8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328006050 0x7fc3401ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc345bb8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc34014cae0 0x7fc3401504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc345bb8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc34014a9a0 0x7fc3400c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc345bb8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc340150010 con 0x7fc328006050 2023-10-30T13:35:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc345bb8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3401565c0 con 0x7fc34014a9a0 2023-10-30T13:35:18.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc345bb8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34014e210 con 0x7fc34014cae0 2023-10-30T13:35:18.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328006050 0x7fc3401ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328006050 0x7fc3401ea6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36108/0 (socket says 172.21.15.31:36108) 2023-10-30T13:35:18.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 -- 172.21.15.31:0/2309017382 learned_addr learned my addr 172.21.15.31:0/2309017382 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc34014a9a0 0x7fc3400c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33effd640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc34014cae0 0x7fc3401504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 -- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc34014cae0 msgr2=0x7fc3401504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc34014cae0 0x7fc3401504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 -- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc34014a9a0 msgr2=0x7fc3400c13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc34014a9a0 0x7fc3400c13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 -- 172.21.15.31:0/2309017382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34014ae20 con 0x7fc328006050 2023-10-30T13:35:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33ffff640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc34014a9a0 0x7fc3400c13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc33f7fe640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328006050 0x7fc3401ea6e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc330006d50 tx=0x7fc33000f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc327fff640 1 -- 172.21.15.31:0/2309017382 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330014870 con 0x7fc328006050 2023-10-30T13:35:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc327fff640 1 -- 172.21.15.31:0/2309017382 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3300149d0 con 0x7fc328006050 2023-10-30T13:35:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.320+0000 7fc327fff640 1 -- 172.21.15.31:0/2309017382 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330014d80 con 0x7fc328006050 2023-10-30T13:35:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.324+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc340155d20 con 0x7fc328006050 2023-10-30T13:35:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.328+0000 7fc327fff640 1 -- 172.21.15.31:0/2309017382 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc330015410 con 0x7fc328006050 2023-10-30T13:35:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.328+0000 7fc327fff640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc32c0664d0 0x7fc32c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.328+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3401562e0 con 0x7fc328006050 2023-10-30T13:35:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.328+0000 7fc327fff640 1 -- 172.21.15.31:0/2309017382 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc330055180 con 0x7fc328006050 2023-10-30T13:35:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.328+0000 7fc33effd640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc32c0664d0 0x7fc32c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.328+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34004e170 con 0x7fc328006050 2023-10-30T13:35:18.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.340+0000 7fc33effd640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc32c0664d0 0x7fc32c068990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc34014ca00 tx=0x7fc338007660 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.344+0000 7fc327fff640 1 -- 172.21.15.31:0/2309017382 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc330019020 con 0x7fc328006050 2023-10-30T13:35:18.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.368+0000 7fb8db59e640 1 Processor -- start 2023-10-30T13:35:18.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.368+0000 7fb8db59e640 1 -- start start 2023-10-30T13:35:18.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7fb8db59e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0727c0 0x7fb8dc162e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7fb8db59e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8dc15b9b0 con 0x7fb8dc0723e0 2023-10-30T13:35:18.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7fb8db59e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8dc157ae0 con 0x7fb8dc0c7120 2023-10-30T13:35:18.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7fb8db59e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc157da0 con 0x7fb8dc0727c0 2023-10-30T13:35:18.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7fb8d1d9b640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fb8dc0c7120 0x7fb8dc1513e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45796/0 (socket says 172.21.15.31:45796) 2023-10-30T13:35:18.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7fb8d1d9b640 1 -- 172.21.15.31:0/3117341194 learned_addr learned my addr 172.21.15.31:0/3117341194 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:18.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7f76f589e640 1 Processor -- start 2023-10-30T13:35:18.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7f76f589e640 1 -- start start 2023-10-30T13:35:18.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.372+0000 7fb8dad9d640 1 --2- 172.21.15.31:0/3117341194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0727c0 0x7fb8dc162e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7f76f589e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f76f006c420 0x7f76f00c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7fb8dad9d640 1 -- 172.21.15.31:0/3117341194 >> v1:172.21.15.33:6789/0 conn(0x7fb8dc0c7120 legacy=0x7fb8dc1513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7fb8dad9d640 1 -- 172.21.15.31:0/3117341194 >> v1:172.21.15.31:6789/0 conn(0x7fb8dc0723e0 legacy=0x7fb8dc15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7fb8dad9d640 1 -- 172.21.15.31:0/3117341194 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc158080 con 0x7fb8dc0727c0 2023-10-30T13:35:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7f76f589e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f0071e00 con 0x7f76f006bb40 2023-10-30T13:35:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7f76f589e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f0070020 con 0x7f76f006b760 2023-10-30T13:35:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7f76f589e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f0070f10 con 0x7f76f006c420 2023-10-30T13:35:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7f76ee7fc640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f76f006bb40 0x7f76f0157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51914/0 (socket says 172.21.15.31:51914) 2023-10-30T13:35:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7f76ee7fc640 1 -- 172.21.15.31:0/985413330 learned_addr learned my addr 172.21.15.31:0/985413330 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:18.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.376+0000 7f76ef7fe640 1 --2- 172.21.15.31:0/985413330 >> v2:172.21.15.33:3300/0 conn(0x7f76f006c420 0x7f76f00c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7f76ef7fe640 1 -- 172.21.15.31:0/985413330 >> v1:172.21.15.33:6789/0 conn(0x7f76f006b760 legacy=0x7f76f015e650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:35:18.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7f76ef7fe640 1 -- 172.21.15.31:0/985413330 >> v1:172.21.15.31:6789/0 conn(0x7f76f006bb40 legacy=0x7f76f0157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7f76ef7fe640 1 -- 172.21.15.31:0/985413330 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00711f0 con 0x7f76f006c420 2023-10-30T13:35:18.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7fb8dad9d640 1 --2- 172.21.15.31:0/3117341194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0727c0 0x7fb8dc162e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40097f0 tx=0x7fb8d401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a856f01b261e280 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7fb8d9d9b640 1 -- 172.21.15.31:0/3117341194 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d40a8970 con 0x7fb8dc0727c0 2023-10-30T13:35:18.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7fb8d9d9b640 1 -- 172.21.15.31:0/3117341194 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d40a8ad0 con 0x7fb8dc0727c0 2023-10-30T13:35:18.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7fb8d9d9b640 1 -- 172.21.15.31:0/3117341194 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d40a8970 con 0x7fb8dc0727c0 2023-10-30T13:35:18.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7fb8db59e640 1 -- 172.21.15.31:0/3117341194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0727c0 msgr2=0x7fb8dc162e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7fb8db59e640 1 --2- 172.21.15.31:0/3117341194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0727c0 0x7fb8dc162e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40097f0 tx=0x7fb8d401ec90 comp rx=0 tx=0).stop 2023-10-30T13:35:18.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.380+0000 7fb8db59e640 1 -- 172.21.15.31:0/3117341194 shutdown_connections 2023-10-30T13:35:18.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7fb8db59e640 1 --2- 172.21.15.31:0/3117341194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0727c0 0x7fb8dc162e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7fb8db59e640 1 -- 172.21.15.31:0/3117341194 >> 172.21.15.31:0/3117341194 conn(0x7fb8dc0caae0 msgr2=0x7fb8dc0caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7fb8db59e640 1 -- 172.21.15.31:0/3117341194 shutdown_connections 2023-10-30T13:35:18.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7fb8db59e640 1 -- 172.21.15.31:0/3117341194 wait complete. 2023-10-30T13:35:18.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7fd5530dc640 1 Processor -- start 2023-10-30T13:35:18.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7fd5530dc640 1 -- start start 2023-10-30T13:35:18.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7f76ef7fe640 1 --2- 172.21.15.31:0/985413330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f00c57a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f76e00097c0 tx=0x7f76e001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85d2c9a10c1e005a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7f76edffb640 1 -- 172.21.15.31:0/985413330 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e00ab610 con 0x7f76f006c420 2023-10-30T13:35:18.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7f76edffb640 1 -- 172.21.15.31:0/985413330 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e00ab770 con 0x7f76f006c420 2023-10-30T13:35:18.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7f76edffb640 1 -- 172.21.15.31:0/985413330 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e00abb00 con 0x7f76f006c420 2023-10-30T13:35:18.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5530dc640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd54c0bc5b0 0x7fd54c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5530dc640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0bcdc0 con 0x7fd54c0be140 2023-10-30T13:35:18.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5530dc640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0bd0a0 con 0x7fd54c153a20 2023-10-30T13:35:18.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5530dc640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c0c5b70 con 0x7fd54c0bc5b0 2023-10-30T13:35:18.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5520da640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fd54c153a20 0x7fd54c15a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45818/0 (socket says 172.21.15.31:45818) 2023-10-30T13:35:18.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5520da640 1 -- 172.21.15.31:0/1605301702 learned_addr learned my addr 172.21.15.31:0/1605301702 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:18.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5528db640 1 --2- 172.21.15.31:0/1605301702 >> v2:172.21.15.33:3300/0 conn(0x7fd54c0bc5b0 0x7fd54c06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fb8db59e640 1 Processor -- start 2023-10-30T13:35:18.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fb8db59e640 1 -- start start 2023-10-30T13:35:18.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7f76f589e640 1 -- 172.21.15.31:0/985413330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 msgr2=0x7f76f00c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.384+0000 7f76f589e640 1 --2- 172.21.15.31:0/985413330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f00c57a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f76e00097c0 tx=0x7f76e001c410 comp rx=0 tx=0).stop 2023-10-30T13:35:18.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7f76f589e640 1 -- 172.21.15.31:0/985413330 shutdown_connections 2023-10-30T13:35:18.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7f76f589e640 1 --2- 172.21.15.31:0/985413330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f00c57a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7f76f589e640 1 -- 172.21.15.31:0/985413330 >> 172.21.15.31:0/985413330 conn(0x7f76f00cac70 msgr2=0x7f76f00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7f76f589e640 1 -- 172.21.15.31:0/985413330 shutdown_connections 2023-10-30T13:35:18.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7f76f589e640 1 -- 172.21.15.31:0/985413330 wait complete. 2023-10-30T13:35:18.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5510d8640 1 -- 172.21.15.31:0/1605301702 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296533533 0 0) 0x7fd54c0bcdc0 con 0x7fd54c0be140 2023-10-30T13:35:18.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5510d8640 1 -- 172.21.15.31:0/1605301702 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd538003530 con 0x7fd54c0be140 2023-10-30T13:35:18.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5528db640 1 -- 172.21.15.31:0/1605301702 >> v1:172.21.15.33:6789/0 conn(0x7fd54c153a20 legacy=0x7fd54c15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5528db640 1 -- 172.21.15.31:0/1605301702 >> v1:172.21.15.31:6789/0 conn(0x7fd54c0be140 legacy=0x7fd54c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fd5528db640 1 -- 172.21.15.31:0/1605301702 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0bd380 con 0x7fd54c0bc5b0 2023-10-30T13:35:18.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fbd4c846640 1 Processor -- start 2023-10-30T13:35:18.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.388+0000 7fbd4c846640 1 -- start start 2023-10-30T13:35:18.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8db59e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0723e0 0x7fb8dc1f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8db59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc0727c0 0x7fb8dc155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8db59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8dc0c7120 0x7fb8dc1f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8db59e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc1f6720 con 0x7fb8dc0c7120 2023-10-30T13:35:18.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8db59e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc14c1a0 con 0x7fb8dc0727c0 2023-10-30T13:35:18.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8db59e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0c90a0 con 0x7fb8dc0723e0 2023-10-30T13:35:18.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8da59c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0723e0 0x7fb8dc1f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8da59c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0723e0 0x7fb8dc1f6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53516/0 (socket says 172.21.15.31:53516) 2023-10-30T13:35:18.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8da59c640 1 -- 172.21.15.31:0/4187893223 learned_addr learned my addr 172.21.15.31:0/4187893223 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8dad9d640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8dc0c7120 0x7fb8dc1f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 Processor -- start 2023-10-30T13:35:18.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 -- start start 2023-10-30T13:35:18.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8dad9d640 1 -- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0723e0 msgr2=0x7fb8dc1f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8dad9d640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0723e0 0x7fb8dc1f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8dad9d640 1 -- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc0727c0 msgr2=0x7fb8dc155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8dad9d640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc0727c0 0x7fb8dc155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fbd4659c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fbd4806b760 0x7fbd4815e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45828/0 (socket says 172.21.15.31:45828) 2023-10-30T13:35:18.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8dad9d640 1 -- 172.21.15.31:0/4187893223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d4009420 con 0x7fb8dc0c7120 2023-10-30T13:35:18.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8da59c640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0723e0 0x7fb8dc1f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:18.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fbd4659c640 1 -- 172.21.15.31:0/623710571 learned_addr learned my addr 172.21.15.31:0/623710571 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:18.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fb8dad9d640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8dc0c7120 0x7fb8dc1f2ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40af000 tx=0x7fb8d401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f006b760 0x7f76f01616b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76f006bb40 0x7f76f015f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f0149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0160b70 con 0x7f76f006b760 2023-10-30T13:35:18.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0161170 con 0x7f76f006c420 2023-10-30T13:35:18.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7f76f589e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0148480 con 0x7f76f006bb40 2023-10-30T13:35:18.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f76ee7fc640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76f006bb40 0x7f76f015f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f76ee7fc640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76f006bb40 0x7f76f015f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53524/0 (socket says 172.21.15.31:53524) 2023-10-30T13:35:18.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f76ef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f0149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f76ef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f0149c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60890/0 (socket says 172.21.15.31:60890) 2023-10-30T13:35:18.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f76ef7fe640 1 -- 172.21.15.31:0/1953099236 learned_addr learned my addr 172.21.15.31:0/1953099236 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.392+0000 7fbd4c846640 1 --2- 172.21.15.31:0/623710571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd480c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f76eeffd640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f006b760 0x7f76f01616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd48071e00 con 0x7fbd4806bb40 2023-10-30T13:35:18.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd48070020 con 0x7fbd4806b760 2023-10-30T13:35:18.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48070f10 con 0x7fbd4806c420 2023-10-30T13:35:18.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fb8d37fe640 1 -- 172.21.15.31:0/4187893223 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4003620 con 0x7fb8dc0c7120 2023-10-30T13:35:18.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fb8d37fe640 1 -- 172.21.15.31:0/4187893223 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d4003780 con 0x7fb8dc0c7120 2023-10-30T13:35:18.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fb8d37fe640 1 -- 172.21.15.31:0/4187893223 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4003af0 con 0x7fb8dc0c7120 2023-10-30T13:35:18.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fb8db59e640 1 -- 172.21.15.31:0/4187893223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8dc0c9340 con 0x7fb8dc0c7120 2023-10-30T13:35:18.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fb8db59e640 1 -- 172.21.15.31:0/4187893223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc1f63b0 con 0x7fb8dc0c7120 2023-10-30T13:35:18.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd46d9d640 1 --2- 172.21.15.31:0/623710571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd480c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032200715 0 0) 0x7fbd48070020 con 0x7fbd4806b760 2023-10-30T13:35:18.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd30003530 con 0x7fbd4806b760 2023-10-30T13:35:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723196998 0 0) 0x7fbd48071e00 con 0x7fbd4806bb40 2023-10-30T13:35:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd48070020 con 0x7fbd4806bb40 2023-10-30T13:35:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fad1c046640 1 Processor -- start 2023-10-30T13:35:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f75cf59e640 1 Processor -- start 2023-10-30T13:35:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4022013015 0 0) 0x7fbd30003530 con 0x7fbd4806b760 2023-10-30T13:35:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd46d9d640 1 -- 172.21.15.31:0/623710571 >> v1:172.21.15.33:6789/0 conn(0x7fbd4806b760 legacy=0x7fbd4815e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd46d9d640 1 -- 172.21.15.31:0/623710571 >> v1:172.21.15.31:6789/0 conn(0x7fbd4806bb40 legacy=0x7fbd48157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fbd46d9d640 1 -- 172.21.15.31:0/623710571 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd480711f0 con 0x7fbd4806c420 2023-10-30T13:35:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fad1c046640 1 -- start start 2023-10-30T13:35:18.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7fd5528db640 1 --2- 172.21.15.31:0/1605301702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c0bc5b0 0x7fd54c06d5d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0097c0 tx=0x7fd53c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=682901e697d53337 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.396+0000 7f75cf59e640 1 -- start start 2023-10-30T13:35:18.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b4005260 con 0x7fb8dc0c7120 2023-10-30T13:35:18.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fd5510d8640 1 -- 172.21.15.31:0/1605301702 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0ab600 con 0x7fd54c0bc5b0 2023-10-30T13:35:18.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76ef7fe640 1 -- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76f006bb40 msgr2=0x7f76f015f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76ef7fe640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76f006bb40 0x7f76f015f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76ef7fe640 1 -- 172.21.15.31:0/1953099236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f006b760 msgr2=0x7f76f01616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76ef7fe640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f006b760 0x7f76f01616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76ef7fe640 1 -- 172.21.15.31:0/1953099236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e0009420 con 0x7f76f006c420 2023-10-30T13:35:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76ee7fc640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76f006bb40 0x7f76f015f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:18.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fd5510d8640 1 -- 172.21.15.31:0/1605301702 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c0a7050 con 0x7fd54c0bc5b0 2023-10-30T13:35:18.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fd5510d8640 1 -- 172.21.15.31:0/1605301702 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0ab980 con 0x7fd54c0bc5b0 2023-10-30T13:35:18.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fb8d37fe640 1 -- 172.21.15.31:0/4187893223 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8d40b7040 con 0x7fb8dc0c7120 2023-10-30T13:35:18.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75cf59e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d0153b50 0x7f75d01597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75cf59e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f75d00cb180 0x7f75d006ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76ef7fe640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f0149c20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f76e0019020 tx=0x7f76e001ac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75cdd9b640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f75d00c62b0 0x7f75d00ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45840/0 (socket says 172.21.15.31:45840) 2023-10-30T13:35:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75cf59e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d015a770 con 0x7f75d00c62b0 2023-10-30T13:35:18.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75cf59e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d0159cf0 con 0x7f75d00cb180 2023-10-30T13:35:18.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75cf59e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d006aa30 con 0x7f75d0153b50 2023-10-30T13:35:18.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75cdd9b640 1 -- 172.21.15.31:0/682711727 learned_addr learned my addr 172.21.15.31:0/682711727 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:18.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad1c046640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fad0c007320 0x7fad0c0df7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad1c046640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c0d92c0 0x7fad0c0e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad1c046640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0c04ef20 con 0x7fad0c007700 2023-10-30T13:35:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad1c046640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0c04fdf0 con 0x7fad0c007320 2023-10-30T13:35:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad1c046640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c037ef0 con 0x7fad0c0d92c0 2023-10-30T13:35:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f76d7fff640 1 -- 172.21.15.31:0/1953099236 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e00b5810 con 0x7f76f006c420 2023-10-30T13:35:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad19de2640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fad0c007320 0x7fad0c0df7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad19de2640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fad0c007320 0x7fad0c0df7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60898/0 (socket says 172.21.15.31:60898) 2023-10-30T13:35:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad1a5e3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c0d92c0 0x7fad0c0e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fad19de2640 1 -- 172.21.15.31:0/662946511 learned_addr learned my addr 172.21.15.31:0/662946511 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fb8d37fe640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8bc0664a0 0x7fb8bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fb8d37fe640 1 -- 172.21.15.31:0/4187893223 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8d4121830 con 0x7fb8dc0c7120 2023-10-30T13:35:18.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75ce59c640 1 --2- 172.21.15.31:0/682711727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d0153b50 0x7f75d01597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7f75ced9d640 1 --2- 172.21.15.31:0/682711727 >> v2:172.21.15.33:3300/0 conn(0x7f75d00cb180 0x7f75d006ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f76d7fff640 1 -- 172.21.15.31:0/1953099236 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e00b5970 con 0x7f76f006c420 2023-10-30T13:35:18.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f76d7fff640 1 -- 172.21.15.31:0/1953099236 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e00b5ce0 con 0x7f76f006c420 2023-10-30T13:35:18.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fb8da59c640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8bc0664a0 0x7fb8bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f76f589e640 1 -- 172.21.15.31:0/1953099236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f0148760 con 0x7f76f006c420 2023-10-30T13:35:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f75ce59c640 1 -- 172.21.15.31:0/682711727 >> v2:172.21.15.33:3300/0 conn(0x7f75d00cb180 msgr2=0x7f75d006ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f75ce59c640 1 --2- 172.21.15.31:0/682711727 >> v2:172.21.15.33:3300/0 conn(0x7f75d00cb180 0x7f75d006ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f75ce59c640 1 -- 172.21.15.31:0/682711727 >> v1:172.21.15.33:6789/0 conn(0x7f75d00c62b0 legacy=0x7f75d00ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f75ce59c640 1 -- 172.21.15.31:0/682711727 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00c9d40 con 0x7f75d0153b50 2023-10-30T13:35:18.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7f76f589e640 1 -- 172.21.15.31:0/1953099236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f0149140 con 0x7f76f006c420 2023-10-30T13:35:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fd5530dc640 1 -- 172.21.15.31:0/1605301702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c0bc5b0 msgr2=0x7fd54c06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.400+0000 7fd5530dc640 1 --2- 172.21.15.31:0/1605301702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c0bc5b0 0x7fd54c06d5d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0097c0 tx=0x7fd53c01c410 comp rx=0 tx=0).stop 2023-10-30T13:35:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fd5530dc640 1 -- 172.21.15.31:0/1605301702 shutdown_connections 2023-10-30T13:35:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fd5530dc640 1 --2- 172.21.15.31:0/1605301702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c0bc5b0 0x7fd54c06d5d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fd5530dc640 1 -- 172.21.15.31:0/1605301702 >> 172.21.15.31:0/1605301702 conn(0x7fd54c0c7560 msgr2=0x7fd54c0c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fa4cffff640 1 Processor -- start 2023-10-30T13:35:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fa4cffff640 1 -- start start 2023-10-30T13:35:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fd5530dc640 1 -- 172.21.15.31:0/1605301702 shutdown_connections 2023-10-30T13:35:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fd5530dc640 1 -- 172.21.15.31:0/1605301702 wait complete. 2023-10-30T13:35:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fbd46d9d640 1 --2- 172.21.15.31:0/623710571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd480c57a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009dc0 tx=0x7fbd4001c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2273028ecfb58b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38001680 con 0x7fbd4806c420 2023-10-30T13:35:18.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd400a56c0 con 0x7fbd4806c420 2023-10-30T13:35:18.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fbd4559a640 1 -- 172.21.15.31:0/623710571 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd400a5a50 con 0x7fbd4806c420 2023-10-30T13:35:18.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 Processor -- start 2023-10-30T13:35:18.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f0066510 con 0x7f76f006c420 2023-10-30T13:35:18.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f76d7fff640 1 -- 172.21.15.31:0/1953099236 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76e00b4390 con 0x7f76f006c420 2023-10-30T13:35:18.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 -- start start 2023-10-30T13:35:18.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fad1a5e3640 1 -- 172.21.15.31:0/662946511 >> v2:172.21.15.33:3300/0 conn(0x7fad0c007320 msgr2=0x7fad0c0df7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fad1a5e3640 1 --2- 172.21.15.31:0/662946511 >> v2:172.21.15.33:3300/0 conn(0x7fad0c007320 0x7fad0c0df7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fad1a5e3640 1 -- 172.21.15.31:0/662946511 >> v1:172.21.15.31:6789/0 conn(0x7fad0c007700 legacy=0x7fad0c0d8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fad1a5e3640 1 -- 172.21.15.31:0/662946511 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c04f170 con 0x7fad0c0d92c0 2023-10-30T13:35:18.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fb8da59c640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8bc0664a0 0x7fb8bc068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4004470 tx=0x7fb8c4003700 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fa4cffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa4d0156180 0x7fa4d014f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fa4cffff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d00c5b10 con 0x7fa4d006d4f0 2023-10-30T13:35:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fa4cffff640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d00c6a70 con 0x7fa4d0156180 2023-10-30T13:35:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.404+0000 7fa4cffff640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d00c5df0 con 0x7fa4d006fb90 2023-10-30T13:35:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4cf7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fa4d006d4f0 0x7fa4d0070430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45854/0 (socket says 172.21.15.31:45854) 2023-10-30T13:35:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4cf7fe640 1 -- 172.21.15.31:0/4200983803 learned_addr learned my addr 172.21.15.31:0/4200983803 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4ceffd640 1 --2- 172.21.15.31:0/4200983803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4c7fff640 1 --2- 172.21.15.31:0/4200983803 >> v2:172.21.15.31:3300/0 conn(0x7fa4d0156180 0x7fa4d014f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4c7fff640 1 -- 172.21.15.31:0/4200983803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 msgr2=0x7fa4d00c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd54c0bc5b0 0x7fd54c14ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4c7fff640 1 --2- 172.21.15.31:0/4200983803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d00c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4c7fff640 1 -- 172.21.15.31:0/4200983803 >> v1:172.21.15.33:6789/0 conn(0x7fa4d006d4f0 legacy=0x7fa4d0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fa4c7fff640 1 -- 172.21.15.31:0/4200983803 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d00c60d0 con 0x7fa4d0156180 2023-10-30T13:35:18.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c153a20 0x7fd54c14f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c155dc0 0x7fd54c156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c155930 con 0x7fd54c155dc0 2023-10-30T13:35:18.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c156790 con 0x7fd54c153a20 2023-10-30T13:35:18.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5530dc640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0c3890 con 0x7fd54c0bc5b0 2023-10-30T13:35:18.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 msgr2=0x7fbd480c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fbd4c846640 1 --2- 172.21.15.31:0/623710571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd480c57a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009dc0 tx=0x7fbd4001c880 comp rx=0 tx=0).stop 2023-10-30T13:35:18.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 shutdown_connections 2023-10-30T13:35:18.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fbd4c846640 1 --2- 172.21.15.31:0/623710571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd480c57a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f75ce59c640 1 --2- 172.21.15.31:0/682711727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d0153b50 0x7f75d01597b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75c00097c0 tx=0x7f75c001c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133b02d0417fd726 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 >> 172.21.15.31:0/623710571 conn(0x7fbd480cac70 msgr2=0x7fbd480cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f76d7fff640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f76dc0664a0 0x7f76dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fad1a5e3640 1 --2- 172.21.15.31:0/662946511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c0d92c0 0x7fad0c0e02a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad14069480 tx=0x7fad14099cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1db20679aec6ad61 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5520da640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd54c0bc5b0 0x7fd54c14ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c155dc0 0x7fd54c156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5520da640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd54c0bc5b0 0x7fd54c14ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53566/0 (socket says 172.21.15.31:53566) 2023-10-30T13:35:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5520da640 1 -- 172.21.15.31:0/3062022607 learned_addr learned my addr 172.21.15.31:0/3062022607 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c155dc0 0x7fd54c156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36146/0 (socket says 172.21.15.31:36146) 2023-10-30T13:35:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5518d9640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c153a20 0x7fd54c14f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 shutdown_connections 2023-10-30T13:35:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f75cd59a640 1 -- 172.21.15.31:0/682711727 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c00ab600 con 0x7f75d0153b50 2023-10-30T13:35:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f75cd59a640 1 -- 172.21.15.31:0/682711727 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c00a7050 con 0x7f75d0153b50 2023-10-30T13:35:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f75cd59a640 1 -- 172.21.15.31:0/682711727 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c00ab980 con 0x7f75d0153b50 2023-10-30T13:35:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fb8d37fe640 1 -- 172.21.15.31:0/4187893223 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8d40eed30 con 0x7fb8dc0c7120 2023-10-30T13:35:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fbd4c846640 1 -- 172.21.15.31:0/623710571 wait complete. 2023-10-30T13:35:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fad18de0640 1 -- 172.21.15.31:0/662946511 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14072040 con 0x7fad0c0d92c0 2023-10-30T13:35:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fad18de0640 1 -- 172.21.15.31:0/662946511 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad141239c0 con 0x7fad0c0d92c0 2023-10-30T13:35:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fad18de0640 1 -- 172.21.15.31:0/662946511 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14123df0 con 0x7fad0c0d92c0 2023-10-30T13:35:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 -- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd54c0bc5b0 msgr2=0x7fd54c14ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd54c0bc5b0 0x7fd54c14ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 -- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c153a20 msgr2=0x7fd54c14f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c153a20 0x7fd54c14f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 -- 172.21.15.31:0/3062022607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c009420 con 0x7fd54c155dc0 2023-10-30T13:35:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7fd5528db640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c155dc0 0x7fd54c156250 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0ab000 tx=0x7fd53c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fd542ffd640 1 -- 172.21.15.31:0/3062022607 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0b4400 con 0x7fd54c155dc0 2023-10-30T13:35:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fd542ffd640 1 -- 172.21.15.31:0/3062022607 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c0b4560 con 0x7fd54c155dc0 2023-10-30T13:35:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fd542ffd640 1 -- 172.21.15.31:0/3062022607 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0b48f0 con 0x7fd54c155dc0 2023-10-30T13:35:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 -- 172.21.15.31:0/662946511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c0d92c0 msgr2=0x7fad0c0e02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 --2- 172.21.15.31:0/662946511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c0d92c0 0x7fad0c0e02a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad14069480 tx=0x7fad14099cd0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f76d7fff640 1 -- 172.21.15.31:0/1953099236 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76e011e560 con 0x7f76f006c420 2023-10-30T13:35:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c0c3b70 con 0x7fd54c155dc0 2023-10-30T13:35:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 -- 172.21.15.31:0/662946511 shutdown_connections 2023-10-30T13:35:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 --2- 172.21.15.31:0/662946511 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c0d92c0 0x7fad0c0e02a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 --2- 172.21.15.31:0/662946511 >> v2:172.21.15.33:3300/0 conn(0x7fad0c007320 0x7fad0c0df7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 -- 172.21.15.31:0/662946511 >> 172.21.15.31:0/662946511 conn(0x7fad0c00e070 msgr2=0x7fad0c00e480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c1552e0 con 0x7fd54c155dc0 2023-10-30T13:35:18.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.408+0000 7f76eeffd640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f76dc0664a0 0x7f76dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 -- 172.21.15.31:0/662946511 shutdown_connections 2023-10-30T13:35:18.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7f75cf59e640 1 -- 172.21.15.31:0/682711727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d0153b50 msgr2=0x7f75d01597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7f75cf59e640 1 --2- 172.21.15.31:0/682711727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d0153b50 0x7f75d01597b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75c00097c0 tx=0x7f75c001c310 comp rx=0 tx=0).stop 2023-10-30T13:35:18.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7f75cf59e640 1 -- 172.21.15.31:0/682711727 shutdown_connections 2023-10-30T13:35:18.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7f75cf59e640 1 --2- 172.21.15.31:0/682711727 >> v2:172.21.15.33:3300/0 conn(0x7f75d00cb180 0x7f75d006ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7f75cf59e640 1 --2- 172.21.15.31:0/682711727 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d0153b50 0x7f75d01597b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7f75cf59e640 1 -- 172.21.15.31:0/682711727 >> 172.21.15.31:0/682711727 conn(0x7f75d0065a80 msgr2=0x7f75d0065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 -- 172.21.15.31:0/662946511 wait complete. 2023-10-30T13:35:18.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c155aa0 con 0x7fd54c155dc0 2023-10-30T13:35:18.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fbd4c846640 1 Processor -- start 2023-10-30T13:35:18.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fbd4c846640 1 -- start start 2023-10-30T13:35:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 Processor -- start 2023-10-30T13:35:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7f76eeffd640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f76dc0664a0 0x7f76dc068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009fd0 tx=0x7f76e800d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fad1c046640 1 -- start start 2023-10-30T13:35:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fbd4c846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd4806b760 0x7fbd481f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.412+0000 7fbd4c846640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd4806bb40 0x7fbd4814be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4c846640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd48149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4c846640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd481f7070 con 0x7fbd4806b760 2023-10-30T13:35:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4c846640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd481491b0 con 0x7fbd4806bb40 2023-10-30T13:35:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4c846640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd480c8f00 con 0x7fbd4806c420 2023-10-30T13:35:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fd542ffd640 1 -- 172.21.15.31:0/3062022607 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd53c0b2040 con 0x7fd54c155dc0 2023-10-30T13:35:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd4806b760 0x7fbd481f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd4806b760 0x7fbd481f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36148/0 (socket says 172.21.15.31:36148) 2023-10-30T13:35:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 -- 172.21.15.31:0/3008302318 learned_addr learned my addr 172.21.15.31:0/3008302318 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 -- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 msgr2=0x7fbd48149660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T13:35:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd48149660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 -- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd4806bb40 msgr2=0x7fbd4814be30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T13:35:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd4806bb40 0x7fbd4814be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 -- 172.21.15.31:0/3008302318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40009a70 con 0x7fbd4806b760 2023-10-30T13:35:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fbd4659c640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd4806b760 0x7fbd481f6b30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c750 tx=0x7fbd3800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fa4c7fff640 1 --2- 172.21.15.31:0/4200983803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 0x7fa4d014f830 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80097c0 tx=0x7fa4b802d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b047b345a8abf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad1c046640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad0c007320 0x7fad0c0d7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fd542ffd640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd530066520 0x7fd5300689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fd542ffd640 1 -- 172.21.15.31:0/3062022607 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd53c11e170 con 0x7fd54c155dc0 2023-10-30T13:35:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad1c046640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c007700 0x7fad0c175c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad1c046640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad0c0d92c0 0x7fad0c176150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad1c046640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c04baf0 con 0x7fad0c0d92c0 2023-10-30T13:35:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad1c046640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c0d8660 con 0x7fad0c007320 2023-10-30T13:35:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad1c046640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c0d6570 con 0x7fad0c007700 2023-10-30T13:35:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fd5520da640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd530066520 0x7fd5300689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad0c007320 0x7fad0c0d7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad0c007320 0x7fad0c0d7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60936/0 (socket says 172.21.15.31:60936) 2023-10-30T13:35:18.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 -- 172.21.15.31:0/567368463 learned_addr learned my addr 172.21.15.31:0/567368463 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad1a5e3640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad0c0d92c0 0x7fad0c176150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad195e1640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c007700 0x7fad0c175c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7f75cf59e640 1 -- 172.21.15.31:0/682711727 shutdown_connections 2023-10-30T13:35:18.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7f75cf59e640 1 -- 172.21.15.31:0/682711727 wait complete. 2023-10-30T13:35:18.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fd5520da640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd530066520 0x7fd5300689e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd544001f60 tx=0x7fd544001ad0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 -- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c007700 msgr2=0x7fad0c175c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c007700 0x7fad0c175c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 -- 172.21.15.31:0/567368463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad0c0d92c0 msgr2=0x7fad0c176150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad0c0d92c0 0x7fad0c176150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 -- 172.21.15.31:0/567368463 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c0d6850 con 0x7fad0c007320 2023-10-30T13:35:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad195e1640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c007700 0x7fad0c175c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.416+0000 7fad19de2640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad0c007320 0x7fad0c0d7fb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad1000e7f0 tx=0x7fad1000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fbd36ffd640 1 -- 172.21.15.31:0/3008302318 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380072e0 con 0x7fbd4806b760 2023-10-30T13:35:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd480c91e0 con 0x7fbd4806b760 2023-10-30T13:35:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4ce7fc640 1 -- 172.21.15.31:0/4200983803 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80bd410 con 0x7fa4d0156180 2023-10-30T13:35:18.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd480c97a0 con 0x7fbd4806b760 2023-10-30T13:35:18.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fbd36ffd640 1 -- 172.21.15.31:0/3008302318 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd38007440 con 0x7fbd4806b760 2023-10-30T13:35:18.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fbd36ffd640 1 -- 172.21.15.31:0/3008302318 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380047f0 con 0x7fbd4806b760 2023-10-30T13:35:18.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fd542ffd640 1 -- 172.21.15.31:0/3062022607 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd53c0eb5d0 con 0x7fd54c155dc0 2023-10-30T13:35:18.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7f75cf59e640 1 Processor -- start 2023-10-30T13:35:18.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7f75cf59e640 1 -- start start 2023-10-30T13:35:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4ce7fc640 1 -- 172.21.15.31:0/4200983803 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b80bd570 con 0x7fa4d0156180 2023-10-30T13:35:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4ce7fc640 1 -- 172.21.15.31:0/4200983803 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80bd900 con 0x7fa4d0156180 2023-10-30T13:35:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 -- 172.21.15.31:0/4200983803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 msgr2=0x7fa4d014f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4200983803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 0x7fa4d014f830 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80097c0 tx=0x7fa4b802d340 comp rx=0 tx=0).stop 2023-10-30T13:35:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c005260 con 0x7fbd4806b760 2023-10-30T13:35:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 -- 172.21.15.31:0/4200983803 shutdown_connections 2023-10-30T13:35:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4200983803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 0x7fa4d014f830 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4200983803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d00c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 -- 172.21.15.31:0/4200983803 >> 172.21.15.31:0/4200983803 conn(0x7fa4d00caae0 msgr2=0x7fa4d00caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 -- 172.21.15.31:0/4200983803 shutdown_connections 2023-10-30T13:35:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fa4cffff640 1 -- 172.21.15.31:0/4200983803 wait complete. 2023-10-30T13:35:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7fad0a7fc640 1 -- 172.21.15.31:0/567368463 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1000c940 con 0x7fad0c007320 2023-10-30T13:35:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7f75cf59e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d00c62b0 0x7f75d01f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7f75cf59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75d00cb180 0x7f75d01589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.420+0000 7f75cf59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75d0153b50 0x7f75d01ef220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fad0a7fc640 1 -- 172.21.15.31:0/567368463 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1000caa0 con 0x7fad0c007320 2023-10-30T13:35:18.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75cf59e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d01eedd0 con 0x7f75d0153b50 2023-10-30T13:35:18.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75cf59e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d014cbc0 con 0x7f75d00cb180 2023-10-30T13:35:18.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75cf59e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d014c040 con 0x7f75d00c62b0 2023-10-30T13:35:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0c0d6b30 con 0x7fad0c007320 2023-10-30T13:35:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0c0d70f0 con 0x7fad0c007320 2023-10-30T13:35:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f76d7fff640 1 -- 172.21.15.31:0/1953099236 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76e00eb9f0 con 0x7f76f006c420 2023-10-30T13:35:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ce59c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d00c62b0 0x7f75d01f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fad0a7fc640 1 -- 172.21.15.31:0/567368463 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1000ce30 con 0x7fad0c007320 2023-10-30T13:35:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75d0153b50 0x7f75d01ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75d0153b50 0x7f75d01ef220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36160/0 (socket says 172.21.15.31:36160) 2023-10-30T13:35:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ced9d640 1 -- 172.21.15.31:0/1863345911 learned_addr learned my addr 172.21.15.31:0/1863345911 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75cdd9b640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75d00cb180 0x7f75d01589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fbd36ffd640 1 -- 172.21.15.31:0/3008302318 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd38004950 con 0x7fbd4806b760 2023-10-30T13:35:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fa4cffff640 1 Processor -- start 2023-10-30T13:35:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fa4cffff640 1 -- start start 2023-10-30T13:35:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fbd36ffd640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbd280664a0 0x7fbd28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fbd36ffd640 1 -- 172.21.15.31:0/3008302318 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd38083650 con 0x7fbd4806b760 2023-10-30T13:35:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fbd45d9b640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbd280664a0 0x7fbd28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facfc005260 con 0x7fad0c007320 2023-10-30T13:35:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d006d4f0 0x7fa4d00c3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fbd45d9b640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbd280664a0 0x7fbd28068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd481575d0 tx=0x7fbd3c0078b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d01eab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 0x7fa4d0158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cffff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0158c60 con 0x7fa4d0156180 2023-10-30T13:35:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cffff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01ea430 con 0x7fa4d006d4f0 2023-10-30T13:35:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ce59c640 1 -- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75d00cb180 msgr2=0x7f75d01589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ce59c640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75d00cb180 0x7f75d01589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ce59c640 1 -- 172.21.15.31:0/1863345911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75d0153b50 msgr2=0x7f75d01ef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ce59c640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75d0153b50 0x7f75d01ef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.424+0000 7f75ce59c640 1 -- 172.21.15.31:0/1863345911 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c0009420 con 0x7f75d00c62b0 2023-10-30T13:35:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7f75ce59c640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d00c62b0 0x7f75d01f2b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75c00ab000 tx=0x7f75c00190b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cffff640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0140e90 con 0x7fa4d006fb90 2023-10-30T13:35:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d006d4f0 0x7fa4d00c3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d006d4f0 0x7fa4d00c3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60954/0 (socket says 172.21.15.31:60954) 2023-10-30T13:35:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 -- 172.21.15.31:0/4250964737 learned_addr learned my addr 172.21.15.31:0/4250964737 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4c7fff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d01eab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cf7fe640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 0x7fa4d0158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 -- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 msgr2=0x7fa4d01eab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7f75beffd640 1 -- 172.21.15.31:0/1863345911 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c00b46c0 con 0x7f75d00c62b0 2023-10-30T13:35:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7f75beffd640 1 -- 172.21.15.31:0/1863345911 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c00b4840 con 0x7f75d00c62b0 2023-10-30T13:35:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7f75beffd640 1 -- 172.21.15.31:0/1863345911 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c00b4bf0 con 0x7f75d00c62b0 2023-10-30T13:35:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d014c320 con 0x7f75d00c62b0 2023-10-30T13:35:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d014c8e0 con 0x7f75d00c62b0 2023-10-30T13:35:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d01eab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 -- 172.21.15.31:0/4250964737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 msgr2=0x7fa4d0158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 0x7fa4d0158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 -- 172.21.15.31:0/4250964737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b8009420 con 0x7fa4d006d4f0 2023-10-30T13:35:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4ceffd640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d006d4f0 0x7fa4d00c3030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000e960 tx=0x7fa4c000ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fad0a7fc640 1 -- 172.21.15.31:0/567368463 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad10013470 con 0x7fad0c007320 2023-10-30T13:35:18.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4c77fe640 1 -- 172.21.15.31:0/4250964737 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0009690 con 0x7fa4d006d4f0 2023-10-30T13:35:18.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d01ea6d0 con 0x7fa4d006d4f0 2023-10-30T13:35:18.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4c77fe640 1 -- 172.21.15.31:0/4250964737 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c000ca00 con 0x7fa4d006d4f0 2023-10-30T13:35:18.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d01ee0c0 con 0x7fa4d006d4f0 2023-10-30T13:35:18.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d01eeba0 con 0x7f75d00c62b0 2023-10-30T13:35:18.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fa4c77fe640 1 -- 172.21.15.31:0/4250964737 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0009690 con 0x7fa4d006d4f0 2023-10-30T13:35:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fad0a7fc640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facf80664f0 0x7facf80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.428+0000 7fad0a7fc640 1 -- 172.21.15.31:0/567368463 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad10083dd0 con 0x7fad0c007320 2023-10-30T13:35:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.432+0000 7fad195e1640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facf80664f0 0x7facf80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.432+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d004e170 con 0x7fa4d006d4f0 2023-10-30T13:35:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.432+0000 7fad195e1640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facf80664f0 0x7facf80689b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad04007fd0 tx=0x7fad04009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.432+0000 7f75beffd640 1 -- 172.21.15.31:0/1863345911 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75c00b2040 con 0x7f75d00c62b0 2023-10-30T13:35:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.432+0000 7fbd36ffd640 1 -- 172.21.15.31:0/3008302318 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd38050a50 con 0x7fbd4806b760 2023-10-30T13:35:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.436+0000 7f75beffd640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75b00664f0 0x7f75b00689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.436+0000 7f75beffd640 1 -- 172.21.15.31:0/1863345911 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75c011e630 con 0x7f75d00c62b0 2023-10-30T13:35:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.436+0000 7f75cdd9b640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75b00664f0 0x7f75b00689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.436+0000 7f75cdd9b640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75b00664f0 0x7f75b00689b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f75d014a830 tx=0x7f75c400aed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.440+0000 7fad0a7fc640 1 -- 172.21.15.31:0/567368463 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad10051360 con 0x7fad0c007320 2023-10-30T13:35:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.444+0000 7fa4c77fe640 1 -- 172.21.15.31:0/4250964737 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4c000cb70 con 0x7fa4d006d4f0 2023-10-30T13:35:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.444+0000 7f75beffd640 1 -- 172.21.15.31:0/1863345911 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75c00ebab0 con 0x7f75d00c62b0 2023-10-30T13:35:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.444+0000 7fa4c77fe640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4b0066500 0x7fa4b00689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.444+0000 7fa4c77fe640 1 -- 172.21.15.31:0/4250964737 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4c0084540 con 0x7fa4d006d4f0 2023-10-30T13:35:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.448+0000 7fa4c7fff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4b0066500 0x7fa4b00689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.452+0000 7fa4c7fff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4b0066500 0x7fa4b00689c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80bf000 tx=0x7fa4b80b7040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.456+0000 7fa4c77fe640 1 -- 172.21.15.31:0/4250964737 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4c00519d0 con 0x7fa4d006d4f0 2023-10-30T13:35:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.680+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc3400c9230 con 0x7fc328006050 2023-10-30T13:35:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.680+0000 7fc327fff640 1 -- 172.21.15.31:0/2309017382 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc330019020 con 0x7fc328006050 2023-10-30T13:35:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:60129542153 2023-10-30T13:35:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc32c0664d0 msgr2=0x7fc32c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc32c0664d0 0x7fc32c068990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc34014ca00 tx=0x7fc338007660 comp rx=0 tx=0).stop 2023-10-30T13:35:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328006050 msgr2=0x7fc3401ea6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328006050 0x7fc3401ea6e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc330006d50 tx=0x7fc33000f920 comp rx=0 tx=0).stop 2023-10-30T13:35:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 shutdown_connections 2023-10-30T13:35:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc32c0664d0 0x7fc32c068990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc34014a9a0 0x7fc3400c13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc34014cae0 0x7fc3401504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 --2- 172.21.15.31:0/2309017382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc328006050 0x7fc3401ea6e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 >> 172.21.15.31:0/2309017382 conn(0x7fc34006d810 msgr2=0x7fc3400c9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 shutdown_connections 2023-10-30T13:35:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.684+0000 7fc345bb8640 1 -- 172.21.15.31:0/2309017382 wait complete. 2023-10-30T13:35:18.720 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.0 2023-10-30T13:35:18.721 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.736+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f76f004e170 con 0x7f76f006c420 2023-10-30T13:35:18.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d7fff640 1 -- 172.21.15.31:0/1953099236 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f76e00ab5d0 con 0x7f76f006c420 2023-10-30T13:35:18.743 INFO:teuthology.orchestra.run.smithi031.stdout:60129542151 2023-10-30T13:35:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f76dc0664a0 msgr2=0x7f76dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f76dc0664a0 0x7f76dc068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009fd0 tx=0x7f76e800d040 comp rx=0 tx=0).stop 2023-10-30T13:35:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 msgr2=0x7f76f0149c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f0149c20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f76e0019020 tx=0x7f76e001ac50 comp rx=0 tx=0).stop 2023-10-30T13:35:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 shutdown_connections 2023-10-30T13:35:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f76dc0664a0 0x7f76dc068960 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76f006c420 0x7f76f0149c20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76f006bb40 0x7f76f015f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 --2- 172.21.15.31:0/1953099236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f006b760 0x7f76f01616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 >> 172.21.15.31:0/1953099236 conn(0x7f76f00cac70 msgr2=0x7f76f0153320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 shutdown_connections 2023-10-30T13:35:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.740+0000 7f76d5ffb640 1 -- 172.21.15.31:0/1953099236 wait complete. 2023-10-30T13:35:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.748+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb8b4005510 con 0x7fb8dc0c7120 2023-10-30T13:35:18.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.748+0000 7fb8d37fe640 1 -- 172.21.15.31:0/4187893223 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8d40f2350 con 0x7fb8dc0c7120 2023-10-30T13:35:18.755 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2023-10-30T13:35:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8bc0664a0 msgr2=0x7fb8bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8bc0664a0 0x7fb8bc068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4004470 tx=0x7fb8c4003700 comp rx=0 tx=0).stop 2023-10-30T13:35:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8dc0c7120 msgr2=0x7fb8dc1f2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8dc0c7120 0x7fb8dc1f2ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40af000 tx=0x7fb8d401ba30 comp rx=0 tx=0).stop 2023-10-30T13:35:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 shutdown_connections 2023-10-30T13:35:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8bc0664a0 0x7fb8bc068960 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8dc0c7120 0x7fb8dc1f2ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc0727c0 0x7fb8dc155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 --2- 172.21.15.31:0/4187893223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8dc0723e0 0x7fb8dc1f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.752+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 >> 172.21.15.31:0/4187893223 conn(0x7fb8dc0caae0 msgr2=0x7fb8dc06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.756+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 shutdown_connections 2023-10-30T13:35:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.756+0000 7fb8d0d99640 1 -- 172.21.15.31:0/4187893223 wait complete. 2023-10-30T13:35:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.756+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd54c04e170 con 0x7fd54c155dc0 2023-10-30T13:35:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.756+0000 7fd542ffd640 1 -- 172.21.15.31:0/3062022607 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd53c0b0090 con 0x7fd54c155dc0 2023-10-30T13:35:18.762 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2023-10-30T13:35:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd530066520 msgr2=0x7fd5300689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd530066520 0x7fd5300689e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd544001f60 tx=0x7fd544001ad0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c155dc0 msgr2=0x7fd54c156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c155dc0 0x7fd54c156250 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0ab000 tx=0x7fd53c0191b0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 shutdown_connections 2023-10-30T13:35:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd530066520 0x7fd5300689e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c155dc0 0x7fd54c156250 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd54c153a20 0x7fd54c14f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 --2- 172.21.15.31:0/3062022607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd54c0bc5b0 0x7fd54c14ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.760+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 >> 172.21.15.31:0/3062022607 conn(0x7fd54c0c7560 msgr2=0x7fd54c0c5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.764+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 shutdown_connections 2023-10-30T13:35:18.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.764+0000 7fd5530dc640 1 -- 172.21.15.31:0/3062022607 wait complete. 2023-10-30T13:35:18.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.764+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbd2c0050d0 con 0x7fbd4806b760 2023-10-30T13:35:18.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.768+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7facfc0057e0 con 0x7fad0c007320 2023-10-30T13:35:18.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.768+0000 7fbd36ffd640 1 -- 172.21.15.31:0/3008302318 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd38018020 con 0x7fbd4806b760 2023-10-30T13:35:18.773 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2023-10-30T13:35:18.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fad0a7fc640 1 -- 172.21.15.31:0/567368463 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fad10017020 con 0x7fad0c007320 2023-10-30T13:35:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbd280664a0 msgr2=0x7fbd28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbd280664a0 0x7fbd28068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd481575d0 tx=0x7fbd3c0078b0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd4806b760 msgr2=0x7fbd481f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd4806b760 0x7fbd481f6b30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c750 tx=0x7fbd3800cc10 comp rx=0 tx=0).stop 2023-10-30T13:35:18.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 shutdown_connections 2023-10-30T13:35:18.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbd280664a0 0x7fbd28068960 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbd4806c420 0x7fbd48149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd4806bb40 0x7fbd4814be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 --2- 172.21.15.31:0/3008302318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd4806b760 0x7fbd481f6b30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 >> 172.21.15.31:0/3008302318 conn(0x7fbd480cac70 msgr2=0x7fbd48157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 shutdown_connections 2023-10-30T13:35:18.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7fbd4c846640 1 -- 172.21.15.31:0/3008302318 wait complete. 2023-10-30T13:35:18.780 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2023-10-30T13:35:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f75d01eeba0 con 0x7f75d00c62b0 2023-10-30T13:35:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.772+0000 7f75beffd640 1 -- 172.21.15.31:0/1863345911 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f75d01eeba0 con 0x7f75d00c62b0 2023-10-30T13:35:18.781 INFO:teuthology.orchestra.run.smithi031.stdout:60129542151 2023-10-30T13:35:18.782 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2023-10-30T13:35:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facf80664f0 msgr2=0x7facf80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facf80664f0 0x7facf80689b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad04007fd0 tx=0x7fad04009040 comp rx=0 tx=0).stop 2023-10-30T13:35:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad0c007320 msgr2=0x7fad0c0d7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad0c007320 0x7fad0c0d7fb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad1000e7f0 tx=0x7fad1000ecb0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 shutdown_connections 2023-10-30T13:35:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facf80664f0 0x7facf80689b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad0c0d92c0 0x7fad0c176150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fad0c007700 0x7fad0c175c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 --2- 172.21.15.31:0/567368463 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad0c007320 0x7fad0c0d7fb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 >> 172.21.15.31:0/567368463 conn(0x7fad0c00e070 msgr2=0x7fad0c0e1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.785 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75b00664f0 msgr2=0x7f75b00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75b00664f0 0x7f75b00689b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f75d014a830 tx=0x7f75c400aed0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d00c62b0 msgr2=0x7f75d01f2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d00c62b0 0x7f75d01f2b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75c00ab000 tx=0x7f75c00190b0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 shutdown_connections 2023-10-30T13:35:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75b00664f0 0x7f75b00689b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75d0153b50 0x7f75d01ef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75d00cb180 0x7f75d01589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 --2- 172.21.15.31:0/1863345911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75d00c62b0 0x7f75d01f2b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 >> 172.21.15.31:0/1863345911 conn(0x7f75d0065a80 msgr2=0x7f75d014e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 shutdown_connections 2023-10-30T13:35:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.776+0000 7fad1c046640 1 -- 172.21.15.31:0/567368463 wait complete. 2023-10-30T13:35:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.780+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 shutdown_connections 2023-10-30T13:35:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.780+0000 7f75cf59e640 1 -- 172.21.15.31:0/1863345911 wait complete. 2023-10-30T13:35:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.780+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa4d006e4b0 con 0x7fa4d006d4f0 2023-10-30T13:35:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.784+0000 7fa4c77fe640 1 -- 172.21.15.31:0/4250964737 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa4c0017020 con 0x7fa4d006d4f0 2023-10-30T13:35:18.790 INFO:teuthology.orchestra.run.smithi031.stdout:60129542153 2023-10-30T13:35:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4b0066500 msgr2=0x7fa4b00689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4b0066500 0x7fa4b00689c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80bf000 tx=0x7fa4b80b7040 comp rx=0 tx=0).stop 2023-10-30T13:35:18.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d006d4f0 msgr2=0x7fa4d00c3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d006d4f0 0x7fa4d00c3030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000e960 tx=0x7fa4c000ee20 comp rx=0 tx=0).stop 2023-10-30T13:35:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 shutdown_connections 2023-10-30T13:35:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4b0066500 0x7fa4b00689c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4d0156180 0x7fa4d0158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4d006fb90 0x7fa4d01eab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 --2- 172.21.15.31:0/4250964737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d006d4f0 0x7fa4d00c3030 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 >> 172.21.15.31:0/4250964737 conn(0x7fa4d00caae0 msgr2=0x7fa4d0070430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 shutdown_connections 2023-10-30T13:35:18.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:18.788+0000 7fa4cffff640 1 -- 172.21.15.31:0/4250964737 wait complete. 2023-10-30T13:35:18.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.2 2023-10-30T13:35:18.794 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.803 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.1 2023-10-30T13:35:18.804 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.804 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.7 2023-10-30T13:35:18.805 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2023-10-30T13:35:18.814 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.823 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.3 2023-10-30T13:35:18.823 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.6 2023-10-30T13:35:18.826 DEBUG:teuthology.parallel:result is None 2023-10-30T13:35:18.826 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T13:35:18.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:35:19.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfbd0c9640 1 Processor -- start 2023-10-30T13:35:19.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfbd0c9640 1 -- start start 2023-10-30T13:35:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfbd0c9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fdfb80c4a10 0x7fdfb815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfbd0c9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fdfb80c42a0 0x7fdfb80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfbd0c9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb8159f20 con 0x7fdfb8153c80 2023-10-30T13:35:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfbd0c9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb815dec0 con 0x7fdfb80c42a0 2023-10-30T13:35:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfbd0c9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb81599a0 con 0x7fdfb80c4a10 2023-10-30T13:35:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb6d9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fdfb8153c80 0x7fdfb815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51960/0 (socket says 172.21.15.31:51960) 2023-10-30T13:35:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb6d9d640 1 -- 172.21.15.31:0/3901231362 learned_addr learned my addr 172.21.15.31:0/3901231362 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:19.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb759e640 1 --2- 172.21.15.31:0/3901231362 >> v2:172.21.15.31:3300/0 conn(0x7fdfb80c42a0 0x7fdfb80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfaffff640 1 --2- 172.21.15.31:0/3901231362 >> v2:172.21.15.33:3300/0 conn(0x7fdfb80c4a10 0x7fdfb815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb659c640 1 -- 172.21.15.31:0/3901231362 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535312243 0 0) 0x7fdfb8159f20 con 0x7fdfb8153c80 2023-10-30T13:35:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb659c640 1 -- 172.21.15.31:0/3901231362 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa4003530 con 0x7fdfb8153c80 2023-10-30T13:35:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb759e640 1 -- 172.21.15.31:0/3901231362 >> v2:172.21.15.33:3300/0 conn(0x7fdfb80c4a10 msgr2=0x7fdfb815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb759e640 1 --2- 172.21.15.31:0/3901231362 >> v2:172.21.15.33:3300/0 conn(0x7fdfb80c4a10 0x7fdfb815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb759e640 1 -- 172.21.15.31:0/3901231362 >> v1:172.21.15.31:6789/0 conn(0x7fdfb8153c80 legacy=0x7fdfb815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.008+0000 7fdfb759e640 1 -- 172.21.15.31:0/3901231362 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb815d620 con 0x7fdfb80c42a0 2023-10-30T13:35:19.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfb759e640 1 --2- 172.21.15.31:0/3901231362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c42a0 0x7fdfb80ca1e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdfb806be50 tx=0x7fdfb001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7064aeba0319e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfb659c640 1 -- 172.21.15.31:0/3901231362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb00b0440 con 0x7fdfb80c42a0 2023-10-30T13:35:19.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfb659c640 1 -- 172.21.15.31:0/3901231362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb00b05a0 con 0x7fdfb80c42a0 2023-10-30T13:35:19.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfb659c640 1 -- 172.21.15.31:0/3901231362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb00b0930 con 0x7fdfb80c42a0 2023-10-30T13:35:19.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/3901231362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c42a0 msgr2=0x7fdfb80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/3901231362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c42a0 0x7fdfb80ca1e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdfb806be50 tx=0x7fdfb001f0f0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/3901231362 shutdown_connections 2023-10-30T13:35:19.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/3901231362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c42a0 0x7fdfb80ca1e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/3901231362 >> v2:172.21.15.33:3300/0 conn(0x7fdfb80c4a10 0x7fdfb815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/3901231362 >> 172.21.15.31:0/3901231362 conn(0x7fdfb80715c0 msgr2=0x7fdfb80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:19.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/3901231362 shutdown_connections 2023-10-30T13:35:19.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.012+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/3901231362 wait complete. 2023-10-30T13:35:19.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 Processor -- start 2023-10-30T13:35:19.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 -- start start 2023-10-30T13:35:19.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdfb80c42a0 0x7fdfb81eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c4a10 0x7fdfb80ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdfb8153c80 0x7fdfb81efee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb80ce350 con 0x7fdfb80c4a10 2023-10-30T13:35:19.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb81574e0 con 0x7fdfb80c42a0 2023-10-30T13:35:19.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb80cd3f0 con 0x7fdfb8153c80 2023-10-30T13:35:19.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c4a10 0x7fdfb80ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c4a10 0x7fdfb80ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36170/0 (socket says 172.21.15.31:36170) 2023-10-30T13:35:19.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 -- 172.21.15.31:0/470206624 learned_addr learned my addr 172.21.15.31:0/470206624 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:19.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfb6d9d640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdfb80c42a0 0x7fdfb81eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfb759e640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdfb8153c80 0x7fdfb81efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 -- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdfb8153c80 msgr2=0x7fdfb81efee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdfb8153c80 0x7fdfb81efee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 -- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdfb80c42a0 msgr2=0x7fdfb81eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdfb80c42a0 0x7fdfb81eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 -- 172.21.15.31:0/470206624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0009a70 con 0x7fdfb80c4a10 2023-10-30T13:35:19.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfb759e640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdfb8153c80 0x7fdfb81efee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:19.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfb6d9d640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdfb80c42a0 0x7fdfb81eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:19.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaffff640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c4a10 0x7fdfb80ce770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000d800 tx=0x7fdfa000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaf7fe640 1 -- 172.21.15.31:0/470206624 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa000b6c0 con 0x7fdfb80c4a10 2023-10-30T13:35:19.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaf7fe640 1 -- 172.21.15.31:0/470206624 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa000b820 con 0x7fdfb80c4a10 2023-10-30T13:35:19.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfaf7fe640 1 -- 172.21.15.31:0/470206624 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa000bbd0 con 0x7fdfb80c4a10 2023-10-30T13:35:19.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb80cd6d0 con 0x7fdfb80c4a10 2023-10-30T13:35:19.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.016+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb80cdc90 con 0x7fdfb80c4a10 2023-10-30T13:35:19.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.020+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb804e170 con 0x7fdfb80c4a10 2023-10-30T13:35:19.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.020+0000 7fdfaf7fe640 1 -- 172.21.15.31:0/470206624 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdfa000bd30 con 0x7fdfb80c4a10 2023-10-30T13:35:19.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.020+0000 7fdfaf7fe640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf980664a0 0x7fdf98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.020+0000 7fdfaf7fe640 1 -- 172.21.15.31:0/470206624 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfa0084450 con 0x7fdfb80c4a10 2023-10-30T13:35:19.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.020+0000 7fdfb6d9d640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf980664a0 0x7fdf98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.024+0000 7fdfb6d9d640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf980664a0 0x7fdf98068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80011f0 tx=0x7fdfa8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.024+0000 7fdfaf7fe640 1 -- 172.21.15.31:0/470206624 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdfa0051830 con 0x7fdfb80c4a10 2023-10-30T13:35:19.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.180+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdfb815dbe0 con 0x7fdf980664a0 2023-10-30T13:35:19.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.200+0000 7fdfaf7fe640 1 -- 172.21.15.31:0/470206624 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264444 (secure 0 0 0) 0x7fdfa80013e0 con 0x7fdf980664a0 2023-10-30T13:35:19.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:19.208 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:35:19.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf980664a0 msgr2=0x7fdf98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf980664a0 0x7fdf98068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80011f0 tx=0x7fdfa8009040 comp rx=0 tx=0).stop 2023-10-30T13:35:19.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c4a10 msgr2=0x7fdfb80ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c4a10 0x7fdfb80ce770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000d800 tx=0x7fdfa000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 shutdown_connections 2023-10-30T13:35:19.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf980664a0 0x7fdf98068960 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdfb8153c80 0x7fdfb81efee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfb80c4a10 0x7fdfb80ce770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 --2- 172.21.15.31:0/470206624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdfb80c42a0 0x7fdfb81eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 >> 172.21.15.31:0/470206624 conn(0x7fdfb80715c0 msgr2=0x7fdfb806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:19.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 shutdown_connections 2023-10-30T13:35:19.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.204+0000 7fdfbd0c9640 1 -- 172.21.15.31:0/470206624 wait complete. 2023-10-30T13:35:19.227 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-30T13:35:19.159549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2924544,"data_stored":1768364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.416499"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994380+0000","last_change":"2023-10-30T13:35:07.792742+0000","last_active":"2023-10-30T13:35:10.994380+0000","last_peered":"2023-10-30T13:35:10.994380+0000","last_clean":"2023-10-30T13:35:10.994380+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:35:10.994380+0000","last_undegraded":"2023-10-30T13:35:10.994380+0000","last_fullsized":"2023-10-30T13:35:10.994380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:02.805042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744019+0000","last_change":"2023-10-30T13:35:07.755187+0000","last_active":"2023-10-30T13:35:08.744019+0000","last_peered":"2023-10-30T13:35:08.744019+0000","last_clean":"2023-10-30T13:35:08.744019+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:35:08.744019+0000","last_undegraded":"2023-10-30T13:35:08.744019+0000","last_fullsized":"2023-10-30T13:35:08.744019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:22.764808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298537+0000","last_change":"2023-10-30T13:35:07.754357+0000","last_active":"2023-10-30T13:35:09.298537+0000","last_peered":"2023-10-30T13:35:09.298537+0000","last_clean":"2023-10-30T13:35:09.298537+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:35:09.298537+0000","last_undegraded":"2023-10-30T13:35:09.298537+0000","last_fullsized":"2023-10-30T13:35:09.298537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:23.201416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748146+0000","last_change":"2023-10-30T13:35:07.792466+0000","last_active":"2023-10-30T13:35:08.748146+0000","last_peered":"2023-10-30T13:35:08.748146+0000","last_clean":"2023-10-30T13:35:08.748146+0000","last_became_active":"2023-10-30T13:35:07.789749+0000","last_became_peered":"2023-10-30T13:35:07.789749+0000","last_unstale":"2023-10-30T13:35:08.748146+0000","last_undegraded":"2023-10-30T13:35:08.748146+0000","last_fullsized":"2023-10-30T13:35:08.748146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:46.527032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993635+0000","last_change":"2023-10-30T13:35:07.793403+0000","last_active":"2023-10-30T13:35:10.993635+0000","last_peered":"2023-10-30T13:35:10.993635+0000","last_clean":"2023-10-30T13:35:10.993635+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:35:10.993635+0000","last_undegraded":"2023-10-30T13:35:10.993635+0000","last_fullsized":"2023-10-30T13:35:10.993635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:27.304652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747210+0000","last_change":"2023-10-30T13:35:07.792285+0000","last_active":"2023-10-30T13:35:08.747210+0000","last_peered":"2023-10-30T13:35:08.747210+0000","last_clean":"2023-10-30T13:35:08.747210+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:35:08.747210+0000","last_undegraded":"2023-10-30T13:35:08.747210+0000","last_fullsized":"2023-10-30T13:35:08.747210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:31.537183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993398+0000","last_change":"2023-10-30T13:35:07.776386+0000","last_active":"2023-10-30T13:35:10.993398+0000","last_peered":"2023-10-30T13:35:10.993398+0000","last_clean":"2023-10-30T13:35:10.993398+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:35:10.993398+0000","last_undegraded":"2023-10-30T13:35:10.993398+0000","last_fullsized":"2023-10-30T13:35:10.993398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:27.492389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531107+0000","last_change":"2023-10-30T13:35:07.790487+0000","last_active":"2023-10-30T13:35:09.531107+0000","last_peered":"2023-10-30T13:35:09.531107+0000","last_clean":"2023-10-30T13:35:09.531107+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:35:09.531107+0000","last_undegraded":"2023-10-30T13:35:09.531107+0000","last_fullsized":"2023-10-30T13:35:09.531107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:27.384592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994435+0000","last_change":"2023-10-30T13:35:07.804038+0000","last_active":"2023-10-30T13:35:10.994435+0000","last_peered":"2023-10-30T13:35:10.994435+0000","last_clean":"2023-10-30T13:35:10.994435+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:35:10.994435+0000","last_undegraded":"2023-10-30T13:35:10.994435+0000","last_fullsized":"2023-10-30T13:35:10.994435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:09.137404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993722+0000","last_change":"2023-10-30T13:35:07.800657+0000","last_active":"2023-10-30T13:35:10.993722+0000","last_peered":"2023-10-30T13:35:10.993722+0000","last_clean":"2023-10-30T13:35:10.993722+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:35:10.993722+0000","last_undegraded":"2023-10-30T13:35:10.993722+0000","last_fullsized":"2023-10-30T13:35:10.993722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:12.080911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993171+0000","last_change":"2023-10-30T13:35:07.802245+0000","last_active":"2023-10-30T13:35:10.993171+0000","last_peered":"2023-10-30T13:35:10.993171+0000","last_clean":"2023-10-30T13:35:10.993171+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:35:10.993171+0000","last_undegraded":"2023-10-30T13:35:10.993171+0000","last_fullsized":"2023-10-30T13:35:10.993171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:58.462586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750176+0000","last_change":"2023-10-30T13:35:07.783940+0000","last_active":"2023-10-30T13:35:08.750176+0000","last_peered":"2023-10-30T13:35:08.750176+0000","last_clean":"2023-10-30T13:35:08.750176+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:35:08.750176+0000","last_undegraded":"2023-10-30T13:35:08.750176+0000","last_fullsized":"2023-10-30T13:35:08.750176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:07.873653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531433+0000","last_change":"2023-10-30T13:35:07.780349+0000","last_active":"2023-10-30T13:35:09.531433+0000","last_peered":"2023-10-30T13:35:09.531433+0000","last_clean":"2023-10-30T13:35:09.531433+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:35:09.531433+0000","last_undegraded":"2023-10-30T13:35:09.531433+0000","last_fullsized":"2023-10-30T13:35:09.531433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:15.619776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298339+0000","last_change":"2023-10-30T13:35:07.750959+0000","last_active":"2023-10-30T13:35:09.298339+0000","last_peered":"2023-10-30T13:35:09.298339+0000","last_clean":"2023-10-30T13:35:09.298339+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:35:09.298339+0000","last_undegraded":"2023-10-30T13:35:09.298339+0000","last_fullsized":"2023-10-30T13:35:09.298339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:56.853163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.768466+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:27.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748523+0000","last_change":"2023-10-30T13:35:07.789918+0000","last_active":"2023-10-30T13:35:08.748523+0000","last_peered":"2023-10-30T13:35:08.748523+0000","last_clean":"2023-10-30T13:35:08.748523+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:35:08.748523+0000","last_undegraded":"2023-10-30T13:35:08.748523+0000","last_fullsized":"2023-10-30T13:35:08.748523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:53.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749638+0000","last_change":"2023-10-30T13:35:07.796429+0000","last_active":"2023-10-30T13:35:08.749638+0000","last_peered":"2023-10-30T13:35:08.749638+0000","last_clean":"2023-10-30T13:35:08.749638+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:35:08.749638+0000","last_undegraded":"2023-10-30T13:35:08.749638+0000","last_fullsized":"2023-10-30T13:35:08.749638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:15.022866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489395+0000","last_change":"2023-10-30T13:35:07.780131+0000","last_active":"2023-10-30T13:35:10.489395+0000","last_peered":"2023-10-30T13:35:10.489395+0000","last_clean":"2023-10-30T13:35:10.489395+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:35:10.489395+0000","last_undegraded":"2023-10-30T13:35:10.489395+0000","last_fullsized":"2023-10-30T13:35:10.489395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:04.046338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994792+0000","last_change":"2023-10-30T13:35:07.791464+0000","last_active":"2023-10-30T13:35:10.994792+0000","last_peered":"2023-10-30T13:35:10.994792+0000","last_clean":"2023-10-30T13:35:10.994792+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:35:10.994792+0000","last_undegraded":"2023-10-30T13:35:10.994792+0000","last_fullsized":"2023-10-30T13:35:10.994792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993329+0000","last_change":"2023-10-30T13:35:07.783079+0000","last_active":"2023-10-30T13:35:10.993329+0000","last_peered":"2023-10-30T13:35:10.993329+0000","last_clean":"2023-10-30T13:35:10.993329+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:35:10.993329+0000","last_undegraded":"2023-10-30T13:35:10.993329+0000","last_fullsized":"2023-10-30T13:35:10.993329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:43:32.360152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298481+0000","last_change":"2023-10-30T13:35:07.794666+0000","last_active":"2023-10-30T13:35:09.298481+0000","last_peered":"2023-10-30T13:35:09.298481+0000","last_clean":"2023-10-30T13:35:09.298481+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:35:09.298481+0000","last_undegraded":"2023-10-30T13:35:09.298481+0000","last_fullsized":"2023-10-30T13:35:09.298481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:52.189022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747346+0000","last_change":"2023-10-30T13:35:07.796601+0000","last_active":"2023-10-30T13:35:08.747346+0000","last_peered":"2023-10-30T13:35:08.747346+0000","last_clean":"2023-10-30T13:35:08.747346+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:35:08.747346+0000","last_undegraded":"2023-10-30T13:35:08.747346+0000","last_fullsized":"2023-10-30T13:35:08.747346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:07.572028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531135+0000","last_change":"2023-10-30T13:35:07.767965+0000","last_active":"2023-10-30T13:35:09.531135+0000","last_peered":"2023-10-30T13:35:09.531135+0000","last_clean":"2023-10-30T13:35:09.531135+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:35:09.531135+0000","last_undegraded":"2023-10-30T13:35:09.531135+0000","last_fullsized":"2023-10-30T13:35:09.531135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:15.544113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748017+0000","last_change":"2023-10-30T13:35:07.772029+0000","last_active":"2023-10-30T13:35:08.748017+0000","last_peered":"2023-10-30T13:35:08.748017+0000","last_clean":"2023-10-30T13:35:08.748017+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:35:08.748017+0000","last_undegraded":"2023-10-30T13:35:08.748017+0000","last_fullsized":"2023-10-30T13:35:08.748017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:13.305147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531716+0000","last_change":"2023-10-30T13:35:07.758274+0000","last_active":"2023-10-30T13:35:09.531716+0000","last_peered":"2023-10-30T13:35:09.531716+0000","last_clean":"2023-10-30T13:35:09.531716+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:35:09.531716+0000","last_undegraded":"2023-10-30T13:35:09.531716+0000","last_fullsized":"2023-10-30T13:35:09.531716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:15.191933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994099+0000","last_change":"2023-10-30T13:35:07.803220+0000","last_active":"2023-10-30T13:35:10.994099+0000","last_peered":"2023-10-30T13:35:10.994099+0000","last_clean":"2023-10-30T13:35:10.994099+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:35:10.994099+0000","last_undegraded":"2023-10-30T13:35:10.994099+0000","last_fullsized":"2023-10-30T13:35:10.994099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:45.165258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743680+0000","last_change":"2023-10-30T13:35:07.753288+0000","last_active":"2023-10-30T13:35:08.743680+0000","last_peered":"2023-10-30T13:35:08.743680+0000","last_clean":"2023-10-30T13:35:08.743680+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:35:08.743680+0000","last_undegraded":"2023-10-30T13:35:08.743680+0000","last_fullsized":"2023-10-30T13:35:08.743680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:43.815379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531511+0000","last_change":"2023-10-30T13:35:07.767931+0000","last_active":"2023-10-30T13:35:09.531511+0000","last_peered":"2023-10-30T13:35:09.531511+0000","last_clean":"2023-10-30T13:35:09.531511+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:35:09.531511+0000","last_undegraded":"2023-10-30T13:35:09.531511+0000","last_fullsized":"2023-10-30T13:35:09.531511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:43.830918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298592+0000","last_change":"2023-10-30T13:35:07.753695+0000","last_active":"2023-10-30T13:35:09.298592+0000","last_peered":"2023-10-30T13:35:09.298592+0000","last_clean":"2023-10-30T13:35:09.298592+0000","last_became_active":"2023-10-30T13:35:07.753358+0000","last_became_peered":"2023-10-30T13:35:07.753358+0000","last_unstale":"2023-10-30T13:35:09.298592+0000","last_undegraded":"2023-10-30T13:35:09.298592+0000","last_fullsized":"2023-10-30T13:35:09.298592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:12.840727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747580+0000","last_change":"2023-10-30T13:35:07.790236+0000","last_active":"2023-10-30T13:35:08.747580+0000","last_peered":"2023-10-30T13:35:08.747580+0000","last_clean":"2023-10-30T13:35:08.747580+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:35:08.747580+0000","last_undegraded":"2023-10-30T13:35:08.747580+0000","last_fullsized":"2023-10-30T13:35:08.747580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:22.029281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489368+0000","last_change":"2023-10-30T13:35:07.790720+0000","last_active":"2023-10-30T13:35:10.489368+0000","last_peered":"2023-10-30T13:35:10.489368+0000","last_clean":"2023-10-30T13:35:10.489368+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:35:10.489368+0000","last_undegraded":"2023-10-30T13:35:10.489368+0000","last_fullsized":"2023-10-30T13:35:10.489368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:16.691133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992429+0000","last_change":"2023-10-30T13:35:07.780576+0000","last_active":"2023-10-30T13:35:10.992429+0000","last_peered":"2023-10-30T13:35:10.992429+0000","last_clean":"2023-10-30T13:35:10.992429+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:35:10.992429+0000","last_undegraded":"2023-10-30T13:35:10.992429+0000","last_fullsized":"2023-10-30T13:35:10.992429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:20.704354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749626+0000","last_change":"2023-10-30T13:35:07.755919+0000","last_active":"2023-10-30T13:35:08.749626+0000","last_peered":"2023-10-30T13:35:08.749626+0000","last_clean":"2023-10-30T13:35:08.749626+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:35:08.749626+0000","last_undegraded":"2023-10-30T13:35:08.749626+0000","last_fullsized":"2023-10-30T13:35:08.749626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:01.978170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531552+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:35:09.531552+0000","last_peered":"2023-10-30T13:35:09.531552+0000","last_clean":"2023-10-30T13:35:09.531552+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:35:09.531552+0000","last_undegraded":"2023-10-30T13:35:09.531552+0000","last_fullsized":"2023-10-30T13:35:09.531552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298837+0000","last_change":"2023-10-30T13:35:07.762428+0000","last_active":"2023-10-30T13:35:09.298837+0000","last_peered":"2023-10-30T13:35:09.298837+0000","last_clean":"2023-10-30T13:35:09.298837+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:35:09.298837+0000","last_undegraded":"2023-10-30T13:35:09.298837+0000","last_fullsized":"2023-10-30T13:35:09.298837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:14.191396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992812+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:35:10.992812+0000","last_peered":"2023-10-30T13:35:10.992812+0000","last_clean":"2023-10-30T13:35:10.992812+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:35:10.992812+0000","last_undegraded":"2023-10-30T13:35:10.992812+0000","last_fullsized":"2023-10-30T13:35:10.992812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236437+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:35:09.236437+0000","last_peered":"2023-10-30T13:35:09.236437+0000","last_clean":"2023-10-30T13:35:09.236437+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:35:09.236437+0000","last_undegraded":"2023-10-30T13:35:09.236437+0000","last_fullsized":"2023-10-30T13:35:09.236437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531211+0000","last_change":"2023-10-30T13:35:07.751131+0000","last_active":"2023-10-30T13:35:09.531211+0000","last_peered":"2023-10-30T13:35:09.531211+0000","last_clean":"2023-10-30T13:35:09.531211+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:35:09.531211+0000","last_undegraded":"2023-10-30T13:35:09.531211+0000","last_fullsized":"2023-10-30T13:35:09.531211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:55:46.387299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173153+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:35:08.173153+0000","last_peered":"2023-10-30T13:35:08.173153+0000","last_clean":"2023-10-30T13:35:08.173153+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:35:08.173153+0000","last_undegraded":"2023-10-30T13:35:08.173153+0000","last_fullsized":"2023-10-30T13:35:08.173153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743438+0000","last_change":"2023-10-30T13:35:07.750392+0000","last_active":"2023-10-30T13:35:08.743438+0000","last_peered":"2023-10-30T13:35:08.743438+0000","last_clean":"2023-10-30T13:35:08.743438+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:35:08.743438+0000","last_undegraded":"2023-10-30T13:35:08.743438+0000","last_fullsized":"2023-10-30T13:35:08.743438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:33.554754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.178473+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:35:08.178473+0000","last_peered":"2023-10-30T13:35:08.178473+0000","last_clean":"2023-10-30T13:35:08.178473+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:35:08.178473+0000","last_undegraded":"2023-10-30T13:35:08.178473+0000","last_fullsized":"2023-10-30T13:35:08.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749206+0000","last_change":"2023-10-30T13:35:07.797513+0000","last_active":"2023-10-30T13:35:08.749206+0000","last_peered":"2023-10-30T13:35:08.749206+0000","last_clean":"2023-10-30T13:35:08.749206+0000","last_became_active":"2023-10-30T13:35:07.796748+0000","last_became_peered":"2023-10-30T13:35:07.796748+0000","last_unstale":"2023-10-30T13:35:08.749206+0000","last_undegraded":"2023-10-30T13:35:08.749206+0000","last_fullsized":"2023-10-30T13:35:08.749206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:41.838836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173774+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:35:08.173774+0000","last_peered":"2023-10-30T13:35:08.173774+0000","last_clean":"2023-10-30T13:35:08.173774+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:35:08.173774+0000","last_undegraded":"2023-10-30T13:35:08.173774+0000","last_fullsized":"2023-10-30T13:35:08.173774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298743+0000","last_change":"2023-10-30T13:35:07.789983+0000","last_active":"2023-10-30T13:35:09.298743+0000","last_peered":"2023-10-30T13:35:09.298743+0000","last_clean":"2023-10-30T13:35:09.298743+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:35:09.298743+0000","last_undegraded":"2023-10-30T13:35:09.298743+0000","last_fullsized":"2023-10-30T13:35:09.298743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:03.304499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.289736+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:35:13.289736+0000","last_peered":"2023-10-30T13:35:13.289736+0000","last_clean":"2023-10-30T13:35:13.289736+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:35:13.289736+0000","last_undegraded":"2023-10-30T13:35:13.289736+0000","last_fullsized":"2023-10-30T13:35:13.289736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750279+0000","last_change":"2023-10-30T13:35:07.794930+0000","last_active":"2023-10-30T13:35:08.750279+0000","last_peered":"2023-10-30T13:35:08.750279+0000","last_clean":"2023-10-30T13:35:08.750279+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:35:08.750279+0000","last_undegraded":"2023-10-30T13:35:08.750279+0000","last_fullsized":"2023-10-30T13:35:08.750279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:19.259009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294025+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:35:13.294025+0000","last_peered":"2023-10-30T13:35:13.294025+0000","last_clean":"2023-10-30T13:35:13.294025+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:35:13.294025+0000","last_undegraded":"2023-10-30T13:35:13.294025+0000","last_fullsized":"2023-10-30T13:35:13.294025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993842+0000","last_change":"2023-10-30T13:35:07.791923+0000","last_active":"2023-10-30T13:35:10.993842+0000","last_peered":"2023-10-30T13:35:10.993842+0000","last_clean":"2023-10-30T13:35:10.993842+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:35:10.993842+0000","last_undegraded":"2023-10-30T13:35:10.993842+0000","last_fullsized":"2023-10-30T13:35:10.993842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:33.080768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.290005+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:35:13.290005+0000","last_peered":"2023-10-30T13:35:13.290005+0000","last_clean":"2023-10-30T13:35:13.290005+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:35:13.290005+0000","last_undegraded":"2023-10-30T13:35:13.290005+0000","last_fullsized":"2023-10-30T13:35:13.290005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298364+0000","last_change":"2023-10-30T13:35:07.794790+0000","last_active":"2023-10-30T13:35:09.298364+0000","last_peered":"2023-10-30T13:35:09.298364+0000","last_clean":"2023-10-30T13:35:09.298364+0000","last_became_active":"2023-10-30T13:35:07.794477+0000","last_became_peered":"2023-10-30T13:35:07.794477+0000","last_unstale":"2023-10-30T13:35:09.298364+0000","last_undegraded":"2023-10-30T13:35:09.298364+0000","last_fullsized":"2023-10-30T13:35:09.298364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:45.565940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173326+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:35:08.173326+0000","last_peered":"2023-10-30T13:35:08.173326+0000","last_clean":"2023-10-30T13:35:08.173326+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:35:08.173326+0000","last_undegraded":"2023-10-30T13:35:08.173326+0000","last_fullsized":"2023-10-30T13:35:08.173326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993612+0000","last_change":"2023-10-30T13:35:07.797247+0000","last_active":"2023-10-30T13:35:10.993612+0000","last_peered":"2023-10-30T13:35:10.993612+0000","last_clean":"2023-10-30T13:35:10.993612+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:35:10.993612+0000","last_undegraded":"2023-10-30T13:35:10.993612+0000","last_fullsized":"2023-10-30T13:35:10.993612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:57:02.556658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.178231+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:35:08.178231+0000","last_peered":"2023-10-30T13:35:08.178231+0000","last_clean":"2023-10-30T13:35:08.178231+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:35:08.178231+0000","last_undegraded":"2023-10-30T13:35:08.178231+0000","last_fullsized":"2023-10-30T13:35:08.178231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993402+0000","last_change":"2023-10-30T13:35:07.773683+0000","last_active":"2023-10-30T13:35:10.993402+0000","last_peered":"2023-10-30T13:35:10.993402+0000","last_clean":"2023-10-30T13:35:10.993402+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:35:10.993402+0000","last_undegraded":"2023-10-30T13:35:10.993402+0000","last_fullsized":"2023-10-30T13:35:10.993402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:30:03.553514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291471+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:35:13.291471+0000","last_peered":"2023-10-30T13:35:13.291471+0000","last_clean":"2023-10-30T13:35:13.291471+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:35:13.291471+0000","last_undegraded":"2023-10-30T13:35:13.291471+0000","last_fullsized":"2023-10-30T13:35:13.291471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993760+0000","last_change":"2023-10-30T13:35:07.791266+0000","last_active":"2023-10-30T13:35:10.993760+0000","last_peered":"2023-10-30T13:35:10.993760+0000","last_clean":"2023-10-30T13:35:10.993760+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:35:10.993760+0000","last_undegraded":"2023-10-30T13:35:10.993760+0000","last_fullsized":"2023-10-30T13:35:10.993760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:20.086559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750401+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:35:08.750401+0000","last_peered":"2023-10-30T13:35:08.750401+0000","last_clean":"2023-10-30T13:35:08.750401+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:35:08.750401+0000","last_undegraded":"2023-10-30T13:35:08.750401+0000","last_fullsized":"2023-10-30T13:35:08.750401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994310+0000","last_change":"2023-10-30T13:35:07.794747+0000","last_active":"2023-10-30T13:35:10.994310+0000","last_peered":"2023-10-30T13:35:10.994310+0000","last_clean":"2023-10-30T13:35:10.994310+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:35:10.994310+0000","last_undegraded":"2023-10-30T13:35:10.994310+0000","last_fullsized":"2023-10-30T13:35:10.994310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:40.026054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.287725+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:35:13.287725+0000","last_peered":"2023-10-30T13:35:13.287725+0000","last_clean":"2023-10-30T13:35:13.287725+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:35:13.287725+0000","last_undegraded":"2023-10-30T13:35:13.287725+0000","last_fullsized":"2023-10-30T13:35:13.287725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994800+0000","last_change":"2023-10-30T13:35:07.793255+0000","last_active":"2023-10-30T13:35:10.994800+0000","last_peered":"2023-10-30T13:35:10.994800+0000","last_clean":"2023-10-30T13:35:10.994800+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:35:10.994800+0000","last_undegraded":"2023-10-30T13:35:10.994800+0000","last_fullsized":"2023-10-30T13:35:10.994800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:56.491002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236788+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:35:09.236788+0000","last_peered":"2023-10-30T13:35:09.236788+0000","last_clean":"2023-10-30T13:35:09.236788+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:35:09.236788+0000","last_undegraded":"2023-10-30T13:35:09.236788+0000","last_fullsized":"2023-10-30T13:35:09.236788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748659+0000","last_change":"2023-10-30T13:35:07.791920+0000","last_active":"2023-10-30T13:35:08.748659+0000","last_peered":"2023-10-30T13:35:08.748659+0000","last_clean":"2023-10-30T13:35:08.748659+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:35:08.748659+0000","last_undegraded":"2023-10-30T13:35:08.748659+0000","last_fullsized":"2023-10-30T13:35:08.748659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:04.379349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.238228+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:35:09.238228+0000","last_peered":"2023-10-30T13:35:09.238228+0000","last_clean":"2023-10-30T13:35:09.238228+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:35:09.238228+0000","last_undegraded":"2023-10-30T13:35:09.238228+0000","last_fullsized":"2023-10-30T13:35:09.238228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994214+0000","last_change":"2023-10-30T13:35:07.803211+0000","last_active":"2023-10-30T13:35:10.994214+0000","last_peered":"2023-10-30T13:35:10.994214+0000","last_clean":"2023-10-30T13:35:10.994214+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:35:10.994214+0000","last_undegraded":"2023-10-30T13:35:10.994214+0000","last_fullsized":"2023-10-30T13:35:10.994214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:55.371993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768067+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:35:07.768067+0000","last_peered":"2023-10-30T13:35:07.768067+0000","last_clean":"2023-10-30T13:35:07.768067+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:35:07.768067+0000","last_undegraded":"2023-10-30T13:35:07.768067+0000","last_fullsized":"2023-10-30T13:35:07.768067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173422+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:35:08.173422+0000","last_peered":"2023-10-30T13:35:08.173422+0000","last_clean":"2023-10-30T13:35:08.173422+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:35:08.173422+0000","last_undegraded":"2023-10-30T13:35:08.173422+0000","last_fullsized":"2023-10-30T13:35:08.173422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531304+0000","last_change":"2023-10-30T13:35:07.792082+0000","last_active":"2023-10-30T13:35:09.531304+0000","last_peered":"2023-10-30T13:35:09.531304+0000","last_clean":"2023-10-30T13:35:09.531304+0000","last_became_active":"2023-10-30T13:35:07.791831+0000","last_became_peered":"2023-10-30T13:35:07.791831+0000","last_unstale":"2023-10-30T13:35:09.531304+0000","last_undegraded":"2023-10-30T13:35:09.531304+0000","last_fullsized":"2023-10-30T13:35:09.531304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:47.863235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.250988+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:35:09.250988+0000","last_peered":"2023-10-30T13:35:09.250988+0000","last_clean":"2023-10-30T13:35:09.250988+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:35:09.250988+0000","last_undegraded":"2023-10-30T13:35:09.250988+0000","last_fullsized":"2023-10-30T13:35:09.250988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298228+0000","last_change":"2023-10-30T13:35:07.760132+0000","last_active":"2023-10-30T13:35:09.298228+0000","last_peered":"2023-10-30T13:35:09.298228+0000","last_clean":"2023-10-30T13:35:09.298228+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:35:09.298228+0000","last_undegraded":"2023-10-30T13:35:09.298228+0000","last_fullsized":"2023-10-30T13:35:09.298228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:14.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.238219+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:35:09.238219+0000","last_peered":"2023-10-30T13:35:09.238219+0000","last_clean":"2023-10-30T13:35:09.238219+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:35:09.238219+0000","last_undegraded":"2023-10-30T13:35:09.238219+0000","last_fullsized":"2023-10-30T13:35:09.238219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.750958+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:53.697186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173375+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:35:08.173375+0000","last_peered":"2023-10-30T13:35:08.173375+0000","last_clean":"2023-10-30T13:35:08.173375+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:35:08.173375+0000","last_undegraded":"2023-10-30T13:35:08.173375+0000","last_fullsized":"2023-10-30T13:35:08.173375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750179+0000","last_change":"2023-10-30T13:35:07.768640+0000","last_active":"2023-10-30T13:35:08.750179+0000","last_peered":"2023-10-30T13:35:08.750179+0000","last_clean":"2023-10-30T13:35:08.750179+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:35:08.750179+0000","last_undegraded":"2023-10-30T13:35:08.750179+0000","last_fullsized":"2023-10-30T13:35:08.750179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:26.852057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743612+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:35:08.743612+0000","last_peered":"2023-10-30T13:35:08.743612+0000","last_clean":"2023-10-30T13:35:08.743612+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:35:08.743612+0000","last_undegraded":"2023-10-30T13:35:08.743612+0000","last_fullsized":"2023-10-30T13:35:08.743612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749093+0000","last_change":"2023-10-30T13:35:07.805535+0000","last_active":"2023-10-30T13:35:08.749093+0000","last_peered":"2023-10-30T13:35:08.749093+0000","last_clean":"2023-10-30T13:35:08.749093+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:35:08.749093+0000","last_undegraded":"2023-10-30T13:35:08.749093+0000","last_fullsized":"2023-10-30T13:35:08.749093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:06.549889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.251374+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:35:09.251374+0000","last_peered":"2023-10-30T13:35:09.251374+0000","last_clean":"2023-10-30T13:35:09.251374+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:35:09.251374+0000","last_undegraded":"2023-10-30T13:35:09.251374+0000","last_fullsized":"2023-10-30T13:35:09.251374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743474+0000","last_change":"2023-10-30T13:35:07.754171+0000","last_active":"2023-10-30T13:35:08.743474+0000","last_peered":"2023-10-30T13:35:08.743474+0000","last_clean":"2023-10-30T13:35:08.743474+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:35:08.743474+0000","last_undegraded":"2023-10-30T13:35:08.743474+0000","last_fullsized":"2023-10-30T13:35:08.743474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:50.849695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744041+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:35:08.744041+0000","last_peered":"2023-10-30T13:35:08.744041+0000","last_clean":"2023-10-30T13:35:08.744041+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:35:08.744041+0000","last_undegraded":"2023-10-30T13:35:08.744041+0000","last_fullsized":"2023-10-30T13:35:08.744041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531754+0000","last_change":"2023-10-30T13:35:07.794804+0000","last_active":"2023-10-30T13:35:09.531754+0000","last_peered":"2023-10-30T13:35:09.531754+0000","last_clean":"2023-10-30T13:35:09.531754+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:35:09.531754+0000","last_undegraded":"2023-10-30T13:35:09.531754+0000","last_fullsized":"2023-10-30T13:35:09.531754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:00.913592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.287281+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:35:13.287281+0000","last_peered":"2023-10-30T13:35:13.287281+0000","last_clean":"2023-10-30T13:35:13.287281+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:35:13.287281+0000","last_undegraded":"2023-10-30T13:35:13.287281+0000","last_fullsized":"2023-10-30T13:35:13.287281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298809+0000","last_change":"2023-10-30T13:35:07.752077+0000","last_active":"2023-10-30T13:35:09.298809+0000","last_peered":"2023-10-30T13:35:09.298809+0000","last_clean":"2023-10-30T13:35:09.298809+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:35:09.298809+0000","last_undegraded":"2023-10-30T13:35:09.298809+0000","last_fullsized":"2023-10-30T13:35:09.298809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:07.661975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.250947+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:35:09.250947+0000","last_peered":"2023-10-30T13:35:09.250947+0000","last_clean":"2023-10-30T13:35:09.250947+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:35:09.250947+0000","last_undegraded":"2023-10-30T13:35:09.250947+0000","last_fullsized":"2023-10-30T13:35:09.250947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748158+0000","last_change":"2023-10-30T13:35:07.759389+0000","last_active":"2023-10-30T13:35:08.748158+0000","last_peered":"2023-10-30T13:35:08.748158+0000","last_clean":"2023-10-30T13:35:08.748158+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:35:08.748158+0000","last_undegraded":"2023-10-30T13:35:08.748158+0000","last_fullsized":"2023-10-30T13:35:08.748158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:15.536615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743540+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:35:08.743540+0000","last_peered":"2023-10-30T13:35:08.743540+0000","last_clean":"2023-10-30T13:35:08.743540+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:35:08.743540+0000","last_undegraded":"2023-10-30T13:35:08.743540+0000","last_fullsized":"2023-10-30T13:35:08.743540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531288+0000","last_change":"2023-10-30T13:35:07.753673+0000","last_active":"2023-10-30T13:35:09.531288+0000","last_peered":"2023-10-30T13:35:09.531288+0000","last_clean":"2023-10-30T13:35:09.531288+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:35:09.531288+0000","last_undegraded":"2023-10-30T13:35:09.531288+0000","last_fullsized":"2023-10-30T13:35:09.531288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:27:18.430183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743863+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:35:08.743863+0000","last_peered":"2023-10-30T13:35:08.743863+0000","last_clean":"2023-10-30T13:35:08.743863+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:35:08.743863+0000","last_undegraded":"2023-10-30T13:35:08.743863+0000","last_fullsized":"2023-10-30T13:35:08.743863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531465+0000","last_change":"2023-10-30T13:35:07.755302+0000","last_active":"2023-10-30T13:35:09.531465+0000","last_peered":"2023-10-30T13:35:09.531465+0000","last_clean":"2023-10-30T13:35:09.531465+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:35:09.531465+0000","last_undegraded":"2023-10-30T13:35:09.531465+0000","last_fullsized":"2023-10-30T13:35:09.531465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:31.543587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.292960+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:35:13.292960+0000","last_peered":"2023-10-30T13:35:13.292960+0000","last_clean":"2023-10-30T13:35:13.292960+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:35:13.292960+0000","last_undegraded":"2023-10-30T13:35:13.292960+0000","last_fullsized":"2023-10-30T13:35:13.292960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992766+0000","last_change":"2023-10-30T13:35:07.780297+0000","last_active":"2023-10-30T13:35:10.992766+0000","last_peered":"2023-10-30T13:35:10.992766+0000","last_clean":"2023-10-30T13:35:10.992766+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:35:10.992766+0000","last_undegraded":"2023-10-30T13:35:10.992766+0000","last_fullsized":"2023-10-30T13:35:10.992766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:48.971885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748946+0000","last_change":"2023-10-30T13:35:07.806203+0000","last_active":"2023-10-30T13:35:08.748946+0000","last_peered":"2023-10-30T13:35:08.748946+0000","last_clean":"2023-10-30T13:35:08.748946+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:35:08.748946+0000","last_undegraded":"2023-10-30T13:35:08.748946+0000","last_fullsized":"2023-10-30T13:35:08.748946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:34.175218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294221+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:35:13.294221+0000","last_peered":"2023-10-30T13:35:13.294221+0000","last_clean":"2023-10-30T13:35:13.294221+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:35:13.294221+0000","last_undegraded":"2023-10-30T13:35:13.294221+0000","last_fullsized":"2023-10-30T13:35:13.294221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748564+0000","last_change":"2023-10-30T13:35:07.791319+0000","last_active":"2023-10-30T13:35:08.748564+0000","last_peered":"2023-10-30T13:35:08.748564+0000","last_clean":"2023-10-30T13:35:08.748564+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:35:08.748564+0000","last_undegraded":"2023-10-30T13:35:08.748564+0000","last_fullsized":"2023-10-30T13:35:08.748564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:42.560856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294002+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:35:13.294002+0000","last_peered":"2023-10-30T13:35:13.294002+0000","last_clean":"2023-10-30T13:35:13.294002+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:35:13.294002+0000","last_undegraded":"2023-10-30T13:35:13.294002+0000","last_fullsized":"2023-10-30T13:35:13.294002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298499+0000","last_change":"2023-10-30T13:35:07.790084+0000","last_active":"2023-10-30T13:35:09.298499+0000","last_peered":"2023-10-30T13:35:09.298499+0000","last_clean":"2023-10-30T13:35:09.298499+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:35:09.298499+0000","last_undegraded":"2023-10-30T13:35:09.298499+0000","last_fullsized":"2023-10-30T13:35:09.298499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:45.193072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531782+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:35:09.531782+0000","last_peered":"2023-10-30T13:35:09.531782+0000","last_clean":"2023-10-30T13:35:09.531782+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:35:09.531782+0000","last_undegraded":"2023-10-30T13:35:09.531782+0000","last_fullsized":"2023-10-30T13:35:09.531782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531695+0000","last_change":"2023-10-30T13:35:07.782341+0000","last_active":"2023-10-30T13:35:09.531695+0000","last_peered":"2023-10-30T13:35:09.531695+0000","last_clean":"2023-10-30T13:35:09.531695+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:35:09.531695+0000","last_undegraded":"2023-10-30T13:35:09.531695+0000","last_fullsized":"2023-10-30T13:35:09.531695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:19:06.392683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.292830+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:35:13.292830+0000","last_peered":"2023-10-30T13:35:13.292830+0000","last_clean":"2023-10-30T13:35:13.292830+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:35:13.292830+0000","last_undegraded":"2023-10-30T13:35:13.292830+0000","last_fullsized":"2023-10-30T13:35:13.292830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":806912,"data_stored":663801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]}]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":74463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":72851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":74463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":72941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":72977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":663857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:35:19.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:35:19.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1eb291640 1 Processor -- start 2023-10-30T13:35:19.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1eb291640 1 -- start start 2023-10-30T13:35:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1eb291640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc1e4153c80 0x7fc1e415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1eb291640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc1e40c4a10 0x7fc1e40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1eb291640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e4159f20 con 0x7fc1e415a3b0 2023-10-30T13:35:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1eb291640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e415dec0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1eb291640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e41599a0 con 0x7fc1e4153c80 2023-10-30T13:35:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e882c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc1e40c4a10 0x7fc1e40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e882c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc1e40c4a10 0x7fc1e40c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36180/0 (socket says 172.21.15.31:36180) 2023-10-30T13:35:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e882c640 1 -- 172.21.15.31:0/1075224855 learned_addr learned my addr 172.21.15.31:0/1075224855 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e982e640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc1e415a3b0 0x7fc1e40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51970/0 (socket says 172.21.15.31:51970) 2023-10-30T13:35:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e902d640 1 --2- 172.21.15.31:0/1075224855 >> v2:172.21.15.33:3300/0 conn(0x7fc1e4153c80 0x7fc1e415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e882c640 1 -- 172.21.15.31:0/1075224855 >> v2:172.21.15.33:3300/0 conn(0x7fc1e4153c80 msgr2=0x7fc1e415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e882c640 1 --2- 172.21.15.31:0/1075224855 >> v2:172.21.15.33:3300/0 conn(0x7fc1e4153c80 0x7fc1e415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e882c640 1 -- 172.21.15.31:0/1075224855 >> v1:172.21.15.31:6789/0 conn(0x7fc1e415a3b0 legacy=0x7fc1e40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1dbfff640 1 -- 172.21.15.31:0/1075224855 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298860394 0 0) 0x7fc1e4159f20 con 0x7fc1e415a3b0 2023-10-30T13:35:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.412+0000 7fc1e882c640 1 -- 172.21.15.31:0/1075224855 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e40c47e0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1e882c640 1 --2- 172.21.15.31:0/1075224855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e40c42a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40d00f0 tx=0x7fc1d401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacad12c27acb719 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1dbfff640 1 -- 172.21.15.31:0/1075224855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d40a89f0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1dbfff640 1 -- 172.21.15.31:0/1075224855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d40a8b50 con 0x7fc1e40c4a10 2023-10-30T13:35:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1dbfff640 1 -- 172.21.15.31:0/1075224855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d40b0440 con 0x7fc1e40c4a10 2023-10-30T13:35:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 -- 172.21.15.31:0/1075224855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 msgr2=0x7fc1e40c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 --2- 172.21.15.31:0/1075224855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e40c42a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40d00f0 tx=0x7fc1d401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 -- 172.21.15.31:0/1075224855 shutdown_connections 2023-10-30T13:35:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 --2- 172.21.15.31:0/1075224855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e40c42a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 --2- 172.21.15.31:0/1075224855 >> v2:172.21.15.33:3300/0 conn(0x7fc1e4153c80 0x7fc1e415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 -- 172.21.15.31:0/1075224855 >> 172.21.15.31:0/1075224855 conn(0x7fc1e40715c0 msgr2=0x7fc1e40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 -- 172.21.15.31:0/1075224855 shutdown_connections 2023-10-30T13:35:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.416+0000 7fc1eb291640 1 -- 172.21.15.31:0/1075224855 wait complete. 2023-10-30T13:35:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 Processor -- start 2023-10-30T13:35:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 -- start start 2023-10-30T13:35:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e414e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1e4153c80 0x7fc1e4152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1e415a3b0 0x7fc1e41eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e4150ab0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e4150d70 con 0x7fc1e4153c80 2023-10-30T13:35:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e41f1fe0 con 0x7fc1e415a3b0 2023-10-30T13:35:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e414e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e414e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36192/0 (socket says 172.21.15.31:36192) 2023-10-30T13:35:19.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 -- 172.21.15.31:0/2440634288 learned_addr learned my addr 172.21.15.31:0/2440634288 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:19.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e982e640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1e415a3b0 0x7fc1e41eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e882c640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1e4153c80 0x7fc1e4152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 -- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1e415a3b0 msgr2=0x7fc1e41eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1e415a3b0 0x7fc1e41eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 -- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1e4153c80 msgr2=0x7fc1e4152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1e4153c80 0x7fc1e4152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 -- 172.21.15.31:0/2440634288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40094d0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1e902d640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e414e240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000d800 tx=0x7fc1e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1d9ffb640 1 -- 172.21.15.31:0/2440634288 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000b6c0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1d9ffb640 1 -- 172.21.15.31:0/2440634288 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e000b820 con 0x7fc1e40c4a10 2023-10-30T13:35:19.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1d9ffb640 1 -- 172.21.15.31:0/2440634288 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000bb90 con 0x7fc1e40c4a10 2023-10-30T13:35:19.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e41f22c0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.420+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e41f2880 con 0x7fc1e40c4a10 2023-10-30T13:35:19.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.424+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e404e170 con 0x7fc1e40c4a10 2023-10-30T13:35:19.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.424+0000 7fc1d9ffb640 1 -- 172.21.15.31:0/2440634288 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1e000bcf0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.424+0000 7fc1d9ffb640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1c80664a0 0x7fc1c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.424+0000 7fc1d9ffb640 1 -- 172.21.15.31:0/2440634288 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1e00844b0 con 0x7fc1e40c4a10 2023-10-30T13:35:19.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.424+0000 7fc1e882c640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1c80664a0 0x7fc1c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.424+0000 7fc1e882c640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1c80664a0 0x7fc1c8068960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc1d40b2000 tx=0x7fc1d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.428+0000 7fc1d9ffb640 1 -- 172.21.15.31:0/2440634288 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1e0051890 con 0x7fc1e40c4a10 2023-10-30T13:35:19.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.580+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1e415d900 con 0x7fc1c80664a0 2023-10-30T13:35:19.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.600+0000 7fc1d9ffb640 1 -- 172.21.15.31:0/2440634288 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264444 (secure 0 0 0) 0x7fc1d40b04c0 con 0x7fc1c80664a0 2023-10-30T13:35:19.608 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:35:19.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1c80664a0 msgr2=0x7fc1c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1c80664a0 0x7fc1c8068960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc1d40b2000 tx=0x7fc1d4008040 comp rx=0 tx=0).stop 2023-10-30T13:35:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 msgr2=0x7fc1e414e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e414e240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000d800 tx=0x7fc1e000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 shutdown_connections 2023-10-30T13:35:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1c80664a0 0x7fc1c8068960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1e415a3b0 0x7fc1e41eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1e4153c80 0x7fc1e4152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 --2- 172.21.15.31:0/2440634288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1e40c4a10 0x7fc1e414e240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 >> 172.21.15.31:0/2440634288 conn(0x7fc1e40715c0 msgr2=0x7fc1e40cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 shutdown_connections 2023-10-30T13:35:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.604+0000 7fc1eb291640 1 -- 172.21.15.31:0/2440634288 wait complete. 2023-10-30T13:35:19.628 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-30T13:35:19.159549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2924544,"data_stored":1768364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.416499"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994380+0000","last_change":"2023-10-30T13:35:07.792742+0000","last_active":"2023-10-30T13:35:10.994380+0000","last_peered":"2023-10-30T13:35:10.994380+0000","last_clean":"2023-10-30T13:35:10.994380+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:35:10.994380+0000","last_undegraded":"2023-10-30T13:35:10.994380+0000","last_fullsized":"2023-10-30T13:35:10.994380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:02.805042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744019+0000","last_change":"2023-10-30T13:35:07.755187+0000","last_active":"2023-10-30T13:35:08.744019+0000","last_peered":"2023-10-30T13:35:08.744019+0000","last_clean":"2023-10-30T13:35:08.744019+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:35:08.744019+0000","last_undegraded":"2023-10-30T13:35:08.744019+0000","last_fullsized":"2023-10-30T13:35:08.744019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:22.764808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298537+0000","last_change":"2023-10-30T13:35:07.754357+0000","last_active":"2023-10-30T13:35:09.298537+0000","last_peered":"2023-10-30T13:35:09.298537+0000","last_clean":"2023-10-30T13:35:09.298537+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:35:09.298537+0000","last_undegraded":"2023-10-30T13:35:09.298537+0000","last_fullsized":"2023-10-30T13:35:09.298537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:23.201416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748146+0000","last_change":"2023-10-30T13:35:07.792466+0000","last_active":"2023-10-30T13:35:08.748146+0000","last_peered":"2023-10-30T13:35:08.748146+0000","last_clean":"2023-10-30T13:35:08.748146+0000","last_became_active":"2023-10-30T13:35:07.789749+0000","last_became_peered":"2023-10-30T13:35:07.789749+0000","last_unstale":"2023-10-30T13:35:08.748146+0000","last_undegraded":"2023-10-30T13:35:08.748146+0000","last_fullsized":"2023-10-30T13:35:08.748146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:46.527032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993635+0000","last_change":"2023-10-30T13:35:07.793403+0000","last_active":"2023-10-30T13:35:10.993635+0000","last_peered":"2023-10-30T13:35:10.993635+0000","last_clean":"2023-10-30T13:35:10.993635+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:35:10.993635+0000","last_undegraded":"2023-10-30T13:35:10.993635+0000","last_fullsized":"2023-10-30T13:35:10.993635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:27.304652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747210+0000","last_change":"2023-10-30T13:35:07.792285+0000","last_active":"2023-10-30T13:35:08.747210+0000","last_peered":"2023-10-30T13:35:08.747210+0000","last_clean":"2023-10-30T13:35:08.747210+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:35:08.747210+0000","last_undegraded":"2023-10-30T13:35:08.747210+0000","last_fullsized":"2023-10-30T13:35:08.747210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:31.537183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993398+0000","last_change":"2023-10-30T13:35:07.776386+0000","last_active":"2023-10-30T13:35:10.993398+0000","last_peered":"2023-10-30T13:35:10.993398+0000","last_clean":"2023-10-30T13:35:10.993398+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:35:10.993398+0000","last_undegraded":"2023-10-30T13:35:10.993398+0000","last_fullsized":"2023-10-30T13:35:10.993398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:27.492389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531107+0000","last_change":"2023-10-30T13:35:07.790487+0000","last_active":"2023-10-30T13:35:09.531107+0000","last_peered":"2023-10-30T13:35:09.531107+0000","last_clean":"2023-10-30T13:35:09.531107+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:35:09.531107+0000","last_undegraded":"2023-10-30T13:35:09.531107+0000","last_fullsized":"2023-10-30T13:35:09.531107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:27.384592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994435+0000","last_change":"2023-10-30T13:35:07.804038+0000","last_active":"2023-10-30T13:35:10.994435+0000","last_peered":"2023-10-30T13:35:10.994435+0000","last_clean":"2023-10-30T13:35:10.994435+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:35:10.994435+0000","last_undegraded":"2023-10-30T13:35:10.994435+0000","last_fullsized":"2023-10-30T13:35:10.994435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:09.137404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993722+0000","last_change":"2023-10-30T13:35:07.800657+0000","last_active":"2023-10-30T13:35:10.993722+0000","last_peered":"2023-10-30T13:35:10.993722+0000","last_clean":"2023-10-30T13:35:10.993722+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:35:10.993722+0000","last_undegraded":"2023-10-30T13:35:10.993722+0000","last_fullsized":"2023-10-30T13:35:10.993722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:12.080911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993171+0000","last_change":"2023-10-30T13:35:07.802245+0000","last_active":"2023-10-30T13:35:10.993171+0000","last_peered":"2023-10-30T13:35:10.993171+0000","last_clean":"2023-10-30T13:35:10.993171+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:35:10.993171+0000","last_undegraded":"2023-10-30T13:35:10.993171+0000","last_fullsized":"2023-10-30T13:35:10.993171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:58.462586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750176+0000","last_change":"2023-10-30T13:35:07.783940+0000","last_active":"2023-10-30T13:35:08.750176+0000","last_peered":"2023-10-30T13:35:08.750176+0000","last_clean":"2023-10-30T13:35:08.750176+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:35:08.750176+0000","last_undegraded":"2023-10-30T13:35:08.750176+0000","last_fullsized":"2023-10-30T13:35:08.750176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:07.873653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531433+0000","last_change":"2023-10-30T13:35:07.780349+0000","last_active":"2023-10-30T13:35:09.531433+0000","last_peered":"2023-10-30T13:35:09.531433+0000","last_clean":"2023-10-30T13:35:09.531433+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:35:09.531433+0000","last_undegraded":"2023-10-30T13:35:09.531433+0000","last_fullsized":"2023-10-30T13:35:09.531433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:15.619776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298339+0000","last_change":"2023-10-30T13:35:07.750959+0000","last_active":"2023-10-30T13:35:09.298339+0000","last_peered":"2023-10-30T13:35:09.298339+0000","last_clean":"2023-10-30T13:35:09.298339+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:35:09.298339+0000","last_undegraded":"2023-10-30T13:35:09.298339+0000","last_fullsized":"2023-10-30T13:35:09.298339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:56.853163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.768466+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:27.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748523+0000","last_change":"2023-10-30T13:35:07.789918+0000","last_active":"2023-10-30T13:35:08.748523+0000","last_peered":"2023-10-30T13:35:08.748523+0000","last_clean":"2023-10-30T13:35:08.748523+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:35:08.748523+0000","last_undegraded":"2023-10-30T13:35:08.748523+0000","last_fullsized":"2023-10-30T13:35:08.748523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:53.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749638+0000","last_change":"2023-10-30T13:35:07.796429+0000","last_active":"2023-10-30T13:35:08.749638+0000","last_peered":"2023-10-30T13:35:08.749638+0000","last_clean":"2023-10-30T13:35:08.749638+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:35:08.749638+0000","last_undegraded":"2023-10-30T13:35:08.749638+0000","last_fullsized":"2023-10-30T13:35:08.749638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:15.022866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489395+0000","last_change":"2023-10-30T13:35:07.780131+0000","last_active":"2023-10-30T13:35:10.489395+0000","last_peered":"2023-10-30T13:35:10.489395+0000","last_clean":"2023-10-30T13:35:10.489395+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:35:10.489395+0000","last_undegraded":"2023-10-30T13:35:10.489395+0000","last_fullsized":"2023-10-30T13:35:10.489395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:04.046338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994792+0000","last_change":"2023-10-30T13:35:07.791464+0000","last_active":"2023-10-30T13:35:10.994792+0000","last_peered":"2023-10-30T13:35:10.994792+0000","last_clean":"2023-10-30T13:35:10.994792+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:35:10.994792+0000","last_undegraded":"2023-10-30T13:35:10.994792+0000","last_fullsized":"2023-10-30T13:35:10.994792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993329+0000","last_change":"2023-10-30T13:35:07.783079+0000","last_active":"2023-10-30T13:35:10.993329+0000","last_peered":"2023-10-30T13:35:10.993329+0000","last_clean":"2023-10-30T13:35:10.993329+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:35:10.993329+0000","last_undegraded":"2023-10-30T13:35:10.993329+0000","last_fullsized":"2023-10-30T13:35:10.993329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:43:32.360152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298481+0000","last_change":"2023-10-30T13:35:07.794666+0000","last_active":"2023-10-30T13:35:09.298481+0000","last_peered":"2023-10-30T13:35:09.298481+0000","last_clean":"2023-10-30T13:35:09.298481+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:35:09.298481+0000","last_undegraded":"2023-10-30T13:35:09.298481+0000","last_fullsized":"2023-10-30T13:35:09.298481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:52.189022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747346+0000","last_change":"2023-10-30T13:35:07.796601+0000","last_active":"2023-10-30T13:35:08.747346+0000","last_peered":"2023-10-30T13:35:08.747346+0000","last_clean":"2023-10-30T13:35:08.747346+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:35:08.747346+0000","last_undegraded":"2023-10-30T13:35:08.747346+0000","last_fullsized":"2023-10-30T13:35:08.747346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:07.572028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531135+0000","last_change":"2023-10-30T13:35:07.767965+0000","last_active":"2023-10-30T13:35:09.531135+0000","last_peered":"2023-10-30T13:35:09.531135+0000","last_clean":"2023-10-30T13:35:09.531135+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:35:09.531135+0000","last_undegraded":"2023-10-30T13:35:09.531135+0000","last_fullsized":"2023-10-30T13:35:09.531135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:15.544113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748017+0000","last_change":"2023-10-30T13:35:07.772029+0000","last_active":"2023-10-30T13:35:08.748017+0000","last_peered":"2023-10-30T13:35:08.748017+0000","last_clean":"2023-10-30T13:35:08.748017+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:35:08.748017+0000","last_undegraded":"2023-10-30T13:35:08.748017+0000","last_fullsized":"2023-10-30T13:35:08.748017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:13.305147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531716+0000","last_change":"2023-10-30T13:35:07.758274+0000","last_active":"2023-10-30T13:35:09.531716+0000","last_peered":"2023-10-30T13:35:09.531716+0000","last_clean":"2023-10-30T13:35:09.531716+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:35:09.531716+0000","last_undegraded":"2023-10-30T13:35:09.531716+0000","last_fullsized":"2023-10-30T13:35:09.531716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:15.191933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994099+0000","last_change":"2023-10-30T13:35:07.803220+0000","last_active":"2023-10-30T13:35:10.994099+0000","last_peered":"2023-10-30T13:35:10.994099+0000","last_clean":"2023-10-30T13:35:10.994099+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:35:10.994099+0000","last_undegraded":"2023-10-30T13:35:10.994099+0000","last_fullsized":"2023-10-30T13:35:10.994099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:45.165258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743680+0000","last_change":"2023-10-30T13:35:07.753288+0000","last_active":"2023-10-30T13:35:08.743680+0000","last_peered":"2023-10-30T13:35:08.743680+0000","last_clean":"2023-10-30T13:35:08.743680+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:35:08.743680+0000","last_undegraded":"2023-10-30T13:35:08.743680+0000","last_fullsized":"2023-10-30T13:35:08.743680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:43.815379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531511+0000","last_change":"2023-10-30T13:35:07.767931+0000","last_active":"2023-10-30T13:35:09.531511+0000","last_peered":"2023-10-30T13:35:09.531511+0000","last_clean":"2023-10-30T13:35:09.531511+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:35:09.531511+0000","last_undegraded":"2023-10-30T13:35:09.531511+0000","last_fullsized":"2023-10-30T13:35:09.531511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:43.830918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298592+0000","last_change":"2023-10-30T13:35:07.753695+0000","last_active":"2023-10-30T13:35:09.298592+0000","last_peered":"2023-10-30T13:35:09.298592+0000","last_clean":"2023-10-30T13:35:09.298592+0000","last_became_active":"2023-10-30T13:35:07.753358+0000","last_became_peered":"2023-10-30T13:35:07.753358+0000","last_unstale":"2023-10-30T13:35:09.298592+0000","last_undegraded":"2023-10-30T13:35:09.298592+0000","last_fullsized":"2023-10-30T13:35:09.298592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:12.840727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747580+0000","last_change":"2023-10-30T13:35:07.790236+0000","last_active":"2023-10-30T13:35:08.747580+0000","last_peered":"2023-10-30T13:35:08.747580+0000","last_clean":"2023-10-30T13:35:08.747580+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:35:08.747580+0000","last_undegraded":"2023-10-30T13:35:08.747580+0000","last_fullsized":"2023-10-30T13:35:08.747580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:22.029281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489368+0000","last_change":"2023-10-30T13:35:07.790720+0000","last_active":"2023-10-30T13:35:10.489368+0000","last_peered":"2023-10-30T13:35:10.489368+0000","last_clean":"2023-10-30T13:35:10.489368+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:35:10.489368+0000","last_undegraded":"2023-10-30T13:35:10.489368+0000","last_fullsized":"2023-10-30T13:35:10.489368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:16.691133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992429+0000","last_change":"2023-10-30T13:35:07.780576+0000","last_active":"2023-10-30T13:35:10.992429+0000","last_peered":"2023-10-30T13:35:10.992429+0000","last_clean":"2023-10-30T13:35:10.992429+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:35:10.992429+0000","last_undegraded":"2023-10-30T13:35:10.992429+0000","last_fullsized":"2023-10-30T13:35:10.992429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:20.704354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749626+0000","last_change":"2023-10-30T13:35:07.755919+0000","last_active":"2023-10-30T13:35:08.749626+0000","last_peered":"2023-10-30T13:35:08.749626+0000","last_clean":"2023-10-30T13:35:08.749626+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:35:08.749626+0000","last_undegraded":"2023-10-30T13:35:08.749626+0000","last_fullsized":"2023-10-30T13:35:08.749626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:01.978170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531552+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:35:09.531552+0000","last_peered":"2023-10-30T13:35:09.531552+0000","last_clean":"2023-10-30T13:35:09.531552+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:35:09.531552+0000","last_undegraded":"2023-10-30T13:35:09.531552+0000","last_fullsized":"2023-10-30T13:35:09.531552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298837+0000","last_change":"2023-10-30T13:35:07.762428+0000","last_active":"2023-10-30T13:35:09.298837+0000","last_peered":"2023-10-30T13:35:09.298837+0000","last_clean":"2023-10-30T13:35:09.298837+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:35:09.298837+0000","last_undegraded":"2023-10-30T13:35:09.298837+0000","last_fullsized":"2023-10-30T13:35:09.298837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:14.191396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992812+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:35:10.992812+0000","last_peered":"2023-10-30T13:35:10.992812+0000","last_clean":"2023-10-30T13:35:10.992812+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:35:10.992812+0000","last_undegraded":"2023-10-30T13:35:10.992812+0000","last_fullsized":"2023-10-30T13:35:10.992812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236437+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:35:09.236437+0000","last_peered":"2023-10-30T13:35:09.236437+0000","last_clean":"2023-10-30T13:35:09.236437+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:35:09.236437+0000","last_undegraded":"2023-10-30T13:35:09.236437+0000","last_fullsized":"2023-10-30T13:35:09.236437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531211+0000","last_change":"2023-10-30T13:35:07.751131+0000","last_active":"2023-10-30T13:35:09.531211+0000","last_peered":"2023-10-30T13:35:09.531211+0000","last_clean":"2023-10-30T13:35:09.531211+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:35:09.531211+0000","last_undegraded":"2023-10-30T13:35:09.531211+0000","last_fullsized":"2023-10-30T13:35:09.531211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:55:46.387299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173153+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:35:08.173153+0000","last_peered":"2023-10-30T13:35:08.173153+0000","last_clean":"2023-10-30T13:35:08.173153+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:35:08.173153+0000","last_undegraded":"2023-10-30T13:35:08.173153+0000","last_fullsized":"2023-10-30T13:35:08.173153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743438+0000","last_change":"2023-10-30T13:35:07.750392+0000","last_active":"2023-10-30T13:35:08.743438+0000","last_peered":"2023-10-30T13:35:08.743438+0000","last_clean":"2023-10-30T13:35:08.743438+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:35:08.743438+0000","last_undegraded":"2023-10-30T13:35:08.743438+0000","last_fullsized":"2023-10-30T13:35:08.743438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:33.554754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.178473+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:35:08.178473+0000","last_peered":"2023-10-30T13:35:08.178473+0000","last_clean":"2023-10-30T13:35:08.178473+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:35:08.178473+0000","last_undegraded":"2023-10-30T13:35:08.178473+0000","last_fullsized":"2023-10-30T13:35:08.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749206+0000","last_change":"2023-10-30T13:35:07.797513+0000","last_active":"2023-10-30T13:35:08.749206+0000","last_peered":"2023-10-30T13:35:08.749206+0000","last_clean":"2023-10-30T13:35:08.749206+0000","last_became_active":"2023-10-30T13:35:07.796748+0000","last_became_peered":"2023-10-30T13:35:07.796748+0000","last_unstale":"2023-10-30T13:35:08.749206+0000","last_undegraded":"2023-10-30T13:35:08.749206+0000","last_fullsized":"2023-10-30T13:35:08.749206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:41.838836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173774+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:35:08.173774+0000","last_peered":"2023-10-30T13:35:08.173774+0000","last_clean":"2023-10-30T13:35:08.173774+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:35:08.173774+0000","last_undegraded":"2023-10-30T13:35:08.173774+0000","last_fullsized":"2023-10-30T13:35:08.173774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298743+0000","last_change":"2023-10-30T13:35:07.789983+0000","last_active":"2023-10-30T13:35:09.298743+0000","last_peered":"2023-10-30T13:35:09.298743+0000","last_clean":"2023-10-30T13:35:09.298743+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:35:09.298743+0000","last_undegraded":"2023-10-30T13:35:09.298743+0000","last_fullsized":"2023-10-30T13:35:09.298743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:03.304499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.289736+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:35:13.289736+0000","last_peered":"2023-10-30T13:35:13.289736+0000","last_clean":"2023-10-30T13:35:13.289736+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:35:13.289736+0000","last_undegraded":"2023-10-30T13:35:13.289736+0000","last_fullsized":"2023-10-30T13:35:13.289736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750279+0000","last_change":"2023-10-30T13:35:07.794930+0000","last_active":"2023-10-30T13:35:08.750279+0000","last_peered":"2023-10-30T13:35:08.750279+0000","last_clean":"2023-10-30T13:35:08.750279+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:35:08.750279+0000","last_undegraded":"2023-10-30T13:35:08.750279+0000","last_fullsized":"2023-10-30T13:35:08.750279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:19.259009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294025+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:35:13.294025+0000","last_peered":"2023-10-30T13:35:13.294025+0000","last_clean":"2023-10-30T13:35:13.294025+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:35:13.294025+0000","last_undegraded":"2023-10-30T13:35:13.294025+0000","last_fullsized":"2023-10-30T13:35:13.294025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993842+0000","last_change":"2023-10-30T13:35:07.791923+0000","last_active":"2023-10-30T13:35:10.993842+0000","last_peered":"2023-10-30T13:35:10.993842+0000","last_clean":"2023-10-30T13:35:10.993842+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:35:10.993842+0000","last_undegraded":"2023-10-30T13:35:10.993842+0000","last_fullsized":"2023-10-30T13:35:10.993842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:33.080768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.290005+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:35:13.290005+0000","last_peered":"2023-10-30T13:35:13.290005+0000","last_clean":"2023-10-30T13:35:13.290005+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:35:13.290005+0000","last_undegraded":"2023-10-30T13:35:13.290005+0000","last_fullsized":"2023-10-30T13:35:13.290005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298364+0000","last_change":"2023-10-30T13:35:07.794790+0000","last_active":"2023-10-30T13:35:09.298364+0000","last_peered":"2023-10-30T13:35:09.298364+0000","last_clean":"2023-10-30T13:35:09.298364+0000","last_became_active":"2023-10-30T13:35:07.794477+0000","last_became_peered":"2023-10-30T13:35:07.794477+0000","last_unstale":"2023-10-30T13:35:09.298364+0000","last_undegraded":"2023-10-30T13:35:09.298364+0000","last_fullsized":"2023-10-30T13:35:09.298364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:45.565940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173326+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:35:08.173326+0000","last_peered":"2023-10-30T13:35:08.173326+0000","last_clean":"2023-10-30T13:35:08.173326+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:35:08.173326+0000","last_undegraded":"2023-10-30T13:35:08.173326+0000","last_fullsized":"2023-10-30T13:35:08.173326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993612+0000","last_change":"2023-10-30T13:35:07.797247+0000","last_active":"2023-10-30T13:35:10.993612+0000","last_peered":"2023-10-30T13:35:10.993612+0000","last_clean":"2023-10-30T13:35:10.993612+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:35:10.993612+0000","last_undegraded":"2023-10-30T13:35:10.993612+0000","last_fullsized":"2023-10-30T13:35:10.993612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:57:02.556658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.178231+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:35:08.178231+0000","last_peered":"2023-10-30T13:35:08.178231+0000","last_clean":"2023-10-30T13:35:08.178231+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:35:08.178231+0000","last_undegraded":"2023-10-30T13:35:08.178231+0000","last_fullsized":"2023-10-30T13:35:08.178231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993402+0000","last_change":"2023-10-30T13:35:07.773683+0000","last_active":"2023-10-30T13:35:10.993402+0000","last_peered":"2023-10-30T13:35:10.993402+0000","last_clean":"2023-10-30T13:35:10.993402+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:35:10.993402+0000","last_undegraded":"2023-10-30T13:35:10.993402+0000","last_fullsized":"2023-10-30T13:35:10.993402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:30:03.553514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291471+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:35:13.291471+0000","last_peered":"2023-10-30T13:35:13.291471+0000","last_clean":"2023-10-30T13:35:13.291471+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:35:13.291471+0000","last_undegraded":"2023-10-30T13:35:13.291471+0000","last_fullsized":"2023-10-30T13:35:13.291471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993760+0000","last_change":"2023-10-30T13:35:07.791266+0000","last_active":"2023-10-30T13:35:10.993760+0000","last_peered":"2023-10-30T13:35:10.993760+0000","last_clean":"2023-10-30T13:35:10.993760+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:35:10.993760+0000","last_undegraded":"2023-10-30T13:35:10.993760+0000","last_fullsized":"2023-10-30T13:35:10.993760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:20.086559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750401+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:35:08.750401+0000","last_peered":"2023-10-30T13:35:08.750401+0000","last_clean":"2023-10-30T13:35:08.750401+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:35:08.750401+0000","last_undegraded":"2023-10-30T13:35:08.750401+0000","last_fullsized":"2023-10-30T13:35:08.750401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994310+0000","last_change":"2023-10-30T13:35:07.794747+0000","last_active":"2023-10-30T13:35:10.994310+0000","last_peered":"2023-10-30T13:35:10.994310+0000","last_clean":"2023-10-30T13:35:10.994310+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:35:10.994310+0000","last_undegraded":"2023-10-30T13:35:10.994310+0000","last_fullsized":"2023-10-30T13:35:10.994310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:40.026054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.287725+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:35:13.287725+0000","last_peered":"2023-10-30T13:35:13.287725+0000","last_clean":"2023-10-30T13:35:13.287725+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:35:13.287725+0000","last_undegraded":"2023-10-30T13:35:13.287725+0000","last_fullsized":"2023-10-30T13:35:13.287725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994800+0000","last_change":"2023-10-30T13:35:07.793255+0000","last_active":"2023-10-30T13:35:10.994800+0000","last_peered":"2023-10-30T13:35:10.994800+0000","last_clean":"2023-10-30T13:35:10.994800+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:35:10.994800+0000","last_undegraded":"2023-10-30T13:35:10.994800+0000","last_fullsized":"2023-10-30T13:35:10.994800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:56.491002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236788+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:35:09.236788+0000","last_peered":"2023-10-30T13:35:09.236788+0000","last_clean":"2023-10-30T13:35:09.236788+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:35:09.236788+0000","last_undegraded":"2023-10-30T13:35:09.236788+0000","last_fullsized":"2023-10-30T13:35:09.236788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748659+0000","last_change":"2023-10-30T13:35:07.791920+0000","last_active":"2023-10-30T13:35:08.748659+0000","last_peered":"2023-10-30T13:35:08.748659+0000","last_clean":"2023-10-30T13:35:08.748659+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:35:08.748659+0000","last_undegraded":"2023-10-30T13:35:08.748659+0000","last_fullsized":"2023-10-30T13:35:08.748659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:04.379349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.238228+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:35:09.238228+0000","last_peered":"2023-10-30T13:35:09.238228+0000","last_clean":"2023-10-30T13:35:09.238228+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:35:09.238228+0000","last_undegraded":"2023-10-30T13:35:09.238228+0000","last_fullsized":"2023-10-30T13:35:09.238228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994214+0000","last_change":"2023-10-30T13:35:07.803211+0000","last_active":"2023-10-30T13:35:10.994214+0000","last_peered":"2023-10-30T13:35:10.994214+0000","last_clean":"2023-10-30T13:35:10.994214+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:35:10.994214+0000","last_undegraded":"2023-10-30T13:35:10.994214+0000","last_fullsized":"2023-10-30T13:35:10.994214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:55.371993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768067+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:35:07.768067+0000","last_peered":"2023-10-30T13:35:07.768067+0000","last_clean":"2023-10-30T13:35:07.768067+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:35:07.768067+0000","last_undegraded":"2023-10-30T13:35:07.768067+0000","last_fullsized":"2023-10-30T13:35:07.768067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173422+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:35:08.173422+0000","last_peered":"2023-10-30T13:35:08.173422+0000","last_clean":"2023-10-30T13:35:08.173422+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:35:08.173422+0000","last_undegraded":"2023-10-30T13:35:08.173422+0000","last_fullsized":"2023-10-30T13:35:08.173422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531304+0000","last_change":"2023-10-30T13:35:07.792082+0000","last_active":"2023-10-30T13:35:09.531304+0000","last_peered":"2023-10-30T13:35:09.531304+0000","last_clean":"2023-10-30T13:35:09.531304+0000","last_became_active":"2023-10-30T13:35:07.791831+0000","last_became_peered":"2023-10-30T13:35:07.791831+0000","last_unstale":"2023-10-30T13:35:09.531304+0000","last_undegraded":"2023-10-30T13:35:09.531304+0000","last_fullsized":"2023-10-30T13:35:09.531304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:47.863235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.250988+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:35:09.250988+0000","last_peered":"2023-10-30T13:35:09.250988+0000","last_clean":"2023-10-30T13:35:09.250988+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:35:09.250988+0000","last_undegraded":"2023-10-30T13:35:09.250988+0000","last_fullsized":"2023-10-30T13:35:09.250988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298228+0000","last_change":"2023-10-30T13:35:07.760132+0000","last_active":"2023-10-30T13:35:09.298228+0000","last_peered":"2023-10-30T13:35:09.298228+0000","last_clean":"2023-10-30T13:35:09.298228+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:35:09.298228+0000","last_undegraded":"2023-10-30T13:35:09.298228+0000","last_fullsized":"2023-10-30T13:35:09.298228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:14.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.238219+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:35:09.238219+0000","last_peered":"2023-10-30T13:35:09.238219+0000","last_clean":"2023-10-30T13:35:09.238219+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:35:09.238219+0000","last_undegraded":"2023-10-30T13:35:09.238219+0000","last_fullsized":"2023-10-30T13:35:09.238219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.750958+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:53.697186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.173375+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:35:08.173375+0000","last_peered":"2023-10-30T13:35:08.173375+0000","last_clean":"2023-10-30T13:35:08.173375+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:35:08.173375+0000","last_undegraded":"2023-10-30T13:35:08.173375+0000","last_fullsized":"2023-10-30T13:35:08.173375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750179+0000","last_change":"2023-10-30T13:35:07.768640+0000","last_active":"2023-10-30T13:35:08.750179+0000","last_peered":"2023-10-30T13:35:08.750179+0000","last_clean":"2023-10-30T13:35:08.750179+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:35:08.750179+0000","last_undegraded":"2023-10-30T13:35:08.750179+0000","last_fullsized":"2023-10-30T13:35:08.750179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:26.852057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743612+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:35:08.743612+0000","last_peered":"2023-10-30T13:35:08.743612+0000","last_clean":"2023-10-30T13:35:08.743612+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:35:08.743612+0000","last_undegraded":"2023-10-30T13:35:08.743612+0000","last_fullsized":"2023-10-30T13:35:08.743612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749093+0000","last_change":"2023-10-30T13:35:07.805535+0000","last_active":"2023-10-30T13:35:08.749093+0000","last_peered":"2023-10-30T13:35:08.749093+0000","last_clean":"2023-10-30T13:35:08.749093+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:35:08.749093+0000","last_undegraded":"2023-10-30T13:35:08.749093+0000","last_fullsized":"2023-10-30T13:35:08.749093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:06.549889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.251374+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:35:09.251374+0000","last_peered":"2023-10-30T13:35:09.251374+0000","last_clean":"2023-10-30T13:35:09.251374+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:35:09.251374+0000","last_undegraded":"2023-10-30T13:35:09.251374+0000","last_fullsized":"2023-10-30T13:35:09.251374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743474+0000","last_change":"2023-10-30T13:35:07.754171+0000","last_active":"2023-10-30T13:35:08.743474+0000","last_peered":"2023-10-30T13:35:08.743474+0000","last_clean":"2023-10-30T13:35:08.743474+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:35:08.743474+0000","last_undegraded":"2023-10-30T13:35:08.743474+0000","last_fullsized":"2023-10-30T13:35:08.743474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:50.849695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744041+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:35:08.744041+0000","last_peered":"2023-10-30T13:35:08.744041+0000","last_clean":"2023-10-30T13:35:08.744041+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:35:08.744041+0000","last_undegraded":"2023-10-30T13:35:08.744041+0000","last_fullsized":"2023-10-30T13:35:08.744041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531754+0000","last_change":"2023-10-30T13:35:07.794804+0000","last_active":"2023-10-30T13:35:09.531754+0000","last_peered":"2023-10-30T13:35:09.531754+0000","last_clean":"2023-10-30T13:35:09.531754+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:35:09.531754+0000","last_undegraded":"2023-10-30T13:35:09.531754+0000","last_fullsized":"2023-10-30T13:35:09.531754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:00.913592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.287281+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:35:13.287281+0000","last_peered":"2023-10-30T13:35:13.287281+0000","last_clean":"2023-10-30T13:35:13.287281+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:35:13.287281+0000","last_undegraded":"2023-10-30T13:35:13.287281+0000","last_fullsized":"2023-10-30T13:35:13.287281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298809+0000","last_change":"2023-10-30T13:35:07.752077+0000","last_active":"2023-10-30T13:35:09.298809+0000","last_peered":"2023-10-30T13:35:09.298809+0000","last_clean":"2023-10-30T13:35:09.298809+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:35:09.298809+0000","last_undegraded":"2023-10-30T13:35:09.298809+0000","last_fullsized":"2023-10-30T13:35:09.298809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:07.661975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.250947+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:35:09.250947+0000","last_peered":"2023-10-30T13:35:09.250947+0000","last_clean":"2023-10-30T13:35:09.250947+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:35:09.250947+0000","last_undegraded":"2023-10-30T13:35:09.250947+0000","last_fullsized":"2023-10-30T13:35:09.250947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748158+0000","last_change":"2023-10-30T13:35:07.759389+0000","last_active":"2023-10-30T13:35:08.748158+0000","last_peered":"2023-10-30T13:35:08.748158+0000","last_clean":"2023-10-30T13:35:08.748158+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:35:08.748158+0000","last_undegraded":"2023-10-30T13:35:08.748158+0000","last_fullsized":"2023-10-30T13:35:08.748158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:15.536615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743540+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:35:08.743540+0000","last_peered":"2023-10-30T13:35:08.743540+0000","last_clean":"2023-10-30T13:35:08.743540+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:35:08.743540+0000","last_undegraded":"2023-10-30T13:35:08.743540+0000","last_fullsized":"2023-10-30T13:35:08.743540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531288+0000","last_change":"2023-10-30T13:35:07.753673+0000","last_active":"2023-10-30T13:35:09.531288+0000","last_peered":"2023-10-30T13:35:09.531288+0000","last_clean":"2023-10-30T13:35:09.531288+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:35:09.531288+0000","last_undegraded":"2023-10-30T13:35:09.531288+0000","last_fullsized":"2023-10-30T13:35:09.531288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:27:18.430183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743863+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:35:08.743863+0000","last_peered":"2023-10-30T13:35:08.743863+0000","last_clean":"2023-10-30T13:35:08.743863+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:35:08.743863+0000","last_undegraded":"2023-10-30T13:35:08.743863+0000","last_fullsized":"2023-10-30T13:35:08.743863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531465+0000","last_change":"2023-10-30T13:35:07.755302+0000","last_active":"2023-10-30T13:35:09.531465+0000","last_peered":"2023-10-30T13:35:09.531465+0000","last_clean":"2023-10-30T13:35:09.531465+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:35:09.531465+0000","last_undegraded":"2023-10-30T13:35:09.531465+0000","last_fullsized":"2023-10-30T13:35:09.531465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:31.543587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.292960+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:35:13.292960+0000","last_peered":"2023-10-30T13:35:13.292960+0000","last_clean":"2023-10-30T13:35:13.292960+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:35:13.292960+0000","last_undegraded":"2023-10-30T13:35:13.292960+0000","last_fullsized":"2023-10-30T13:35:13.292960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992766+0000","last_change":"2023-10-30T13:35:07.780297+0000","last_active":"2023-10-30T13:35:10.992766+0000","last_peered":"2023-10-30T13:35:10.992766+0000","last_clean":"2023-10-30T13:35:10.992766+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:35:10.992766+0000","last_undegraded":"2023-10-30T13:35:10.992766+0000","last_fullsized":"2023-10-30T13:35:10.992766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:48.971885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748946+0000","last_change":"2023-10-30T13:35:07.806203+0000","last_active":"2023-10-30T13:35:08.748946+0000","last_peered":"2023-10-30T13:35:08.748946+0000","last_clean":"2023-10-30T13:35:08.748946+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:35:08.748946+0000","last_undegraded":"2023-10-30T13:35:08.748946+0000","last_fullsized":"2023-10-30T13:35:08.748946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:34.175218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294221+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:35:13.294221+0000","last_peered":"2023-10-30T13:35:13.294221+0000","last_clean":"2023-10-30T13:35:13.294221+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:35:13.294221+0000","last_undegraded":"2023-10-30T13:35:13.294221+0000","last_fullsized":"2023-10-30T13:35:13.294221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748564+0000","last_change":"2023-10-30T13:35:07.791319+0000","last_active":"2023-10-30T13:35:08.748564+0000","last_peered":"2023-10-30T13:35:08.748564+0000","last_clean":"2023-10-30T13:35:08.748564+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:35:08.748564+0000","last_undegraded":"2023-10-30T13:35:08.748564+0000","last_fullsized":"2023-10-30T13:35:08.748564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:42.560856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294002+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:35:13.294002+0000","last_peered":"2023-10-30T13:35:13.294002+0000","last_clean":"2023-10-30T13:35:13.294002+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:35:13.294002+0000","last_undegraded":"2023-10-30T13:35:13.294002+0000","last_fullsized":"2023-10-30T13:35:13.294002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298499+0000","last_change":"2023-10-30T13:35:07.790084+0000","last_active":"2023-10-30T13:35:09.298499+0000","last_peered":"2023-10-30T13:35:09.298499+0000","last_clean":"2023-10-30T13:35:09.298499+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:35:09.298499+0000","last_undegraded":"2023-10-30T13:35:09.298499+0000","last_fullsized":"2023-10-30T13:35:09.298499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:45.193072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531782+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:35:09.531782+0000","last_peered":"2023-10-30T13:35:09.531782+0000","last_clean":"2023-10-30T13:35:09.531782+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:35:09.531782+0000","last_undegraded":"2023-10-30T13:35:09.531782+0000","last_fullsized":"2023-10-30T13:35:09.531782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531695+0000","last_change":"2023-10-30T13:35:07.782341+0000","last_active":"2023-10-30T13:35:09.531695+0000","last_peered":"2023-10-30T13:35:09.531695+0000","last_clean":"2023-10-30T13:35:09.531695+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:35:09.531695+0000","last_undegraded":"2023-10-30T13:35:09.531695+0000","last_fullsized":"2023-10-30T13:35:09.531695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:19:06.392683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.292830+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:35:13.292830+0000","last_peered":"2023-10-30T13:35:13.292830+0000","last_clean":"2023-10-30T13:35:13.292830+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:35:13.292830+0000","last_undegraded":"2023-10-30T13:35:13.292830+0000","last_fullsized":"2023-10-30T13:35:13.292830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":806912,"data_stored":663801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]}]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":74463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":72851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":74463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":72941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":72977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":663857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:35:19.632 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T13:35:19.633 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T13:35:19.633 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T13:35:19.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T13:35:19.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e858c9640 1 Processor -- start 2023-10-30T13:35:19.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e858c9640 1 -- start start 2023-10-30T13:35:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e858c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e80153f00 0x7f8e80159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e858c9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e800cfe40 con 0x7f8e800c9ad0 2023-10-30T13:35:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e858c9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e800caea0 con 0x7f8e8006a0f0 2023-10-30T13:35:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e858c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800ca010 con 0x7f8e80153f00 2023-10-30T13:35:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e77fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8e800c9ad0 0x7f8e801537f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51982/0 (socket says 172.21.15.31:51982) 2023-10-30T13:35:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e77fff640 1 -- 172.21.15.31:0/1877182041 learned_addr learned my addr 172.21.15.31:0/1877182041 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7f7fe640 1 --2- 172.21.15.31:0/1877182041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e80153f00 0x7f8e80159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737000474 0 0) 0x7f8e800cfe40 con 0x7f8e800c9ad0 2023-10-30T13:35:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e6c003530 con 0x7f8e800c9ad0 2023-10-30T13:35:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260100663 0 0) 0x7f8e800caea0 con 0x7f8e8006a0f0 2023-10-30T13:35:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e800cfe40 con 0x7f8e8006a0f0 2023-10-30T13:35:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7f7fe640 1 -- 172.21.15.31:0/1877182041 >> v1:172.21.15.33:6789/0 conn(0x7f8e8006a0f0 legacy=0x7f8e8006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2286178105 0 0) 0x7f8e6c003530 con 0x7f8e800c9ad0 2023-10-30T13:35:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7f7fe640 1 -- 172.21.15.31:0/1877182041 >> v1:172.21.15.31:6789/0 conn(0x7f8e800c9ad0 legacy=0x7f8e801537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.808+0000 7f8e7f7fe640 1 -- 172.21.15.31:0/1877182041 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e800ca2b0 con 0x7f8e80153f00 2023-10-30T13:35:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e7f7fe640 1 --2- 172.21.15.31:0/1877182041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e80153f00 0x7f8e80159a40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009dc0 tx=0x7f8e7801f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e04419a9ae5e41 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68001680 con 0x7f8e80153f00 2023-10-30T13:35:19.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e780a8c10 con 0x7f8e80153f00 2023-10-30T13:35:19.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e7e7fc640 1 -- 172.21.15.31:0/1877182041 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780b0440 con 0x7f8e80153f00 2023-10-30T13:35:19.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e858c9640 1 -- 172.21.15.31:0/1877182041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e80153f00 msgr2=0x7f8e80159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1877182041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e80153f00 0x7f8e80159a40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009dc0 tx=0x7f8e7801f080 comp rx=0 tx=0).stop 2023-10-30T13:35:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e858c9640 1 -- 172.21.15.31:0/1877182041 shutdown_connections 2023-10-30T13:35:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1877182041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e80153f00 0x7f8e80159a40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e858c9640 1 -- 172.21.15.31:0/1877182041 >> 172.21.15.31:0/1877182041 conn(0x7f8e800cf150 msgr2=0x7f8e800cf530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e858c9640 1 -- 172.21.15.31:0/1877182041 shutdown_connections 2023-10-30T13:35:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.812+0000 7f8e858c9640 1 -- 172.21.15.31:0/1877182041 wait complete. 2023-10-30T13:35:19.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 Processor -- start 2023-10-30T13:35:19.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 -- start start 2023-10-30T13:35:19.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e8006a0f0 0x7f8e800c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e800c9ad0 0x7f8e8015cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e80153f00 0x7f8e801ef050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8015dd40 con 0x7f8e8006a0f0 2023-10-30T13:35:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e80156420 con 0x7f8e80153f00 2023-10-30T13:35:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800c7c30 con 0x7f8e800c9ad0 2023-10-30T13:35:19.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e8006a0f0 0x7f8e800c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e8006a0f0 0x7f8e800c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36206/0 (socket says 172.21.15.31:36206) 2023-10-30T13:35:19.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 -- 172.21.15.31:0/1233633245 learned_addr learned my addr 172.21.15.31:0/1233633245 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:19.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e77fff640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e800c9ad0 0x7f8e8015cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7f7fe640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e80153f00 0x7f8e801ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 -- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e800c9ad0 msgr2=0x7f8e8015cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e800c9ad0 0x7f8e8015cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 -- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e80153f00 msgr2=0x7f8e801ef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e80153f00 0x7f8e801ef050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 -- 172.21.15.31:0/1233633245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78009a70 con 0x7f8e8006a0f0 2023-10-30T13:35:19.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7effd640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e8006a0f0 0x7f8e800c76f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c4f0 tx=0x7f8e7000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7cff9640 1 -- 172.21.15.31:0/1233633245 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700052d0 con 0x7f8e8006a0f0 2023-10-30T13:35:19.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7cff9640 1 -- 172.21.15.31:0/1233633245 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e70005430 con 0x7f8e8006a0f0 2023-10-30T13:35:19.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e7cff9640 1 -- 172.21.15.31:0/1233633245 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70012850 con 0x7f8e8006a0f0 2023-10-30T13:35:19.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8015d7c0 con 0x7f8e8006a0f0 2023-10-30T13:35:19.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.816+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e80155d60 con 0x7f8e8006a0f0 2023-10-30T13:35:19.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.820+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8004e170 con 0x7f8e8006a0f0 2023-10-30T13:35:19.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.820+0000 7f8e7cff9640 1 -- 172.21.15.31:0/1233633245 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e700129b0 con 0x7f8e8006a0f0 2023-10-30T13:35:19.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.820+0000 7f8e7cff9640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8e600664a0 0x7f8e60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:19.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.820+0000 7f8e7cff9640 1 -- 172.21.15.31:0/1233633245 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e700837b0 con 0x7f8e8006a0f0 2023-10-30T13:35:19.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.820+0000 7f8e77fff640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8e600664a0 0x7f8e60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:19.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.820+0000 7f8e77fff640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8e600664a0 0x7f8e60068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8e800b1ef0 tx=0x7f8e68001a60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:19.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:19.824+0000 7f8e7cff9640 1 -- 172.21.15.31:0/1233633245 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e70050d70 con 0x7f8e8006a0f0 2023-10-30T13:35:20.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.012+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8e8015ded0 con 0x7f8e8006a0f0 2023-10-30T13:35:20.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.012+0000 7f8e7cff9640 1 -- 172.21.15.31:0/1233633245 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f8e70054390 con 0x7f8e8006a0f0 2023-10-30T13:35:20.018 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:20.018 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T13:35:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8e600664a0 msgr2=0x7f8e60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8e600664a0 0x7f8e60068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8e800b1ef0 tx=0x7f8e68001a60 comp rx=0 tx=0).stop 2023-10-30T13:35:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e8006a0f0 msgr2=0x7f8e800c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e8006a0f0 0x7f8e800c76f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c4f0 tx=0x7f8e7000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 shutdown_connections 2023-10-30T13:35:20.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8e600664a0 0x7f8e60068960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e80153f00 0x7f8e801ef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8e800c9ad0 0x7f8e8015cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 --2- 172.21.15.31:0/1233633245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e8006a0f0 0x7f8e800c76f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 >> 172.21.15.31:0/1233633245 conn(0x7f8e800cf150 msgr2=0x7f8e800b4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:20.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 shutdown_connections 2023-10-30T13:35:20.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.016+0000 7f8e858c9640 1 -- 172.21.15.31:0/1233633245 wait complete. 2023-10-30T13:35:20.039 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T13:35:20.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:20.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb401f3e640 1 Processor -- start 2023-10-30T13:35:20.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb401f3e640 1 -- start start 2023-10-30T13:35:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb401f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc0c4a10 0x7fb3fc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb401f3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3fc0c42a0 0x7fb3fc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb401f3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc159f20 con 0x7fb3fc153c80 2023-10-30T13:35:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb401f3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc15dec0 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb401f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1599a0 con 0x7fb3fc0c4a10 2023-10-30T13:35:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3fbfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3fc0c42a0 0x7fb3fc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3fbfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3fc0c42a0 0x7fb3fc0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36220/0 (socket says 172.21.15.31:36220) 2023-10-30T13:35:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3fbfff640 1 -- 172.21.15.31:0/1094619212 learned_addr learned my addr 172.21.15.31:0/1094619212 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3faffd640 1 --2- 172.21.15.31:0/1094619212 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc0c4a10 0x7fb3fc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3fbfff640 1 -- 172.21.15.31:0/1094619212 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc0c4a10 msgr2=0x7fb3fc15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3fbfff640 1 --2- 172.21.15.31:0/1094619212 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc0c4a10 0x7fb3fc15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3fbfff640 1 -- 172.21.15.31:0/1094619212 >> v1:172.21.15.33:6789/0 conn(0x7fb3fc153c80 legacy=0x7fb3fc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.180+0000 7fb3fbfff640 1 -- 172.21.15.31:0/1094619212 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc15d620 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb3fbfff640 1 --2- 172.21.15.31:0/1094619212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc0ca1e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009e50 tx=0x7fb3f401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e24820965189e79a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb3fa7fc640 1 -- 172.21.15.31:0/1094619212 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40af400 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb3fa7fc640 1 -- 172.21.15.31:0/1094619212 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f40ab050 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb3fa7fc640 1 -- 172.21.15.31:0/1094619212 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40af780 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 -- 172.21.15.31:0/1094619212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 msgr2=0x7fb3fc0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 --2- 172.21.15.31:0/1094619212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc0ca1e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009e50 tx=0x7fb3f401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 -- 172.21.15.31:0/1094619212 shutdown_connections 2023-10-30T13:35:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 --2- 172.21.15.31:0/1094619212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc0ca1e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 --2- 172.21.15.31:0/1094619212 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc0c4a10 0x7fb3fc15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 -- 172.21.15.31:0/1094619212 >> 172.21.15.31:0/1094619212 conn(0x7fb3fc0715c0 msgr2=0x7fb3fc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 -- 172.21.15.31:0/1094619212 shutdown_connections 2023-10-30T13:35:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 -- 172.21.15.31:0/1094619212 wait complete. 2023-10-30T13:35:20.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.184+0000 7fb401f3e640 1 Processor -- start 2023-10-30T13:35:20.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 -- start start 2023-10-30T13:35:20.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc1f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3fc0c4a10 0x7fb3fc15c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc153c80 0x7fb3fc0c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1ef340 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1ee7d0 con 0x7fb3fc0c4a10 2023-10-30T13:35:20.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1ef760 con 0x7fb3fc153c80 2023-10-30T13:35:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc1f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc1f2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36224/0 (socket says 172.21.15.31:36224) 2023-10-30T13:35:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 -- 172.21.15.31:0/575210414 learned_addr learned my addr 172.21.15.31:0/575210414 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fbfff640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc153c80 0x7fb3fc0c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3faffd640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3fc0c4a10 0x7fb3fc15c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 -- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc153c80 msgr2=0x7fb3fc0c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc153c80 0x7fb3fc0c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 -- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3fc0c4a10 msgr2=0x7fb3fc15c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3fc0c4a10 0x7fb3fc15c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 -- 172.21.15.31:0/575210414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4009b30 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3fb7fe640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc1f2720 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00b440 tx=0x7fb3ec00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3e3fff640 1 -- 172.21.15.31:0/575210414 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec011890 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3e3fff640 1 -- 172.21.15.31:0/575210414 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec0119f0 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb3e3fff640 1 -- 172.21.15.31:0/575210414 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec011da0 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc1f2c60 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc1ee9e0 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.188+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc15bf80 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.192+0000 7fb3e3fff640 1 -- 172.21.15.31:0/575210414 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3ec01a400 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.192+0000 7fb3e3fff640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3e80664a0 0x7fb3e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.192+0000 7fb3e3fff640 1 -- 172.21.15.31:0/575210414 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3ec083dc0 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.192+0000 7fb3faffd640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3e80664a0 0x7fb3e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.192+0000 7fb3faffd640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3e80664a0 0x7fb3e8068960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc14dc60 tx=0x7fb3f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.196+0000 7fb3e3fff640 1 -- 172.21.15.31:0/575210414 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3ec0512f0 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.352+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3fc04e170 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb3e3fff640 1 -- 172.21.15.31:0/575210414 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fb3ec054910 con 0x7fb3fc0c42a0 2023-10-30T13:35:20.359 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:20.360 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:20.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3e80664a0 msgr2=0x7fb3e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3e80664a0 0x7fb3e8068960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc14dc60 tx=0x7fb3f0008040 comp rx=0 tx=0).stop 2023-10-30T13:35:20.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 msgr2=0x7fb3fc1f2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc1f2720 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00b440 tx=0x7fb3ec00b900 comp rx=0 tx=0).stop 2023-10-30T13:35:20.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 shutdown_connections 2023-10-30T13:35:20.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3e80664a0 0x7fb3e8068960 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3fc153c80 0x7fb3fc0c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3fc0c4a10 0x7fb3fc15c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 --2- 172.21.15.31:0/575210414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3fc0c42a0 0x7fb3fc1f2720 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 >> 172.21.15.31:0/575210414 conn(0x7fb3fc0715c0 msgr2=0x7fb3fc0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:20.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 shutdown_connections 2023-10-30T13:35:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.356+0000 7fb401f3e640 1 -- 172.21.15.31:0/575210414 wait complete. 2023-10-30T13:35:20.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:35:05.304460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:35:06.652927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:35:20.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:35:20.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c1609e640 1 Processor -- start 2023-10-30T13:35:20.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c1609e640 1 -- start start 2023-10-30T13:35:20.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c1609e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f6c10153ba0 0x7f6c100cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c1609e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c10153f80 con 0x7f6c100c4410 2023-10-30T13:35:20.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c1609e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c10159df0 con 0x7f6c100c4810 2023-10-30T13:35:20.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c1609e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c101597c0 con 0x7f6c10153ba0 2023-10-30T13:35:20.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0effd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f6c100c4410 0x7f6c1015cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51994/0 (socket says 172.21.15.31:51994) 2023-10-30T13:35:20.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0effd640 1 -- 172.21.15.31:0/4248178347 learned_addr learned my addr 172.21.15.31:0/4248178347 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:20.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0f7fe640 1 --2- 172.21.15.31:0/4248178347 >> v2:172.21.15.33:3300/0 conn(0x7f6c10153ba0 0x7f6c100cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049234093 0 0) 0x7f6c10153f80 con 0x7f6c100c4410 2023-10-30T13:35:20.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf8003530 con 0x7f6c100c4410 2023-10-30T13:35:20.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871686379 0 0) 0x7f6bf8003530 con 0x7f6c100c4410 2023-10-30T13:35:20.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 >> v2:172.21.15.33:3300/0 conn(0x7f6c10153ba0 msgr2=0x7f6c100cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:20.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 --2- 172.21.15.31:0/4248178347 >> v2:172.21.15.33:3300/0 conn(0x7f6c10153ba0 0x7f6c100cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 >> v1:172.21.15.33:6789/0 conn(0x7f6c100c4810 legacy=0x7f6c100cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1015d580 con 0x7f6c100c4410 2023-10-30T13:35:20.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f6c04001680 con 0x7f6c100c4410 2023-10-30T13:35:20.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 >> v1:172.21.15.31:6789/0 conn(0x7f6c100c4410 legacy=0x7f6c1015cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf8003980 0x7f6bf8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf80047d0 0x7f6bf8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf8005190 0x7f6bf800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c101597c0 con 0x7f6bf80047d0 2023-10-30T13:35:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf800c130 con 0x7f6bf8005190 2023-10-30T13:35:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf800c2f0 con 0x7f6bf8003980 2023-10-30T13:35:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0effd640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf80047d0 0x7f6bf8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.524+0000 7f6c0f7fe640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf8003980 0x7f6bf8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0ffff640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf8005190 0x7f6bf800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0effd640 1 -- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf8003980 msgr2=0x7f6bf8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0effd640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf8003980 0x7f6bf8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0effd640 1 -- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf8005190 msgr2=0x7f6bf800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0effd640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf8005190 0x7f6bf800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0effd640 1 -- 172.21.15.31:0/4248178347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8003530 con 0x7f6bf80047d0 2023-10-30T13:35:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0ffff640 1 -- 172.21.15.31:0/4248178347 reap_dead start 2023-10-30T13:35:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0f7fe640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf8003980 0x7f6bf8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:20.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0effd640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf80047d0 0x7f6bf8004c50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400c1c0 tx=0x7f6c0401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f6993c15b4b4b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04005ae0 con 0x7f6bf80047d0 2023-10-30T13:35:20.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c04005c40 con 0x7f6bf80047d0 2023-10-30T13:35:20.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c0e7fc640 1 -- 172.21.15.31:0/4248178347 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c040a5a20 con 0x7f6bf80047d0 2023-10-30T13:35:20.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c1609e640 1 -- 172.21.15.31:0/4248178347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf80047d0 msgr2=0x7f6bf8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c1609e640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf80047d0 0x7f6bf8004c50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400c1c0 tx=0x7f6c0401cbf0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c1609e640 1 -- 172.21.15.31:0/4248178347 shutdown_connections 2023-10-30T13:35:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c1609e640 1 --2- 172.21.15.31:0/4248178347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf80047d0 0x7f6bf8004c50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c1609e640 1 -- 172.21.15.31:0/4248178347 >> 172.21.15.31:0/4248178347 conn(0x7f6c10071790 msgr2=0x7f6c10071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c1609e640 1 -- 172.21.15.31:0/4248178347 shutdown_connections 2023-10-30T13:35:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.528+0000 7f6c1609e640 1 -- 172.21.15.31:0/4248178347 wait complete. 2023-10-30T13:35:20.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 Processor -- start 2023-10-30T13:35:20.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 -- start start 2023-10-30T13:35:20.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf80047d0 0x7f6c10151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c1015c870 0x7f6c100cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c100cd770 0x7f6c10152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c10151fb0 con 0x7f6c1015c870 2023-10-30T13:35:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c100cdc20 con 0x7f6c100cd770 2023-10-30T13:35:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1014f610 con 0x7f6bf80047d0 2023-10-30T13:35:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c1015c870 0x7f6c100cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c1015c870 0x7f6c100cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36240/0 (socket says 172.21.15.31:36240) 2023-10-30T13:35:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 -- 172.21.15.31:0/506838223 learned_addr learned my addr 172.21.15.31:0/506838223 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0f7fe640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf80047d0 0x7f6c10151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0ffff640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c100cd770 0x7f6c10152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 -- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf80047d0 msgr2=0x7f6c10151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf80047d0 0x7f6c10151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 -- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c100cd770 msgr2=0x7f6c10152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c100cd770 0x7f6c10152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 -- 172.21.15.31:0/506838223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1014f8d0 con 0x7f6c1015c870 2023-10-30T13:35:20.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0f7fe640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf80047d0 0x7f6c10151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:20.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c0effd640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c1015c870 0x7f6c100cd230 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6c04001940 tx=0x7f6bf8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6bf7fff640 1 -- 172.21.15.31:0/506838223 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c040b4070 con 0x7f6c1015c870 2023-10-30T13:35:20.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6bf7fff640 1 -- 172.21.15.31:0/506838223 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c040015c0 con 0x7f6c1015c870 2023-10-30T13:35:20.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6bf7fff640 1 -- 172.21.15.31:0/506838223 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c040a5a00 con 0x7f6c1015c870 2023-10-30T13:35:20.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.532+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c101530a0 con 0x7f6c1015c870 2023-10-30T13:35:20.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.536+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c10153660 con 0x7f6c1015c870 2023-10-30T13:35:20.546 INFO:tasks.ceph.mds.b.smithi033.stderr:2023-10-30T13:35:20.539+0000 7f01d8981640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:35:20.546 INFO:tasks.ceph.mds.b.smithi033.stderr:2023-10-30T13:35:20.539+0000 7f01d8981640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:35:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.536+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1004e170 con 0x7f6c1015c870 2023-10-30T13:35:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.536+0000 7f6bf7fff640 1 -- 172.21.15.31:0/506838223 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c040a5b60 con 0x7f6c1015c870 2023-10-30T13:35:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.536+0000 7f6bf7fff640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6bfc0664d0 0x7f6bfc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.536+0000 7f6bf7fff640 1 -- 172.21.15.31:0/506838223 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c0411d6b0 con 0x7f6c1015c870 2023-10-30T13:35:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.536+0000 7f6c0f7fe640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6bfc0664d0 0x7f6bfc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.540+0000 7f6c0f7fe640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6bfc0664d0 0x7f6bfc068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6c1004fb10 tx=0x7f6c00008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.540+0000 7f6bf7fff640 1 -- 172.21.15.31:0/506838223 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c040eaaa0 con 0x7f6c1015c870 2023-10-30T13:35:20.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.696+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c100cabb0 con 0x7f6c1015c870 2023-10-30T13:35:20.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.696+0000 7f6bf7fff640 1 -- 172.21.15.31:0/506838223 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c040ee0c0 con 0x7f6c1015c870 2023-10-30T13:35:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:35:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6bfc0664d0 msgr2=0x7f6bfc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6bfc0664d0 0x7f6bfc068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6c1004fb10 tx=0x7f6c00008040 comp rx=0 tx=0).stop 2023-10-30T13:35:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c1015c870 msgr2=0x7f6c100cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c1015c870 0x7f6c100cd230 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6c04001940 tx=0x7f6bf8005800 comp rx=0 tx=0).stop 2023-10-30T13:35:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 shutdown_connections 2023-10-30T13:35:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6bfc0664d0 0x7f6bfc068990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c100cd770 0x7f6c10152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c1015c870 0x7f6c100cd230 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 --2- 172.21.15.31:0/506838223 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6bf80047d0 0x7f6c10151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 >> 172.21.15.31:0/506838223 conn(0x7f6c10071790 msgr2=0x7f6c10158020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:20.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 shutdown_connections 2023-10-30T13:35:20.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.700+0000 7f6c1609e640 1 -- 172.21.15.31:0/506838223 wait complete. 2023-10-30T13:35:20.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:35:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.864+0000 7f97f3f2a640 1 Processor -- start 2023-10-30T13:35:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.864+0000 7f97f3f2a640 1 -- start start 2023-10-30T13:35:20.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f3f2a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec153ba0 0x7f97ec0cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f3f2a640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f97ec0c4410 0x7f97ec15a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f3f2a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec153f80 con 0x7f97ec0c4810 2023-10-30T13:35:20.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f3f2a640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec159df0 con 0x7f97ec0c4410 2023-10-30T13:35:20.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f3f2a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec15a7c0 con 0x7f97ec153ba0 2023-10-30T13:35:20.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f24c7640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f97ec0c4810 0x7f97ec15cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:45892/0 (socket says 172.21.15.31:45892) 2023-10-30T13:35:20.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f24c7640 1 -- 172.21.15.31:0/2580665556 learned_addr learned my addr 172.21.15.31:0/2580665556 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:20.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f1cc6640 1 --2- 172.21.15.31:0/2580665556 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec153ba0 0x7f97ec0cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f14c5640 1 --2- 172.21.15.31:0/2580665556 >> v2:172.21.15.33:3300/0 conn(0x7f97ec0c4410 0x7f97ec15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f0cc4640 1 -- 172.21.15.31:0/2580665556 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890361587 0 0) 0x7f97ec153f80 con 0x7f97ec0c4810 2023-10-30T13:35:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f0cc4640 1 -- 172.21.15.31:0/2580665556 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d8003530 con 0x7f97ec0c4810 2023-10-30T13:35:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f1cc6640 1 -- 172.21.15.31:0/2580665556 >> v2:172.21.15.33:3300/0 conn(0x7f97ec0c4410 msgr2=0x7f97ec15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f1cc6640 1 --2- 172.21.15.31:0/2580665556 >> v2:172.21.15.33:3300/0 conn(0x7f97ec0c4410 0x7f97ec15a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f1cc6640 1 -- 172.21.15.31:0/2580665556 >> v1:172.21.15.33:6789/0 conn(0x7f97ec0c4810 legacy=0x7f97ec15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f1cc6640 1 -- 172.21.15.31:0/2580665556 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec1597c0 con 0x7f97ec153ba0 2023-10-30T13:35:20.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.868+0000 7f97f14c5640 1 --2- 172.21.15.31:0/2580665556 >> v2:172.21.15.33:3300/0 conn(0x7f97ec0c4410 0x7f97ec15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:20.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f1cc6640 1 --2- 172.21.15.31:0/2580665556 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec153ba0 0x7f97ec0cb4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97e80097c0 tx=0x7f97e801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbb79299d319135 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f0cc4640 1 -- 172.21.15.31:0/2580665556 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80ac620 con 0x7f97ec153ba0 2023-10-30T13:35:20.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f0cc4640 1 -- 172.21.15.31:0/2580665556 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e80ac780 con 0x7f97ec153ba0 2023-10-30T13:35:20.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f0cc4640 1 -- 172.21.15.31:0/2580665556 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80acb10 con 0x7f97ec153ba0 2023-10-30T13:35:20.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2580665556 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec153ba0 msgr2=0x7f97ec0cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2580665556 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec153ba0 0x7f97ec0cb4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97e80097c0 tx=0x7f97e801c3c0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2580665556 shutdown_connections 2023-10-30T13:35:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2580665556 >> v2:172.21.15.33:3300/0 conn(0x7f97ec0c4410 0x7f97ec15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2580665556 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec153ba0 0x7f97ec0cb4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2580665556 >> 172.21.15.31:0/2580665556 conn(0x7f97ec071790 msgr2=0x7f97ec071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2580665556 shutdown_connections 2023-10-30T13:35:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2580665556 wait complete. 2023-10-30T13:35:20.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 Processor -- start 2023-10-30T13:35:20.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.872+0000 7f97f3f2a640 1 -- start start 2023-10-30T13:35:20.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97ec0c4410 0x7f97ec15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec0c4810 0x7f97ec15b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97ec153ba0 0x7f97ec0ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0c94c0 con 0x7f97ec0c4410 2023-10-30T13:35:20.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0c8f20 con 0x7f97ec153ba0 2023-10-30T13:35:20.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0c8aa0 con 0x7f97ec0c4810 2023-10-30T13:35:20.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97ec0c4410 0x7f97ec15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97ec0c4410 0x7f97ec15cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36254/0 (socket says 172.21.15.31:36254) 2023-10-30T13:35:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 -- 172.21.15.31:0/2878766383 learned_addr learned my addr 172.21.15.31:0/2878766383 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f14c5640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec0c4810 0x7f97ec15b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f24c7640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97ec153ba0 0x7f97ec0ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 -- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec0c4810 msgr2=0x7f97ec15b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec0c4810 0x7f97ec15b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 -- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97ec153ba0 msgr2=0x7f97ec0ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97ec153ba0 0x7f97ec0ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 -- 172.21.15.31:0/2878766383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc009420 con 0x7f97ec0c4410 2023-10-30T13:35:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f24c7640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97ec153ba0 0x7f97ec0ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f1cc6640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97ec0c4410 0x7f97ec15cbf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f97e8019ff0 tx=0x7f97e8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97e27fc640 1 -- 172.21.15.31:0/2878766383 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80ac620 con 0x7f97ec0c4410 2023-10-30T13:35:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97e27fc640 1 -- 172.21.15.31:0/2878766383 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e80b5410 con 0x7f97ec0c4410 2023-10-30T13:35:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e8009420 con 0x7f97ec0c4410 2023-10-30T13:35:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97e27fc640 1 -- 172.21.15.31:0/2878766383 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80b57a0 con 0x7f97ec0c4410 2023-10-30T13:35:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec1f5580 con 0x7f97ec0c4410 2023-10-30T13:35:20.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.876+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec04e170 con 0x7f97ec0c4410 2023-10-30T13:35:20.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.880+0000 7f97e27fc640 1 -- 172.21.15.31:0/2878766383 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97e80b3040 con 0x7f97ec0c4410 2023-10-30T13:35:20.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.880+0000 7f97e27fc640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97d00664a0 0x7f97d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:20.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.880+0000 7f97e27fc640 1 -- 172.21.15.31:0/2878766383 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97e811dfa0 con 0x7f97ec0c4410 2023-10-30T13:35:20.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.880+0000 7f97f14c5640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97d00664a0 0x7f97d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:20.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.880+0000 7f97f14c5640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97d00664a0 0x7f97d0068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97ec0cf0c0 tx=0x7f97dc009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:20.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:20.884+0000 7f97e27fc640 1 -- 172.21.15.31:0/2878766383 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97e80eb380 con 0x7f97ec0c4410 2023-10-30T13:35:21.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.040+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f97ec0b2110 con 0x7f97ec0c4410 2023-10-30T13:35:21.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.040+0000 7f97e27fc640 1 -- 172.21.15.31:0/2878766383 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f97e80ee9a0 con 0x7f97ec0c4410 2023-10-30T13:35:21.045 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-10-30T13:35:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97d00664a0 msgr2=0x7f97d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97d00664a0 0x7f97d0068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97ec0cf0c0 tx=0x7f97dc009190 comp rx=0 tx=0).stop 2023-10-30T13:35:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97ec0c4410 msgr2=0x7f97ec15cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97ec0c4410 0x7f97ec15cbf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f97e8019ff0 tx=0x7f97e8019160 comp rx=0 tx=0).stop 2023-10-30T13:35:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 shutdown_connections 2023-10-30T13:35:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97d00664a0 0x7f97d0068960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97ec153ba0 0x7f97ec0ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97ec0c4810 0x7f97ec15b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 --2- 172.21.15.31:0/2878766383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97ec0c4410 0x7f97ec15cbf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 >> 172.21.15.31:0/2878766383 conn(0x7f97ec071790 msgr2=0x7f97ec157e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 shutdown_connections 2023-10-30T13:35:21.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.044+0000 7f97f3f2a640 1 -- 172.21.15.31:0/2878766383 wait complete. 2023-10-30T13:35:21.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:35:21.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f08748d6640 1 Processor -- start 2023-10-30T13:35:21.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f08748d6640 1 -- start start 2023-10-30T13:35:21.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f08748d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0870153ad0 0x7f08700c0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f08748d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0870159b00 con 0x7f08700c0840 2023-10-30T13:35:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f08748d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087015dc70 con 0x7f087015cdc0 2023-10-30T13:35:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f08748d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0870071330 con 0x7f0870153ad0 2023-10-30T13:35:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f086e59c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0870153ad0 0x7f08700c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f0867fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f08700c0840 0x7f08700723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52004/0 (socket says 172.21.15.31:52004) 2023-10-30T13:35:21.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f0867fff640 1 -- 172.21.15.31:0/4276114193 learned_addr learned my addr 172.21.15.31:0/4276114193 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:21.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f086e59c640 1 -- 172.21.15.31:0/4276114193 >> v1:172.21.15.33:6789/0 conn(0x7f087015cdc0 legacy=0x7f087015d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f086e59c640 1 -- 172.21.15.31:0/4276114193 >> v1:172.21.15.31:6789/0 conn(0x7f08700c0840 legacy=0x7f08700723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.208+0000 7f086e59c640 1 -- 172.21.15.31:0/4276114193 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0870071610 con 0x7f0870153ad0 2023-10-30T13:35:21.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f086e59c640 1 --2- 172.21.15.31:0/4276114193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08700c0300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f08600097c0 tx=0x7f086001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572bb66890fb93b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f086dd9b640 1 -- 172.21.15.31:0/4276114193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08600a8970 con 0x7f0870153ad0 2023-10-30T13:35:21.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f086dd9b640 1 -- 172.21.15.31:0/4276114193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08600a8ad0 con 0x7f0870153ad0 2023-10-30T13:35:21.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f086dd9b640 1 -- 172.21.15.31:0/4276114193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08600a8e60 con 0x7f0870153ad0 2023-10-30T13:35:21.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f08748d6640 1 -- 172.21.15.31:0/4276114193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 msgr2=0x7f08700c0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f08748d6640 1 --2- 172.21.15.31:0/4276114193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08700c0300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f08600097c0 tx=0x7f086001ec70 comp rx=0 tx=0).stop 2023-10-30T13:35:21.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f08748d6640 1 -- 172.21.15.31:0/4276114193 shutdown_connections 2023-10-30T13:35:21.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f08748d6640 1 --2- 172.21.15.31:0/4276114193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08700c0300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f08748d6640 1 -- 172.21.15.31:0/4276114193 >> 172.21.15.31:0/4276114193 conn(0x7f087006da40 msgr2=0x7f087006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f08748d6640 1 -- 172.21.15.31:0/4276114193 shutdown_connections 2023-10-30T13:35:21.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.212+0000 7f08748d6640 1 -- 172.21.15.31:0/4276114193 wait complete. 2023-10-30T13:35:21.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 Processor -- start 2023-10-30T13:35:21.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 -- start start 2023-10-30T13:35:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08700c0840 0x7f08701521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08701e5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f087015cdc0 0x7f08701e6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08700ae2b0 con 0x7f0870153ad0 2023-10-30T13:35:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08701f0040 con 0x7f08700c0840 2023-10-30T13:35:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08700c5360 con 0x7f087015cdc0 2023-10-30T13:35:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08701e5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08701e5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36270/0 (socket says 172.21.15.31:36270) 2023-10-30T13:35:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 -- 172.21.15.31:0/3268681027 learned_addr learned my addr 172.21.15.31:0/3268681027 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f086e59c640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08700c0840 0x7f08701521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f086ed9d640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f087015cdc0 0x7f08701e6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 -- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f087015cdc0 msgr2=0x7f08701e6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f087015cdc0 0x7f08701e6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 -- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08700c0840 msgr2=0x7f08701521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08700c0840 0x7f08701521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 -- 172.21.15.31:0/3268681027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860009420 con 0x7f0870153ad0 2023-10-30T13:35:21.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f086e59c640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08700c0840 0x7f08701521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:21.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0867fff640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08701e5f40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f085800c500 tx=0x7f085800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0866ffd640 1 -- 172.21.15.31:0/3268681027 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0858005cf0 con 0x7f0870153ad0 2023-10-30T13:35:21.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0866ffd640 1 -- 172.21.15.31:0/3268681027 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0858005e50 con 0x7f0870153ad0 2023-10-30T13:35:21.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f0866ffd640 1 -- 172.21.15.31:0/3268681027 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08580072f0 con 0x7f0870153ad0 2023-10-30T13:35:21.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08701ef430 con 0x7f0870153ad0 2023-10-30T13:35:21.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.216+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08701ef9f0 con 0x7f0870153ad0 2023-10-30T13:35:21.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.220+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087004e170 con 0x7f0870153ad0 2023-10-30T13:35:21.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.220+0000 7f0866ffd640 1 -- 172.21.15.31:0/3268681027 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0858007450 con 0x7f0870153ad0 2023-10-30T13:35:21.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.220+0000 7f0866ffd640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08500664a0 0x7f0850068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.220+0000 7f0866ffd640 1 -- 172.21.15.31:0/3268681027 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0858083350 con 0x7f0870153ad0 2023-10-30T13:35:21.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.220+0000 7f086e59c640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08500664a0 0x7f0850068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.220+0000 7f086e59c640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08500664a0 0x7f0850068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f08600b2000 tx=0x7f0860008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.224+0000 7f0866ffd640 1 -- 172.21.15.31:0/3268681027 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0858050870 con 0x7f0870153ad0 2023-10-30T13:35:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08700c8480 con 0x7f0870153ad0 2023-10-30T13:35:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f0866ffd640 1 -- 172.21.15.31:0/3268681027 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0858053e90 con 0x7f0870153ad0 2023-10-30T13:35:21.384 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-10-30T13:35:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08500664a0 msgr2=0x7f0850068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08500664a0 0x7f0850068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f08600b2000 tx=0x7f0860008040 comp rx=0 tx=0).stop 2023-10-30T13:35:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 msgr2=0x7f08701e5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08701e5f40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f085800c500 tx=0x7f085800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 shutdown_connections 2023-10-30T13:35:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08500664a0 0x7f0850068960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f087015cdc0 0x7f08701e6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870153ad0 0x7f08701e5f40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 --2- 172.21.15.31:0/3268681027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08700c0840 0x7f08701521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 >> 172.21.15.31:0/3268681027 conn(0x7f087006da40 msgr2=0x7f087015d230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 shutdown_connections 2023-10-30T13:35:21.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.380+0000 7f08748d6640 1 -- 172.21.15.31:0/3268681027 wait complete. 2023-10-30T13:35:21.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:21.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9460cd6640 1 Processor -- start 2023-10-30T13:35:21.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9460cd6640 1 -- start start 2023-10-30T13:35:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9460cd6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c153c80 0x7f945c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9460cd6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f945c0c4a10 0x7f945c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9460cd6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c159f20 con 0x7f945c15a3b0 2023-10-30T13:35:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9460cd6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c15dec0 con 0x7f945c0c4a10 2023-10-30T13:35:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9460cd6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1599a0 con 0x7f945c153c80 2023-10-30T13:35:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9451d9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f945c0c4a10 0x7f945c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f945ad9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f945c15a3b0 0x7f945c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52018/0 (socket says 172.21.15.31:52018) 2023-10-30T13:35:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f945ad9d640 1 -- 172.21.15.31:0/1755256419 learned_addr learned my addr 172.21.15.31:0/1755256419 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9451d9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f945c0c4a10 0x7f945c0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36284/0 (socket says 172.21.15.31:36284) 2023-10-30T13:35:21.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f945a59c640 1 --2- 172.21.15.31:0/1755256419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c153c80 0x7f945c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9459d9b640 1 -- 172.21.15.31:0/1755256419 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522724704 0 0) 0x7f945c159f20 con 0x7f945c15a3b0 2023-10-30T13:35:21.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9459d9b640 1 -- 172.21.15.31:0/1755256419 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9448003530 con 0x7f945c15a3b0 2023-10-30T13:35:21.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9451d9b640 1 -- 172.21.15.31:0/1755256419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c153c80 msgr2=0x7f945c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:21.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9451d9b640 1 --2- 172.21.15.31:0/1755256419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c153c80 0x7f945c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9451d9b640 1 -- 172.21.15.31:0/1755256419 >> v1:172.21.15.31:6789/0 conn(0x7f945c15a3b0 legacy=0x7f945c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.544+0000 7f9451d9b640 1 -- 172.21.15.31:0/1755256419 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0c47e0 con 0x7f945c0c4a10 2023-10-30T13:35:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9451d9b640 1 --2- 172.21.15.31:0/1755256419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c0c4a10 0x7f945c0c42a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94440097c0 tx=0x7f944401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bae1c0918ae5be3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9459d9b640 1 -- 172.21.15.31:0/1755256419 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94440a8970 con 0x7f945c0c4a10 2023-10-30T13:35:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9459d9b640 1 -- 172.21.15.31:0/1755256419 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94440a8ad0 con 0x7f945c0c4a10 2023-10-30T13:35:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9459d9b640 1 -- 172.21.15.31:0/1755256419 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94440a8e60 con 0x7f945c0c4a10 2023-10-30T13:35:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 -- 172.21.15.31:0/1755256419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c0c4a10 msgr2=0x7f945c0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1755256419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c0c4a10 0x7f945c0c42a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94440097c0 tx=0x7f944401ec70 comp rx=0 tx=0).stop 2023-10-30T13:35:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 -- 172.21.15.31:0/1755256419 shutdown_connections 2023-10-30T13:35:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1755256419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c0c4a10 0x7f945c0c42a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1755256419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c153c80 0x7f945c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 -- 172.21.15.31:0/1755256419 >> 172.21.15.31:0/1755256419 conn(0x7f945c0715c0 msgr2=0x7f945c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 -- 172.21.15.31:0/1755256419 shutdown_connections 2023-10-30T13:35:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.548+0000 7f9460cd6640 1 -- 172.21.15.31:0/1755256419 wait complete. 2023-10-30T13:35:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 Processor -- start 2023-10-30T13:35:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 -- start start 2023-10-30T13:35:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f945c0c4a10 0x7f945c149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c153c80 0x7f945c14ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c15a3b0 0x7f945c0ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c14fbe0 con 0x7f945c153c80 2023-10-30T13:35:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c150000 con 0x7f945c0c4a10 2023-10-30T13:35:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c14f280 con 0x7f945c15a3b0 2023-10-30T13:35:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c153c80 0x7f945c14ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c153c80 0x7f945c14ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36300/0 (socket says 172.21.15.31:36300) 2023-10-30T13:35:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 -- 172.21.15.31:0/1555117436 learned_addr learned my addr 172.21.15.31:0/1555117436 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f945ad9d640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c15a3b0 0x7f945c0ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f945a59c640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f945c0c4a10 0x7f945c149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 -- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c15a3b0 msgr2=0x7f945c0ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c15a3b0 0x7f945c0ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 -- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f945c0c4a10 msgr2=0x7f945c149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f945c0c4a10 0x7f945c149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 -- 172.21.15.31:0/1555117436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444009420 con 0x7f945c153c80 2023-10-30T13:35:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f945a59c640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f945c0c4a10 0x7f945c149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9451d9b640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c153c80 0x7f945c14ded0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f94440b3040 tx=0x7f944401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f94537fe640 1 -- 172.21.15.31:0/1555117436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94440b0d60 con 0x7f945c153c80 2023-10-30T13:35:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f94537fe640 1 -- 172.21.15.31:0/1555117436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94440b9430 con 0x7f945c153c80 2023-10-30T13:35:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c15b670 con 0x7f945c153c80 2023-10-30T13:35:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f94537fe640 1 -- 172.21.15.31:0/1555117436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94440b97e0 con 0x7f945c153c80 2023-10-30T13:35:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c14f520 con 0x7f945c153c80 2023-10-30T13:35:21.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.552+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c04e170 con 0x7f945c153c80 2023-10-30T13:35:21.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.556+0000 7f94537fe640 1 -- 172.21.15.31:0/1555117436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94440b7040 con 0x7f945c153c80 2023-10-30T13:35:21.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.556+0000 7f94537fe640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f943c0664a0 0x7f943c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.556+0000 7f94537fe640 1 -- 172.21.15.31:0/1555117436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94441219d0 con 0x7f945c153c80 2023-10-30T13:35:21.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.556+0000 7f945a59c640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f943c0664a0 0x7f943c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.556+0000 7f945a59c640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f943c0664a0 0x7f943c068960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f944c0074c0 tx=0x7f944c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.560+0000 7f94537fe640 1 -- 172.21.15.31:0/1555117436 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94440eedb0 con 0x7f945c153c80 2023-10-30T13:35:21.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f945c0b1b20 con 0x7f945c153c80 2023-10-30T13:35:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f94537fe640 1 -- 172.21.15.31:0/1555117436 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6519 (secure 0 0 0) 0x7f94440b5020 con 0x7f945c153c80 2023-10-30T13:35:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:21.756 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:19.778393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:creating","state_seq":1,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 12 2023-10-30T13:35:21.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f943c0664a0 msgr2=0x7f943c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f943c0664a0 0x7f943c068960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f944c0074c0 tx=0x7f944c00b040 comp rx=0 tx=0).stop 2023-10-30T13:35:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c153c80 msgr2=0x7f945c14ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c153c80 0x7f945c14ded0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f94440b3040 tx=0x7f944401ba10 comp rx=0 tx=0).stop 2023-10-30T13:35:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 shutdown_connections 2023-10-30T13:35:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f943c0664a0 0x7f943c068960 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f945c15a3b0 0x7f945c0ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f945c153c80 0x7f945c14ded0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 --2- 172.21.15.31:0/1555117436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f945c0c4a10 0x7f945c149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 >> 172.21.15.31:0/1555117436 conn(0x7f945c0715c0 msgr2=0x7f945c0c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 shutdown_connections 2023-10-30T13:35:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.752+0000 7f9460cd6640 1 -- 172.21.15.31:0/1555117436 wait complete. 2023-10-30T13:35:21.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16af7d3640 1 Processor -- start 2023-10-30T13:35:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16af7d3640 1 -- start start 2023-10-30T13:35:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16af7d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a8153c80 0x7f16a815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16af7d3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f16a80c42a0 0x7f16a80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16af7d3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a8159f20 con 0x7f16a80c4a10 2023-10-30T13:35:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16af7d3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a815dec0 con 0x7f16a80c42a0 2023-10-30T13:35:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16af7d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a81599a0 con 0x7f16a8153c80 2023-10-30T13:35:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16add70640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f16a80c42a0 0x7f16a80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16add70640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f16a80c42a0 0x7f16a80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36306/0 (socket says 172.21.15.31:36306) 2023-10-30T13:35:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16add70640 1 -- 172.21.15.31:0/1069402310 learned_addr learned my addr 172.21.15.31:0/1069402310 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16ad56f640 1 --2- 172.21.15.31:0/1069402310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a8153c80 0x7f16a815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16add70640 1 -- 172.21.15.31:0/1069402310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a8153c80 msgr2=0x7f16a815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16add70640 1 --2- 172.21.15.31:0/1069402310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a8153c80 0x7f16a815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16add70640 1 -- 172.21.15.31:0/1069402310 >> v1:172.21.15.31:6789/0 conn(0x7f16a80c4a10 legacy=0x7f16a815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f16add70640 1 -- 172.21.15.31:0/1069402310 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a815d620 con 0x7f16a80c42a0 2023-10-30T13:35:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.916+0000 7f169ffff640 1 -- 172.21.15.31:0/1069402310 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458357054 0 0) 0x7f16a8159f20 con 0x7f16a80c4a10 2023-10-30T13:35:21.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16add70640 1 --2- 172.21.15.31:0/1069402310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c42a0 0x7f16a80ca1e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009e50 tx=0x7f16a001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbf28573ee941ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f169ffff640 1 -- 172.21.15.31:0/1069402310 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a00af400 con 0x7f16a80c42a0 2023-10-30T13:35:21.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f169ffff640 1 -- 172.21.15.31:0/1069402310 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a00ab050 con 0x7f16a80c42a0 2023-10-30T13:35:21.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f169ffff640 1 -- 172.21.15.31:0/1069402310 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a00af780 con 0x7f16a80c42a0 2023-10-30T13:35:21.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 -- 172.21.15.31:0/1069402310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c42a0 msgr2=0x7f16a80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 --2- 172.21.15.31:0/1069402310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c42a0 0x7f16a80ca1e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009e50 tx=0x7f16a001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 -- 172.21.15.31:0/1069402310 shutdown_connections 2023-10-30T13:35:21.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 --2- 172.21.15.31:0/1069402310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c42a0 0x7f16a80ca1e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 --2- 172.21.15.31:0/1069402310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a8153c80 0x7f16a815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 -- 172.21.15.31:0/1069402310 >> 172.21.15.31:0/1069402310 conn(0x7f16a80715c0 msgr2=0x7f16a80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 -- 172.21.15.31:0/1069402310 shutdown_connections 2023-10-30T13:35:21.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.920+0000 7f16af7d3640 1 -- 172.21.15.31:0/1069402310 wait complete. 2023-10-30T13:35:21.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 Processor -- start 2023-10-30T13:35:21.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 -- start start 2023-10-30T13:35:21.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a80c42a0 0x7f16a80b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c4a10 0x7f16a80ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16a8153c80 0x7f16a8146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80b4c40 con 0x7f16a80c4a10 2023-10-30T13:35:21.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80b3b60 con 0x7f16a8153c80 2023-10-30T13:35:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80b4fa0 con 0x7f16a80c42a0 2023-10-30T13:35:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c4a10 0x7f16a80ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c4a10 0x7f16a80ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36314/0 (socket says 172.21.15.31:36314) 2023-10-30T13:35:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 -- 172.21.15.31:0/961234271 learned_addr learned my addr 172.21.15.31:0/961234271 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16ad56f640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a80c42a0 0x7f16a80b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16add70640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16a8153c80 0x7f16a8146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 -- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a80c42a0 msgr2=0x7f16a80b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a80c42a0 0x7f16a80b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 -- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16a8153c80 msgr2=0x7f16a8146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16a8153c80 0x7f16a8146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 -- 172.21.15.31:0/961234271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a0009b30 con 0x7f16a80c4a10 2023-10-30T13:35:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16acd6e640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c4a10 0x7f16a80ce2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f169800c550 tx=0x7f169800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f169dffb640 1 -- 172.21.15.31:0/961234271 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698003600 con 0x7f16a80c4a10 2023-10-30T13:35:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f169dffb640 1 -- 172.21.15.31:0/961234271 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1698003760 con 0x7f16a80c4a10 2023-10-30T13:35:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f169dffb640 1 -- 172.21.15.31:0/961234271 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698003b10 con 0x7f16a80c4a10 2023-10-30T13:35:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a8147960 con 0x7f16a80c4a10 2023-10-30T13:35:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a8147c00 con 0x7f16a80c4a10 2023-10-30T13:35:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.924+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a804e170 con 0x7f16a80c4a10 2023-10-30T13:35:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.928+0000 7f169dffb640 1 -- 172.21.15.31:0/961234271 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1698003c70 con 0x7f16a80c4a10 2023-10-30T13:35:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.928+0000 7f169dffb640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f168c0664a0 0x7f168c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.928+0000 7f169dffb640 1 -- 172.21.15.31:0/961234271 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1698083440 con 0x7f16a80c4a10 2023-10-30T13:35:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.928+0000 7f16ad56f640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f168c0664a0 0x7f168c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.928+0000 7f16ad56f640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f168c0664a0 0x7f168c068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f16a4009ac0 tx=0x7f16a4009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:21.932+0000 7f169dffb640 1 -- 172.21.15.31:0/961234271 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1698050960 con 0x7f16a80c4a10 2023-10-30T13:35:22.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.088+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16a8159b80 con 0x7f16a80c4a10 2023-10-30T13:35:22.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.088+0000 7f169dffb640 1 -- 172.21.15.31:0/961234271 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f1698017090 con 0x7f16a80c4a10 2023-10-30T13:35:22.094 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:22.094 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f168c0664a0 msgr2=0x7f168c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f168c0664a0 0x7f168c068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f16a4009ac0 tx=0x7f16a4009190 comp rx=0 tx=0).stop 2023-10-30T13:35:22.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c4a10 msgr2=0x7f16a80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c4a10 0x7f16a80ce2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f169800c550 tx=0x7f169800ca10 comp rx=0 tx=0).stop 2023-10-30T13:35:22.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 shutdown_connections 2023-10-30T13:35:22.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f168c0664a0 0x7f168c068960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16a8153c80 0x7f16a8146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a80c4a10 0x7f16a80ce2f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 --2- 172.21.15.31:0/961234271 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16a80c42a0 0x7f16a80b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 >> 172.21.15.31:0/961234271 conn(0x7f16a80715c0 msgr2=0x7f16a80cb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:22.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 shutdown_connections 2023-10-30T13:35:22.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.092+0000 7f16af7d3640 1 -- 172.21.15.31:0/961234271 wait complete. 2023-10-30T13:35:22.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:35:05.304460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:35:06.652927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:35:22.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:35:22.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.256+0000 7fb7821f2640 1 Processor -- start 2023-10-30T13:35:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.256+0000 7fb7821f2640 1 -- start start 2023-10-30T13:35:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7821f2640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb77c071cb0 0x7fb77c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7821f2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb77c153300 0x7fb77c153770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7821f2640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c153cb0 con 0x7fb77c0718d0 2023-10-30T13:35:22.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7821f2640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c0cca10 con 0x7fb77c153300 2023-10-30T13:35:22.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7821f2640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c15d0a0 con 0x7fb77c071cb0 2023-10-30T13:35:22.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7819f1640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb77c153300 0x7fb77c153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7819f1640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb77c153300 0x7fb77c153770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36316/0 (socket says 172.21.15.31:36316) 2023-10-30T13:35:22.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7819f1640 1 -- 172.21.15.31:0/3179527381 learned_addr learned my addr 172.21.15.31:0/3179527381 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:22.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7809ef640 1 --2- 172.21.15.31:0/3179527381 >> v2:172.21.15.33:3300/0 conn(0x7fb77c071cb0 0x7fb77c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7819f1640 1 -- 172.21.15.31:0/3179527381 >> v2:172.21.15.33:3300/0 conn(0x7fb77c071cb0 msgr2=0x7fb77c0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:22.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7819f1640 1 --2- 172.21.15.31:0/3179527381 >> v2:172.21.15.33:3300/0 conn(0x7fb77c071cb0 0x7fb77c0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7819f1640 1 -- 172.21.15.31:0/3179527381 >> v1:172.21.15.33:6789/0 conn(0x7fb77c0718d0 legacy=0x7fb77c15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.260+0000 7fb7819f1640 1 -- 172.21.15.31:0/3179527381 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c15d380 con 0x7fb77c153300 2023-10-30T13:35:22.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7819f1640 1 --2- 172.21.15.31:0/3179527381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c153770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb77c072470 tx=0x7fb77401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab33eed31911 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb773fff640 1 -- 172.21.15.31:0/3179527381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7740af400 con 0x7fb77c153300 2023-10-30T13:35:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb773fff640 1 -- 172.21.15.31:0/3179527381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7740ab050 con 0x7fb77c153300 2023-10-30T13:35:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb773fff640 1 -- 172.21.15.31:0/3179527381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7740af780 con 0x7fb77c153300 2023-10-30T13:35:22.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- 172.21.15.31:0/3179527381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 msgr2=0x7fb77c153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 --2- 172.21.15.31:0/3179527381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c153770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb77c072470 tx=0x7fb77401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- 172.21.15.31:0/3179527381 shutdown_connections 2023-10-30T13:35:22.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 --2- 172.21.15.31:0/3179527381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c153770 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 --2- 172.21.15.31:0/3179527381 >> v2:172.21.15.33:3300/0 conn(0x7fb77c071cb0 0x7fb77c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- 172.21.15.31:0/3179527381 >> 172.21.15.31:0/3179527381 conn(0x7fb77c06ab00 msgr2=0x7fb77c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- 172.21.15.31:0/3179527381 shutdown_connections 2023-10-30T13:35:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- 172.21.15.31:0/3179527381 wait complete. 2023-10-30T13:35:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 Processor -- start 2023-10-30T13:35:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- start start 2023-10-30T13:35:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb77c0718d0 0x7fb77c1ef060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb77c071cb0 0x7fb77c0c4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c0b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c1ef5a0 con 0x7fb77c153300 2023-10-30T13:35:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c15ccd0 con 0x7fb77c071cb0 2023-10-30T13:35:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.264+0000 7fb7821f2640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c15bc70 con 0x7fb77c0718d0 2023-10-30T13:35:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c0b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c0b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36322/0 (socket says 172.21.15.31:36322) 2023-10-30T13:35:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 -- 172.21.15.31:0/1748814772 learned_addr learned my addr 172.21.15.31:0/1748814772 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7811f0640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb77c0718d0 0x7fb77c1ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7809ef640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb77c071cb0 0x7fb77c0c4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 -- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb77c0718d0 msgr2=0x7fb77c1ef060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb77c0718d0 0x7fb77c1ef060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 -- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb77c071cb0 msgr2=0x7fb77c0c4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb77c071cb0 0x7fb77c0c4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 -- 172.21.15.31:0/1748814772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb774009b30 con 0x7fb77c153300 2023-10-30T13:35:22.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7809ef640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb77c071cb0 0x7fb77c0c4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:22.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7819f1640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c0b3a60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7740b2040 tx=0x7fb77401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb771ffb640 1 -- 172.21.15.31:0/1748814772 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7740afd30 con 0x7fb77c153300 2023-10-30T13:35:22.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb771ffb640 1 -- 172.21.15.31:0/1748814772 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774009510 con 0x7fb77c153300 2023-10-30T13:35:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb771ffb640 1 -- 172.21.15.31:0/1748814772 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7740ba660 con 0x7fb77c153300 2023-10-30T13:35:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c15bf10 con 0x7fb77c153300 2023-10-30T13:35:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77c15c4d0 con 0x7fb77c153300 2023-10-30T13:35:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c04e170 con 0x7fb77c153300 2023-10-30T13:35:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.268+0000 7fb771ffb640 1 -- 172.21.15.31:0/1748814772 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7740b6040 con 0x7fb77c153300 2023-10-30T13:35:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.272+0000 7fb771ffb640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb7600664a0 0x7fb760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.272+0000 7fb771ffb640 1 -- 172.21.15.31:0/1748814772 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb774121fe0 con 0x7fb77c153300 2023-10-30T13:35:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.272+0000 7fb7811f0640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb7600664a0 0x7fb760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.272+0000 7fb7811f0640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb7600664a0 0x7fb760068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb778007fd0 tx=0x7fb778009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.272+0000 7fb771ffb640 1 -- 172.21.15.31:0/1748814772 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7740ef330 con 0x7fb77c153300 2023-10-30T13:35:22.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.428+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb77c0c9f80 con 0x7fb77c153300 2023-10-30T13:35:22.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.428+0000 7fb771ffb640 1 -- 172.21.15.31:0/1748814772 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb7740b4070 con 0x7fb77c153300 2023-10-30T13:35:22.434 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:35:22.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb7600664a0 msgr2=0x7fb760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb7600664a0 0x7fb760068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb778007fd0 tx=0x7fb778009040 comp rx=0 tx=0).stop 2023-10-30T13:35:22.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 msgr2=0x7fb77c0b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c0b3a60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7740b2040 tx=0x7fb77401bf60 comp rx=0 tx=0).stop 2023-10-30T13:35:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 shutdown_connections 2023-10-30T13:35:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb7600664a0 0x7fb760068960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb77c153300 0x7fb77c0b3a60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb77c071cb0 0x7fb77c0c4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 --2- 172.21.15.31:0/1748814772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb77c0718d0 0x7fb77c1ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 >> 172.21.15.31:0/1748814772 conn(0x7fb77c06ab00 msgr2=0x7fb77c0cace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:22.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 shutdown_connections 2023-10-30T13:35:22.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.432+0000 7fb7821f2640 1 -- 172.21.15.31:0/1748814772 wait complete. 2023-10-30T13:35:22.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:35:22.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.596+0000 7f9e24bd9640 1 Processor -- start 2023-10-30T13:35:22.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.596+0000 7f9e24bd9640 1 -- start start 2023-10-30T13:35:22.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.596+0000 7f9e24bd9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9e20072820 0x7f9e2015d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.596+0000 7f9e24bd9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2015da10 con 0x7f9e200723d0 2023-10-30T13:35:22.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.596+0000 7f9e24bd9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20159b00 con 0x7f9e20153ad0 2023-10-30T13:35:22.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.596+0000 7f9e24bd9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e20072c90 con 0x7f9e20072820 2023-10-30T13:35:22.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1ed9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9e20072820 0x7f9e2015d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1ed9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9e20072820 0x7f9e2015d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36330/0 (socket says 172.21.15.31:36330) 2023-10-30T13:35:22.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e17fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f9e200723d0 0x7f9e2015cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52034/0 (socket says 172.21.15.31:52034) 2023-10-30T13:35:22.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1ed9d640 1 -- 172.21.15.31:0/3142091249 learned_addr learned my addr 172.21.15.31:0/3142091249 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:22.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1ed9d640 1 -- 172.21.15.31:0/3142091249 >> v1:172.21.15.33:6789/0 conn(0x7f9e20153ad0 legacy=0x7f9e200c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1ed9d640 1 -- 172.21.15.31:0/3142091249 >> v1:172.21.15.31:6789/0 conn(0x7f9e200723d0 legacy=0x7f9e2015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1ed9d640 1 -- 172.21.15.31:0/3142091249 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2015dc70 con 0x7f9e20072820 2023-10-30T13:35:22.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1dd9b640 1 -- 172.21.15.31:0/3142091249 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710789660 0 0) 0x7f9e2015da10 con 0x7f9e200723d0 2023-10-30T13:35:22.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1ed9d640 1 --2- 172.21.15.31:0/3142091249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2015d4d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009e50 tx=0x7f9e1801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6346cabfec3847ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1dd9b640 1 -- 172.21.15.31:0/3142091249 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180af400 con 0x7f9e20072820 2023-10-30T13:35:22.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1dd9b640 1 -- 172.21.15.31:0/3142091249 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e180ab050 con 0x7f9e20072820 2023-10-30T13:35:22.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.600+0000 7f9e1dd9b640 1 -- 172.21.15.31:0/3142091249 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180af780 con 0x7f9e20072820 2023-10-30T13:35:22.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- 172.21.15.31:0/3142091249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 msgr2=0x7f9e2015d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/3142091249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2015d4d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009e50 tx=0x7f9e1801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- 172.21.15.31:0/3142091249 shutdown_connections 2023-10-30T13:35:22.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/3142091249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2015d4d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- 172.21.15.31:0/3142091249 >> 172.21.15.31:0/3142091249 conn(0x7f9e2006da40 msgr2=0x7f9e2006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:22.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- 172.21.15.31:0/3142091249 shutdown_connections 2023-10-30T13:35:22.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- 172.21.15.31:0/3142091249 wait complete. 2023-10-30T13:35:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 Processor -- start 2023-10-30T13:35:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- start start 2023-10-30T13:35:22.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e200723d0 0x7f9e200c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2014fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e20153ad0 0x7f9e200c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200c2980 con 0x7f9e20072820 2023-10-30T13:35:22.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200c51e0 con 0x7f9e20153ad0 2023-10-30T13:35:22.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e24bd9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200c2c00 con 0x7f9e200723d0 2023-10-30T13:35:22.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2014fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2014fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36346/0 (socket says 172.21.15.31:36346) 2023-10-30T13:35:22.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 -- 172.21.15.31:0/2743629660 learned_addr learned my addr 172.21.15.31:0/2743629660 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:22.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e1e59c640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e200723d0 0x7f9e200c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e1ed9d640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e20153ad0 0x7f9e200c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 -- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e200723d0 msgr2=0x7f9e200c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e200723d0 0x7f9e200c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 -- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e20153ad0 msgr2=0x7f9e200c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e20153ad0 0x7f9e200c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.604+0000 7f9e17fff640 1 -- 172.21.15.31:0/2743629660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18009b30 con 0x7f9e20072820 2023-10-30T13:35:22.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e17fff640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2014fcb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c4a0 tx=0x7f9e0800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e16ffd640 1 -- 172.21.15.31:0/2743629660 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080074e0 con 0x7f9e20072820 2023-10-30T13:35:22.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e16ffd640 1 -- 172.21.15.31:0/2743629660 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e08007640 con 0x7f9e20072820 2023-10-30T13:35:22.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e16ffd640 1 -- 172.21.15.31:0/2743629660 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08004a30 con 0x7f9e20072820 2023-10-30T13:35:22.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e200c2ee0 con 0x7f9e20072820 2023-10-30T13:35:22.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e200c34a0 con 0x7f9e20072820 2023-10-30T13:35:22.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e200b11e0 con 0x7f9e20072820 2023-10-30T13:35:22.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e16ffd640 1 -- 172.21.15.31:0/2743629660 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e08004b90 con 0x7f9e20072820 2023-10-30T13:35:22.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e16ffd640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e000664d0 0x7f9e00068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e16ffd640 1 -- 172.21.15.31:0/2743629660 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e08083040 con 0x7f9e20072820 2023-10-30T13:35:22.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.608+0000 7f9e1e59c640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e000664d0 0x7f9e00068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.612+0000 7f9e1e59c640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e000664d0 0x7f9e00068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9e10000f70 tx=0x7f9e10009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.612+0000 7f9e16ffd640 1 -- 172.21.15.31:0/2743629660 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e080504c0 con 0x7f9e20072820 2023-10-30T13:35:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e2004e170 con 0x7f9e20072820 2023-10-30T13:35:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e16ffd640 1 -- 172.21.15.31:0/2743629660 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9e08053ae0 con 0x7f9e20072820 2023-10-30T13:35:22.773 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-10-30T13:35:22.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e000664d0 msgr2=0x7f9e00068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e000664d0 0x7f9e00068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9e10000f70 tx=0x7f9e10009040 comp rx=0 tx=0).stop 2023-10-30T13:35:22.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 msgr2=0x7f9e2014fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2014fcb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c4a0 tx=0x7f9e0800c960 comp rx=0 tx=0).stop 2023-10-30T13:35:22.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 shutdown_connections 2023-10-30T13:35:22.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e000664d0 0x7f9e00068990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e20153ad0 0x7f9e200c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20072820 0x7f9e2014fcb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 --2- 172.21.15.31:0/2743629660 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e200723d0 0x7f9e200c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 >> 172.21.15.31:0/2743629660 conn(0x7f9e2006da40 msgr2=0x7f9e201590d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:22.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.768+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 shutdown_connections 2023-10-30T13:35:22.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.772+0000 7f9e24bd9640 1 -- 172.21.15.31:0/2743629660 wait complete. 2023-10-30T13:35:22.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:35:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5c50f640 1 Processor -- start 2023-10-30T13:35:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5c50f640 1 -- start start 2023-10-30T13:35:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5c50f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 0x7f1b540c8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5c50f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f1b5415a4d0 0x7f1b54069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5c50f640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b541597b0 con 0x7f1b540c5b90 2023-10-30T13:35:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5c50f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b540c74e0 con 0x7f1b5415a4d0 2023-10-30T13:35:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5c50f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5415a040 con 0x7f1b540c5f70 2023-10-30T13:35:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5a2ab640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f1b540c5b90 0x7f1b540c66e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52036/0 (socket says 172.21.15.31:52036) 2023-10-30T13:35:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5a2ab640 1 -- 172.21.15.31:0/3483131089 learned_addr learned my addr 172.21.15.31:0/3483131089 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5aaac640 1 --2- 172.21.15.31:0/3483131089 >> v2:172.21.15.33:3300/0 conn(0x7f1b5415a4d0 0x7f1b54069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b59aaa640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 0x7f1b540c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047650405 0 0) 0x7f1b50001120 con 0x7f1b540c5b90 2023-10-30T13:35:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b40003530 con 0x7f1b540c5b90 2023-10-30T13:35:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539435258 0 0) 0x7f1b40003530 con 0x7f1b540c5b90 2023-10-30T13:35:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 msgr2=0x7f1b540c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 0x7f1b540c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 >> v2:172.21.15.33:3300/0 conn(0x7f1b5415a4d0 msgr2=0x7f1b54069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 --2- 172.21.15.31:0/3483131089 >> v2:172.21.15.33:3300/0 conn(0x7f1b5415a4d0 0x7f1b54069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b540c6f20 con 0x7f1b540c5b90 2023-10-30T13:35:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f1b50001810 con 0x7f1b540c5b90 2023-10-30T13:35:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 >> v1:172.21.15.31:6789/0 conn(0x7f1b540c5b90 legacy=0x7f1b540c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b40003980 0x7f1b40003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b400047d0 0x7f1b40004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 0x7f1b40005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540c74e0 con 0x7f1b400047d0 2023-10-30T13:35:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5415a040 con 0x7f1b40003980 2023-10-30T13:35:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4000b990 con 0x7f1b540c5f70 2023-10-30T13:35:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b59aaa640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b400047d0 0x7f1b40004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5a2ab640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b40003980 0x7f1b40003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b5aaac640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 0x7f1b40005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b59aaa640 1 -- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 msgr2=0x7f1b40005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b59aaa640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b540c5f70 0x7f1b40005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b59aaa640 1 -- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b40003980 msgr2=0x7f1b40003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b59aaa640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b40003980 0x7f1b40003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.936+0000 7f1b59aaa640 1 -- 172.21.15.31:0/3483131089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b40003530 con 0x7f1b400047d0 2023-10-30T13:35:22.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b59aaa640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b400047d0 0x7f1b40004c50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009ae0 tx=0x7f1b4401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56219ebe99f4bd83 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440a5560 con 0x7f1b400047d0 2023-10-30T13:35:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b440a56c0 con 0x7f1b400047d0 2023-10-30T13:35:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b592a9640 1 -- 172.21.15.31:0/3483131089 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440a5a70 con 0x7f1b400047d0 2023-10-30T13:35:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3483131089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b400047d0 msgr2=0x7f1b40004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b5c50f640 1 --2- 172.21.15.31:0/3483131089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b400047d0 0x7f1b40004c50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009ae0 tx=0x7f1b4401ca50 comp rx=0 tx=0).stop 2023-10-30T13:35:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b5aaac640 1 -- 172.21.15.31:0/3483131089 reap_dead start 2023-10-30T13:35:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3483131089 shutdown_connections 2023-10-30T13:35:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3483131089 >> 172.21.15.31:0/3483131089 conn(0x7f1b5406afa0 msgr2=0x7f1b5406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3483131089 shutdown_connections 2023-10-30T13:35:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.940+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3483131089 wait complete. 2023-10-30T13:35:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 Processor -- start 2023-10-30T13:35:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 -- start start 2023-10-30T13:35:22.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b400047d0 0x7f1b540c0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b54152930 0x7f1b54154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b540bfe00 0x7f1b54155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b54154450 con 0x7f1b540bfe00 2023-10-30T13:35:22.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b54156500 con 0x7f1b400047d0 2023-10-30T13:35:22.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b54154170 con 0x7f1b54152930 2023-10-30T13:35:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b540bfe00 0x7f1b54155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b540bfe00 0x7f1b54155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36370/0 (socket says 172.21.15.31:36370) 2023-10-30T13:35:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 -- 172.21.15.31:0/3043633133 learned_addr learned my addr 172.21.15.31:0/3043633133 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b59aaa640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b54152930 0x7f1b54154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5a2ab640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b400047d0 0x7f1b540c0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 -- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b54152930 msgr2=0x7f1b54154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b54152930 0x7f1b54154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 -- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b400047d0 msgr2=0x7f1b540c0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b400047d0 0x7f1b540c0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 -- 172.21.15.31:0/3043633133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b541567a0 con 0x7f1b540bfe00 2023-10-30T13:35:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5aaac640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b540bfe00 0x7f1b54155330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b540c9810 tx=0x7f1b4c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b4affd640 1 -- 172.21.15.31:0/3043633133 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c00e070 con 0x7f1b540bfe00 2023-10-30T13:35:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b4affd640 1 -- 172.21.15.31:0/3043633133 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c011e10 con 0x7f1b540bfe00 2023-10-30T13:35:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b4affd640 1 -- 172.21.15.31:0/3043633133 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0039a0 con 0x7f1b540bfe00 2023-10-30T13:35:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b540c25c0 con 0x7f1b540bfe00 2023-10-30T13:35:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.944+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b54158140 con 0x7f1b540bfe00 2023-10-30T13:35:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.948+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5404e170 con 0x7f1b540bfe00 2023-10-30T13:35:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.948+0000 7f1b4affd640 1 -- 172.21.15.31:0/3043633133 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b4c003b00 con 0x7f1b540bfe00 2023-10-30T13:35:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.948+0000 7f1b4affd640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b380664a0 0x7f1b38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.948+0000 7f1b4affd640 1 -- 172.21.15.31:0/3043633133 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b4c084530 con 0x7f1b540bfe00 2023-10-30T13:35:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.948+0000 7f1b5a2ab640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b380664a0 0x7f1b38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.948+0000 7f1b5a2ab640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b380664a0 0x7f1b38068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1b50001060 tx=0x7f1b50009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:22.952+0000 7f1b4affd640 1 -- 172.21.15.31:0/3043633133 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b4c0519a0 con 0x7f1b540bfe00 2023-10-30T13:35:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.104+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1b54157bf0 con 0x7f1b540bfe00 2023-10-30T13:35:23.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b4affd640 1 -- 172.21.15.31:0/3043633133 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1b4c003db0 con 0x7f1b540bfe00 2023-10-30T13:35:23.111 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-10-30T13:35:23.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b380664a0 msgr2=0x7f1b38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b380664a0 0x7f1b38068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1b50001060 tx=0x7f1b50009040 comp rx=0 tx=0).stop 2023-10-30T13:35:23.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b540bfe00 msgr2=0x7f1b54155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b540bfe00 0x7f1b54155330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b540c9810 tx=0x7f1b4c0115a0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 shutdown_connections 2023-10-30T13:35:23.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1b380664a0 0x7f1b38068960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b540bfe00 0x7f1b54155330 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1b54152930 0x7f1b54154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 --2- 172.21.15.31:0/3043633133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b400047d0 0x7f1b540c0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 >> 172.21.15.31:0/3043633133 conn(0x7f1b5406afa0 msgr2=0x7f1b54069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:23.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 shutdown_connections 2023-10-30T13:35:23.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.108+0000 7f1b5c50f640 1 -- 172.21.15.31:0/3043633133 wait complete. 2023-10-30T13:35:23.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c9190f640 1 Processor -- start 2023-10-30T13:35:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c9190f640 1 -- start start 2023-10-30T13:35:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c9190f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0c8c0c5c10 0x7f0c8c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c9190f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c9190f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8c15cd50 con 0x7f0c8c15da00 2023-10-30T13:35:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c9190f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8c15d0a0 con 0x7f0c8c0c5c10 2023-10-30T13:35:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c9190f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0ca000 con 0x7f0c8c159760 2023-10-30T13:35:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c15d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53842/0 (socket says 172.21.15.31:53842) 2023-10-30T13:35:23.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c83fff640 1 -- 172.21.15.31:0/3151648026 learned_addr learned my addr 172.21.15.31:0/3151648026 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:23.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.272+0000 7f0c8affd640 1 --2- 172.21.15.31:0/3151648026 >> v2:172.21.15.33:3300/0 conn(0x7f0c8c0c5c10 0x7f0c8c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c83fff640 1 -- 172.21.15.31:0/3151648026 >> v2:172.21.15.33:3300/0 conn(0x7f0c8c0c5c10 msgr2=0x7f0c8c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c83fff640 1 --2- 172.21.15.31:0/3151648026 >> v2:172.21.15.33:3300/0 conn(0x7f0c8c0c5c10 0x7f0c8c06eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c83fff640 1 -- 172.21.15.31:0/3151648026 >> v1:172.21.15.33:6789/0 conn(0x7f0c8c15da00 legacy=0x7f0c8c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c83fff640 1 -- 172.21.15.31:0/3151648026 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c159cf0 con 0x7f0c8c159760 2023-10-30T13:35:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c83fff640 1 --2- 172.21.15.31:0/3151648026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c15d4c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009870 tx=0x7f0c7401c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ba635b0ab868d2f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c8a7fc640 1 -- 172.21.15.31:0/3151648026 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c740aa670 con 0x7f0c8c159760 2023-10-30T13:35:23.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c8a7fc640 1 -- 172.21.15.31:0/3151648026 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c740aa7d0 con 0x7f0c8c159760 2023-10-30T13:35:23.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.276+0000 7f0c8a7fc640 1 -- 172.21.15.31:0/3151648026 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c740aab60 con 0x7f0c8c159760 2023-10-30T13:35:23.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- 172.21.15.31:0/3151648026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 msgr2=0x7f0c8c15d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 --2- 172.21.15.31:0/3151648026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c15d4c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009870 tx=0x7f0c7401c510 comp rx=0 tx=0).stop 2023-10-30T13:35:23.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- 172.21.15.31:0/3151648026 shutdown_connections 2023-10-30T13:35:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 --2- 172.21.15.31:0/3151648026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c15d4c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 --2- 172.21.15.31:0/3151648026 >> v2:172.21.15.33:3300/0 conn(0x7f0c8c0c5c10 0x7f0c8c06eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- 172.21.15.31:0/3151648026 >> 172.21.15.31:0/3151648026 conn(0x7f0c8c06afa0 msgr2=0x7f0c8c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- 172.21.15.31:0/3151648026 shutdown_connections 2023-10-30T13:35:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- 172.21.15.31:0/3151648026 wait complete. 2023-10-30T13:35:23.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 Processor -- start 2023-10-30T13:35:23.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- start start 2023-10-30T13:35:23.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c0c5c10 0x7f0c8c1f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c1f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c8c15da00 0x7f0c8c1f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c1f2b50 con 0x7f0c8c15da00 2023-10-30T13:35:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c14e700 con 0x7f0c8c0c5c10 2023-10-30T13:35:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c9190f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c14dbc0 con 0x7f0c8c159760 2023-10-30T13:35:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c1f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c8c15da00 0x7f0c8c1f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c0c5c10 0x7f0c8c1f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c8c15da00 0x7f0c8c1f4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36378/0 (socket says 172.21.15.31:36378) 2023-10-30T13:35:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c1f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53848/0 (socket says 172.21.15.31:53848) 2023-10-30T13:35:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 -- 172.21.15.31:0/1759673690 learned_addr learned my addr 172.21.15.31:0/1759673690 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:23.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 -- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 msgr2=0x7f0c8c1f3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c1f3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 -- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c0c5c10 msgr2=0x7f0c8c1f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c0c5c10 0x7f0c8c1f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.280+0000 7f0c8b7fe640 1 -- 172.21.15.31:0/1759673690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740094d0 con 0x7f0c8c15da00 2023-10-30T13:35:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c8b7fe640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c8c15da00 0x7f0c8c1f4cb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c6c0 tx=0x7f0c8400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c88ff9640 1 -- 172.21.15.31:0/1759673690 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84003b80 con 0x7f0c8c15da00 2023-10-30T13:35:23.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c88ff9640 1 -- 172.21.15.31:0/1759673690 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c84003ce0 con 0x7f0c8c15da00 2023-10-30T13:35:23.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8c14de60 con 0x7f0c8c15da00 2023-10-30T13:35:23.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c88ff9640 1 -- 172.21.15.31:0/1759673690 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c840052f0 con 0x7f0c8c15da00 2023-10-30T13:35:23.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8c14e420 con 0x7f0c8c15da00 2023-10-30T13:35:23.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8c06a580 con 0x7f0c8c15da00 2023-10-30T13:35:23.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c88ff9640 1 -- 172.21.15.31:0/1759673690 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c84005450 con 0x7f0c8c15da00 2023-10-30T13:35:23.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c88ff9640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0c6c0664d0 0x7f0c6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c88ff9640 1 -- 172.21.15.31:0/1759673690 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c84083ac0 con 0x7f0c8c15da00 2023-10-30T13:35:23.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.284+0000 7f0c8affd640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0c6c0664d0 0x7f0c6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.288+0000 7f0c8affd640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0c6c0664d0 0x7f0c6c068990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009ac0 tx=0x7f0c7c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.288+0000 7f0c88ff9640 1 -- 172.21.15.31:0/1759673690 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c8c06a580 con 0x7f0c8c15da00 2023-10-30T13:35:23.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.480+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c8c06a580 con 0x7f0c8c15da00 2023-10-30T13:35:23.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.480+0000 7f0c88ff9640 1 -- 172.21.15.31:0/1759673690 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6519 (secure 0 0 0) 0x7f0c8c06a580 con 0x7f0c8c15da00 2023-10-30T13:35:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:23.487 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:19.778393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:creating","state_seq":1,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 12 2023-10-30T13:35:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0c6c0664d0 msgr2=0x7f0c6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0c6c0664d0 0x7f0c6c068990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009ac0 tx=0x7f0c7c009190 comp rx=0 tx=0).stop 2023-10-30T13:35:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c8c15da00 msgr2=0x7f0c8c1f4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c8c15da00 0x7f0c8c1f4cb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c6c0 tx=0x7f0c8400cb80 comp rx=0 tx=0).stop 2023-10-30T13:35:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 shutdown_connections 2023-10-30T13:35:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0c6c0664d0 0x7f0c6c068990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c8c15da00 0x7f0c8c1f4cb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0c8c159760 0x7f0c8c1f3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 --2- 172.21.15.31:0/1759673690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c0c5c10 0x7f0c8c1f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 >> 172.21.15.31:0/1759673690 conn(0x7f0c8c06afa0 msgr2=0x7f0c8c070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 shutdown_connections 2023-10-30T13:35:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.484+0000 7f0c9190f640 1 -- 172.21.15.31:0/1759673690 wait complete. 2023-10-30T13:35:23.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:23.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.644+0000 7f4b4cad6640 1 Processor -- start 2023-10-30T13:35:23.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.644+0000 7f4b4cad6640 1 -- start start 2023-10-30T13:35:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b4cad6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f4b48150830 0x7f4b4806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b4cad6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b48070520 0x7f4b480709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b4cad6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b480c2310 con 0x7f4b480cb7d0 2023-10-30T13:35:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b4cad6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b48156710 con 0x7f4b48150830 2023-10-30T13:35:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b4cad6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c4360 con 0x7f4b48070520 2023-10-30T13:35:23.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f4b480cb7d0 0x7f4b480c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52048/0 (socket says 172.21.15.31:52048) 2023-10-30T13:35:23.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 -- 172.21.15.31:0/3543410018 learned_addr learned my addr 172.21.15.31:0/3543410018 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:23.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b46d9d640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b48070520 0x7f4b480709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b4659c640 1 --2- 172.21.15.31:0/3543410018 >> v2:172.21.15.33:3300/0 conn(0x7f4b48150830 0x7f4b4806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868668343 0 0) 0x7f4b480c2310 con 0x7f4b480cb7d0 2023-10-30T13:35:23.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b34003530 con 0x7f4b480cb7d0 2023-10-30T13:35:23.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 146630697 0 0) 0x7f4b34003530 con 0x7f4b480cb7d0 2023-10-30T13:35:23.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b48070520 msgr2=0x7f4b480709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:23.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b48070520 0x7f4b480709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 >> v2:172.21.15.33:3300/0 conn(0x7f4b48150830 msgr2=0x7f4b4806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:23.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 --2- 172.21.15.31:0/3543410018 >> v2:172.21.15.33:3300/0 conn(0x7f4b48150830 0x7f4b4806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b481569b0 con 0x7f4b480cb7d0 2023-10-30T13:35:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f4b30001680 con 0x7f4b480cb7d0 2023-10-30T13:35:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 >> v1:172.21.15.31:6789/0 conn(0x7f4b480cb7d0 legacy=0x7f4b480c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34003980 0x7f4b34003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b34004370 0x7f4b34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b34004ec0 0x7f4b3400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48156710 con 0x7f4b34004370 2023-10-30T13:35:23.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c4360 con 0x7f4b34004ec0 2023-10-30T13:35:23.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3400bed0 con 0x7f4b34003980 2023-10-30T13:35:23.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b46d9d640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b48070520 0x7f4b480709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:23.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b34004370 0x7f4b34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b4659c640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34003980 0x7f4b34003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b46d9d640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b34004ec0 0x7f4b3400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 -- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34003980 msgr2=0x7f4b34003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:23.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34003980 0x7f4b34003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 -- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b34004ec0 msgr2=0x7f4b3400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:23.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b34004ec0 0x7f4b3400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.648+0000 7f4b3ffff640 1 -- 172.21.15.31:0/3543410018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34003530 con 0x7f4b34004370 2023-10-30T13:35:23.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b3ffff640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b34004370 0x7f4b34004850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000c270 tx=0x7f4b3001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f771224c6b44df90 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b300a5960 con 0x7f4b34004370 2023-10-30T13:35:23.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b300a5ac0 con 0x7f4b34004370 2023-10-30T13:35:23.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b45d9b640 1 -- 172.21.15.31:0/3543410018 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b300058e0 con 0x7f4b34004370 2023-10-30T13:35:23.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b4cad6640 1 -- 172.21.15.31:0/3543410018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b34004370 msgr2=0x7f4b34004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b4cad6640 1 --2- 172.21.15.31:0/3543410018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b34004370 0x7f4b34004850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000c270 tx=0x7f4b3001cc60 comp rx=0 tx=0).stop 2023-10-30T13:35:23.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b46d9d640 1 -- 172.21.15.31:0/3543410018 reap_dead start 2023-10-30T13:35:23.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b4cad6640 1 -- 172.21.15.31:0/3543410018 shutdown_connections 2023-10-30T13:35:23.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b4cad6640 1 -- 172.21.15.31:0/3543410018 >> 172.21.15.31:0/3543410018 conn(0x7f4b480cac70 msgr2=0x7f4b480cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:23.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b4cad6640 1 -- 172.21.15.31:0/3543410018 shutdown_connections 2023-10-30T13:35:23.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.652+0000 7f4b4cad6640 1 -- 172.21.15.31:0/3543410018 wait complete. 2023-10-30T13:35:23.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 Processor -- start 2023-10-30T13:35:23.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 -- start start 2023-10-30T13:35:23.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34004370 0x7f4b4814b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b481577c0 0x7f4b481f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b48158ba0 0x7f4b48155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48159a80 con 0x7f4b48158ba0 2023-10-30T13:35:23.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c1c90 con 0x7f4b481577c0 2023-10-30T13:35:23.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b481f1b20 con 0x7f4b34004370 2023-10-30T13:35:23.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b48158ba0 0x7f4b48155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b48158ba0 0x7f4b48155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36398/0 (socket says 172.21.15.31:36398) 2023-10-30T13:35:23.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4659c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34004370 0x7f4b4814b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 -- 172.21.15.31:0/114208594 learned_addr learned my addr 172.21.15.31:0/114208594 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:23.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b3ffff640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b481577c0 0x7f4b481f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 -- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34004370 msgr2=0x7f4b4814b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34004370 0x7f4b4814b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 -- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b481577c0 msgr2=0x7f4b481f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b481577c0 0x7f4b481f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 -- 172.21.15.31:0/114208594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b48152b90 con 0x7f4b48158ba0 2023-10-30T13:35:23.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b46d9d640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b48158ba0 0x7f4b48155480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b481541f0 tx=0x7f4b40011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b3effd640 1 -- 172.21.15.31:0/114208594 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4000e070 con 0x7f4b48158ba0 2023-10-30T13:35:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b3effd640 1 -- 172.21.15.31:0/114208594 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b400039a0 con 0x7f4b48158ba0 2023-10-30T13:35:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b3effd640 1 -- 172.21.15.31:0/114208594 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40003b50 con 0x7f4b48158ba0 2023-10-30T13:35:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4815a3f0 con 0x7f4b48158ba0 2023-10-30T13:35:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.656+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4815a9b0 con 0x7f4b48158ba0 2023-10-30T13:35:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.660+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4804e170 con 0x7f4b48158ba0 2023-10-30T13:35:23.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.660+0000 7f4b3effd640 1 -- 172.21.15.31:0/114208594 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b40003cb0 con 0x7f4b48158ba0 2023-10-30T13:35:23.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.660+0000 7f4b3effd640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b280664a0 0x7f4b28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.660+0000 7f4b3effd640 1 -- 172.21.15.31:0/114208594 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b400845c0 con 0x7f4b48158ba0 2023-10-30T13:35:23.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.660+0000 7f4b4659c640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b280664a0 0x7f4b28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.660+0000 7f4b4659c640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b280664a0 0x7f4b28068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009a60 tx=0x7f4b38008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.664+0000 7f4b3effd640 1 -- 172.21.15.31:0/114208594 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b40051ae0 con 0x7f4b48158ba0 2023-10-30T13:35:23.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b48070300 con 0x7f4b48158ba0 2023-10-30T13:35:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b3effd640 1 -- 172.21.15.31:0/114208594 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f4b40055100 con 0x7f4b48158ba0 2023-10-30T13:35:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:23.832 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:23.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b280664a0 msgr2=0x7f4b28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b280664a0 0x7f4b28068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009a60 tx=0x7f4b38008040 comp rx=0 tx=0).stop 2023-10-30T13:35:23.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b48158ba0 msgr2=0x7f4b48155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b48158ba0 0x7f4b48155480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b481541f0 tx=0x7f4b40011890 comp rx=0 tx=0).stop 2023-10-30T13:35:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 shutdown_connections 2023-10-30T13:35:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b280664a0 0x7f4b28068960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b48158ba0 0x7f4b48155480 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b481577c0 0x7f4b481f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 --2- 172.21.15.31:0/114208594 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b34004370 0x7f4b4814b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 >> 172.21.15.31:0/114208594 conn(0x7f4b480cac70 msgr2=0x7f4b480c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:23.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 shutdown_connections 2023-10-30T13:35:23.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.828+0000 7f4b4cad6640 1 -- 172.21.15.31:0/114208594 wait complete. 2023-10-30T13:35:23.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.992+0000 7fac2cdd3640 1 Processor -- start 2023-10-30T13:35:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.992+0000 7fac2cdd3640 1 -- start start 2023-10-30T13:35:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.992+0000 7fac2cdd3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fac28151840 0x7fac280c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.992+0000 7fac2cdd3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac280c5030 con 0x7fac280cb8b0 2023-10-30T13:35:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.992+0000 7fac2cdd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac28157830 con 0x7fac28151c20 2023-10-30T13:35:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.992+0000 7fac2cdd3640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac28157b10 con 0x7fac28151840 2023-10-30T13:35:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fac280cb8b0 0x7fac2806b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52058/0 (socket says 172.21.15.31:52058) 2023-10-30T13:35:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 -- 172.21.15.31:0/1722696932 learned_addr learned my addr 172.21.15.31:0/1722696932 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:24.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2659c640 1 --2- 172.21.15.31:0/1722696932 >> v2:172.21.15.33:3300/0 conn(0x7fac28151840 0x7fac280c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113934245 0 0) 0x7fac280c5030 con 0x7fac280cb8b0 2023-10-30T13:35:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac10003530 con 0x7fac280cb8b0 2023-10-30T13:35:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2887041721 0 0) 0x7fac10003530 con 0x7fac280cb8b0 2023-10-30T13:35:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 >> v2:172.21.15.33:3300/0 conn(0x7fac28151840 msgr2=0x7fac280c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 --2- 172.21.15.31:0/1722696932 >> v2:172.21.15.33:3300/0 conn(0x7fac28151840 0x7fac280c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 >> v1:172.21.15.33:6789/0 conn(0x7fac28151c20 legacy=0x7fac28072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac28157df0 con 0x7fac280cb8b0 2023-10-30T13:35:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fac20001a00 con 0x7fac280cb8b0 2023-10-30T13:35:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 >> v1:172.21.15.31:6789/0 conn(0x7fac280cb8b0 legacy=0x7fac2806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10003980 0x7fac10003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac10004380 0x7fac10004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac10005050 0x7fac1000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac28157b10 con 0x7fac10005050 2023-10-30T13:35:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1000bf00 con 0x7fac10003980 2023-10-30T13:35:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1000c0c0 con 0x7fac10004380 2023-10-30T13:35:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac10005050 0x7fac1000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac2659c640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10003980 0x7fac10003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac25d9b640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac10004380 0x7fac10004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 -- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac10004380 msgr2=0x7fac10004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac10004380 0x7fac10004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 -- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10003980 msgr2=0x7fac10003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10003980 0x7fac10003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:23.996+0000 7fac26d9d640 1 -- 172.21.15.31:0/1722696932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10003530 con 0x7fac10005050 2023-10-30T13:35:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac26d9d640 1 -- 172.21.15.31:0/1722696932 reap_dead start 2023-10-30T13:35:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac26d9d640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac10005050 0x7fac1000b990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fac2000a830 tx=0x7fac2001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12554ff774951880 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac200ad630 con 0x7fac10005050 2023-10-30T13:35:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac200a8070 con 0x7fac10005050 2023-10-30T13:35:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2559a640 1 -- 172.21.15.31:0/1722696932 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac200ad9f0 con 0x7fac10005050 2023-10-30T13:35:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1722696932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac10005050 msgr2=0x7fac1000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac10005050 0x7fac1000b990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fac2000a830 tx=0x7fac2001d340 comp rx=0 tx=0).stop 2023-10-30T13:35:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1722696932 shutdown_connections 2023-10-30T13:35:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1722696932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac10005050 0x7fac1000b990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1722696932 >> 172.21.15.31:0/1722696932 conn(0x7fac280cac70 msgr2=0x7fac280cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1722696932 shutdown_connections 2023-10-30T13:35:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1722696932 wait complete. 2023-10-30T13:35:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 Processor -- start 2023-10-30T13:35:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.000+0000 7fac2cdd3640 1 -- start start 2023-10-30T13:35:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10005050 0x7fac281e5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac28154640 0x7fac28152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac28152570 0x7fac28145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac281534e0 con 0x7fac28152570 2023-10-30T13:35:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac281f22a0 con 0x7fac10005050 2023-10-30T13:35:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac28154ea0 con 0x7fac28154640 2023-10-30T13:35:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac28152570 0x7fac28145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac28152570 0x7fac28145900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36404/0 (socket says 172.21.15.31:36404) 2023-10-30T13:35:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 -- 172.21.15.31:0/1749390253 learned_addr learned my addr 172.21.15.31:0/1749390253 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac25d9b640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac28154640 0x7fac28152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2659c640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10005050 0x7fac281e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 -- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac28154640 msgr2=0x7fac28152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac28154640 0x7fac28152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 -- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10005050 msgr2=0x7fac281e5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10005050 0x7fac281e5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 -- 172.21.15.31:0/1749390253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac281f2730 con 0x7fac28152570 2023-10-30T13:35:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac26d9d640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac28152570 0x7fac28145900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac2000a830 tx=0x7fac10005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac16ffd640 1 -- 172.21.15.31:0/1749390253 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac200b9670 con 0x7fac28152570 2023-10-30T13:35:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac16ffd640 1 -- 172.21.15.31:0/1749390253 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac200b97d0 con 0x7fac28152570 2023-10-30T13:35:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac16ffd640 1 -- 172.21.15.31:0/1749390253 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac200b9b60 con 0x7fac28152570 2023-10-30T13:35:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac28154bc0 con 0x7fac28152570 2023-10-30T13:35:24.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac281f1c50 con 0x7fac28152570 2023-10-30T13:35:24.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac281f2450 con 0x7fac28152570 2023-10-30T13:35:24.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.004+0000 7fac16ffd640 1 -- 172.21.15.31:0/1749390253 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac200b6070 con 0x7fac28152570 2023-10-30T13:35:24.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.008+0000 7fac16ffd640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fac080664d0 0x7fac08068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:24.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.008+0000 7fac16ffd640 1 -- 172.21.15.31:0/1749390253 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac2011ddc0 con 0x7fac28152570 2023-10-30T13:35:24.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.008+0000 7fac2659c640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fac080664d0 0x7fac08068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:24.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.008+0000 7fac2659c640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fac080664d0 0x7fac08068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac180099b0 tx=0x7fac18008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:24.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.012+0000 7fac16ffd640 1 -- 172.21.15.31:0/1749390253 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac281f2450 con 0x7fac28152570 2023-10-30T13:35:24.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.200+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac281f2450 con 0x7fac28152570 2023-10-30T13:35:24.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.200+0000 7fac16ffd640 1 -- 172.21.15.31:0/1749390253 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6519 (secure 0 0 0) 0x7fac281f2450 con 0x7fac28152570 2023-10-30T13:35:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:24.206 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:19.778393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:creating","state_seq":1,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 12 2023-10-30T13:35:24.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fac080664d0 msgr2=0x7fac08068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:24.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fac080664d0 0x7fac08068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac180099b0 tx=0x7fac18008040 comp rx=0 tx=0).stop 2023-10-30T13:35:24.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac28152570 msgr2=0x7fac28145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:24.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac28152570 0x7fac28145900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac2000a830 tx=0x7fac10005800 comp rx=0 tx=0).stop 2023-10-30T13:35:24.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 shutdown_connections 2023-10-30T13:35:24.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fac080664d0 0x7fac08068990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac28152570 0x7fac28145900 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fac28154640 0x7fac28152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 --2- 172.21.15.31:0/1749390253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac10005050 0x7fac281e5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 >> 172.21.15.31:0/1749390253 conn(0x7fac280cac70 msgr2=0x7fac281564a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 shutdown_connections 2023-10-30T13:35:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:24.204+0000 7fac2cdd3640 1 -- 172.21.15.31:0/1749390253 wait complete. 2023-10-30T13:35:24.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T13:35:07.654733+0000', 'modified': '2023-10-30T13:35:19.778393+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4282, 'mds_1': 4276, 'mds_2': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.33:6836/2545654386', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6834', 'nonce': 2545654386}, {'type': 'v1', 'addr': '172.21.15.33:6836', 'nonce': 2545654386}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4276': {'gid': 4276, 'name': 'f', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.33:6837/3402459191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6835', 'nonce': 3402459191}, {'type': 'v1', 'addr': '172.21.15.33:6837', 'nonce': 3402459191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.33:6839/2419808617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 2419808617}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 2419808617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T13:35:24.831 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4273:up:creating 2023-10-30T13:35:24.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:24.918 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:25.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:25.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 Processor -- start 2023-10-30T13:35:25.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 -- start start 2023-10-30T13:35:25.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a94150830 0x7f7a9406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:25.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7a940cb7d0 0x7f7a940c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940c3e40 0x7f7a94070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a940c2310 con 0x7f7a94150830 2023-10-30T13:35:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a94156710 con 0x7f7a940cb7d0 2023-10-30T13:35:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a9ad3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a940c4360 con 0x7f7a940c3e40 2023-10-30T13:35:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a94150830 0x7f7a9406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a94150830 0x7f7a9406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36208/0 (socket says 172.21.15.31:36208) 2023-10-30T13:35:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 -- 172.21.15.31:0/242533382 learned_addr learned my addr 172.21.15.31:0/242533382 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a93fff640 1 --2- 172.21.15.31:0/242533382 >> v2:172.21.15.33:3300/0 conn(0x7f7a940cb7d0 0x7f7a940c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a992db640 1 --2- 172.21.15.31:0/242533382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940c3e40 0x7f7a94070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 -- 172.21.15.31:0/242533382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940c3e40 msgr2=0x7f7a94070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 --2- 172.21.15.31:0/242533382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940c3e40 0x7f7a94070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 -- 172.21.15.31:0/242533382 >> v2:172.21.15.33:3300/0 conn(0x7f7a940cb7d0 msgr2=0x7f7a940c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 --2- 172.21.15.31:0/242533382 >> v2:172.21.15.33:3300/0 conn(0x7f7a940cb7d0 0x7f7a940c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.980+0000 7f7a98ada640 1 -- 172.21.15.31:0/242533382 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a941569b0 con 0x7f7a94150830 2023-10-30T13:35:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a98ada640 1 --2- 172.21.15.31:0/242533382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a9406b500 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a9406f9f0 tx=0x7f7a8801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3421971742076c11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a937fe640 1 -- 172.21.15.31:0/242533382 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a880a89b0 con 0x7f7a94150830 2023-10-30T13:35:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a937fe640 1 -- 172.21.15.31:0/242533382 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a880a8b10 con 0x7f7a94150830 2023-10-30T13:35:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a937fe640 1 -- 172.21.15.31:0/242533382 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a880a8ea0 con 0x7f7a94150830 2023-10-30T13:35:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/242533382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 msgr2=0x7f7a9406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/242533382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a9406b500 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a9406f9f0 tx=0x7f7a8801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/242533382 shutdown_connections 2023-10-30T13:35:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/242533382 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940c3e40 0x7f7a94070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/242533382 >> v2:172.21.15.33:3300/0 conn(0x7f7a940cb7d0 0x7f7a940c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/242533382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a9406b500 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/242533382 >> 172.21.15.31:0/242533382 conn(0x7f7a940cac70 msgr2=0x7f7a940cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/242533382 shutdown_connections 2023-10-30T13:35:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.984+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/242533382 wait complete. 2023-10-30T13:35:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 Processor -- start 2023-10-30T13:35:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 -- start start 2023-10-30T13:35:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a940c3e40 0x7f7a940717d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:25.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940cb7d0 0x7f7a94150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:25.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a941eab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:25.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941565d0 con 0x7f7a94150830 2023-10-30T13:35:25.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941562f0 con 0x7f7a940c3e40 2023-10-30T13:35:25.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94152d40 con 0x7f7a940cb7d0 2023-10-30T13:35:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a941eab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a941eab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36222/0 (socket says 172.21.15.31:36222) 2023-10-30T13:35:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 -- 172.21.15.31:0/1059902908 learned_addr learned my addr 172.21.15.31:0/1059902908 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a98ada640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a940c3e40 0x7f7a940717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a93fff640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940cb7d0 0x7f7a94150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 -- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940cb7d0 msgr2=0x7f7a94150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940cb7d0 0x7f7a94150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 -- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a940c3e40 msgr2=0x7f7a940717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a940c3e40 0x7f7a940717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 -- 172.21.15.31:0/1059902908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880094d0 con 0x7f7a94150830 2023-10-30T13:35:25.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a992db640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a941eab20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c007d10 tx=0x7f7a8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:25.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a917fa640 1 -- 172.21.15.31:0/1059902908 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c014420 con 0x7f7a94150830 2023-10-30T13:35:25.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a917fa640 1 -- 172.21.15.31:0/1059902908 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8c014580 con 0x7f7a94150830 2023-10-30T13:35:25.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a917fa640 1 -- 172.21.15.31:0/1059902908 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c014930 con 0x7f7a94150830 2023-10-30T13:35:25.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a94153020 con 0x7f7a94150830 2023-10-30T13:35:25.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.988+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a941535e0 con 0x7f7a94150830 2023-10-30T13:35:25.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.992+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9404e170 con 0x7f7a94150830 2023-10-30T13:35:26.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.992+0000 7f7a917fa640 1 -- 172.21.15.31:0/1059902908 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a8c014a90 con 0x7f7a94150830 2023-10-30T13:35:26.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.992+0000 7f7a917fa640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a780664d0 0x7f7a78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.992+0000 7f7a917fa640 1 -- 172.21.15.31:0/1059902908 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a8c0845c0 con 0x7f7a94150830 2023-10-30T13:35:26.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.992+0000 7f7a98ada640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a780664d0 0x7f7a78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.992+0000 7f7a98ada640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a780664d0 0x7f7a78068990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a880b2000 tx=0x7f7a88009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:25.996+0000 7f7a917fa640 1 -- 172.21.15.31:0/1059902908 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a8c051ae0 con 0x7f7a94150830 2023-10-30T13:35:26.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.184+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a94070300 con 0x7f7a94150830 2023-10-30T13:35:26.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.184+0000 7f7a917fa640 1 -- 172.21.15.31:0/1059902908 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7f7a8c055100 con 0x7f7a94150830 2023-10-30T13:35:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:26.191 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:26.191 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:35:26.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a780664d0 msgr2=0x7f7a78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a780664d0 0x7f7a78068990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a880b2000 tx=0x7f7a88009040 comp rx=0 tx=0).stop 2023-10-30T13:35:26.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 msgr2=0x7f7a941eab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a941eab20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c007d10 tx=0x7f7a8c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 shutdown_connections 2023-10-30T13:35:26.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a780664d0 0x7f7a78068990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94150830 0x7f7a941eab20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a940cb7d0 0x7f7a94150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 --2- 172.21.15.31:0/1059902908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a940c3e40 0x7f7a940717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 >> 172.21.15.31:0/1059902908 conn(0x7f7a940cac70 msgr2=0x7f7a940c5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 shutdown_connections 2023-10-30T13:35:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.188+0000 7f7a9ad3e640 1 -- 172.21.15.31:0/1059902908 wait complete. 2023-10-30T13:35:26.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T13:35:07.654733+0000', 'modified': '2023-10-30T13:35:24.811863+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4282, 'mds_1': 4276, 'mds_2': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.33:6836/2545654386', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6834', 'nonce': 2545654386}, {'type': 'v1', 'addr': '172.21.15.33:6836', 'nonce': 2545654386}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4276': {'gid': 4276, 'name': 'f', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.33:6837/3402459191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6835', 'nonce': 3402459191}, {'type': 'v1', 'addr': '172.21.15.33:6837', 'nonce': 3402459191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.33:6839/2419808617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 2419808617}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 2419808617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T13:35:26.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-30T13:35:26.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-30T13:35:26.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187c0d6640 1 Processor -- start 2023-10-30T13:35:26.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187c0d6640 1 -- start start 2023-10-30T13:35:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187c0d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 0x7f18740c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187c0d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1874159f20 con 0x7f1874153c80 2023-10-30T13:35:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187c0d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18740c47e0 con 0x7f18740c4a10 2023-10-30T13:35:26.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187c0d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187415dec0 con 0x7f187415cf90 2023-10-30T13:35:26.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879e72640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f1874153c80 0x7f187415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55064/0 (socket says 172.21.15.31:55064) 2023-10-30T13:35:26.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879e72640 1 -- 172.21.15.31:0/2336138028 learned_addr learned my addr 172.21.15.31:0/2336138028 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:26.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187a673640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 0x7f18740c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519796695 0 0) 0x7f1874159f20 con 0x7f1874153c80 2023-10-30T13:35:26.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1860003530 con 0x7f1874153c80 2023-10-30T13:35:26.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1364021674 0 0) 0x7f1860003530 con 0x7f1874153c80 2023-10-30T13:35:26.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 msgr2=0x7f18740c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:26.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 0x7f18740c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 >> v1:172.21.15.33:6789/0 conn(0x7f18740c4a10 legacy=0x7f18740ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187415d620 con 0x7f1874153c80 2023-10-30T13:35:26.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f1870001680 con 0x7f1874153c80 2023-10-30T13:35:26.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 >> v1:172.21.15.31:6789/0 conn(0x7f1874153c80 legacy=0x7f187415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 0x7f18600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1860003ef0 0x7f18600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1860004d10 0x7f1860005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187415dec0 con 0x7f1860003ef0 2023-10-30T13:35:26.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186000b9f0 con 0x7f1860004d10 2023-10-30T13:35:26.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186000bbb0 con 0x7f187415cf90 2023-10-30T13:35:26.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879671640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1860003ef0 0x7f18600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879e72640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 0x7f18600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f187a673640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1860004d10 0x7f1860005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879671640 1 -- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 msgr2=0x7f18600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:26.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879671640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f187415cf90 0x7f18600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879671640 1 -- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1860004d10 msgr2=0x7f1860005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:26.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879671640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1860004d10 0x7f1860005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.964+0000 7f1879671640 1 -- 172.21.15.31:0/2336138028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860003530 con 0x7f1860003ef0 2023-10-30T13:35:26.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f1879671640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1860003ef0 0x7f18600047d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f18740c4e30 tx=0x7f186401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280d8bbd826d09a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640a4ea0 con 0x7f1860003ef0 2023-10-30T13:35:26.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18640a7050 con 0x7f1860003ef0 2023-10-30T13:35:26.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f1878e70640 1 -- 172.21.15.31:0/2336138028 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640ab880 con 0x7f1860003ef0 2023-10-30T13:35:26.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f187c0d6640 1 -- 172.21.15.31:0/2336138028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1860003ef0 msgr2=0x7f18600047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f187c0d6640 1 --2- 172.21.15.31:0/2336138028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1860003ef0 0x7f18600047d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f18740c4e30 tx=0x7f186401c3e0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f187a673640 1 -- 172.21.15.31:0/2336138028 reap_dead start 2023-10-30T13:35:26.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f187c0d6640 1 -- 172.21.15.31:0/2336138028 shutdown_connections 2023-10-30T13:35:26.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f187c0d6640 1 -- 172.21.15.31:0/2336138028 >> 172.21.15.31:0/2336138028 conn(0x7f18740715c0 msgr2=0x7f18740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:26.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f187c0d6640 1 -- 172.21.15.31:0/2336138028 shutdown_connections 2023-10-30T13:35:26.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.968+0000 7f187c0d6640 1 -- 172.21.15.31:0/2336138028 wait complete. 2023-10-30T13:35:26.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 Processor -- start 2023-10-30T13:35:26.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 -- start start 2023-10-30T13:35:26.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1860003ef0 0x7f18741ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18741597d0 0x7f18740ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1874158fe0 0x7f1874155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874158770 con 0x7f1874158fe0 2023-10-30T13:35:26.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874157810 con 0x7f18741597d0 2023-10-30T13:35:26.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1874157af0 con 0x7f1860003ef0 2023-10-30T13:35:26.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1874158fe0 0x7f1874155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1874158fe0 0x7f1874155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36246/0 (socket says 172.21.15.31:36246) 2023-10-30T13:35:26.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 -- 172.21.15.31:0/1139526157 learned_addr learned my addr 172.21.15.31:0/1139526157 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:26.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f1879e72640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1860003ef0 0x7f18741ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f1879671640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18741597d0 0x7f18740ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 -- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1860003ef0 msgr2=0x7f18741ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1860003ef0 0x7f18741ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 -- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18741597d0 msgr2=0x7f18740ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18741597d0 0x7f18740ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 -- 172.21.15.31:0/1139526157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874157dd0 con 0x7f1874158fe0 2023-10-30T13:35:26.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f1879e72640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1860003ef0 0x7f18741ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:26.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187a673640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1874158fe0 0x7f1874155630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f18740c97d0 tx=0x7f186c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f186a7fc640 1 -- 172.21.15.31:0/1139526157 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c00e070 con 0x7f1874158fe0 2023-10-30T13:35:26.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f186a7fc640 1 -- 172.21.15.31:0/1139526157 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c011e10 con 0x7f1874158fe0 2023-10-30T13:35:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f186a7fc640 1 -- 172.21.15.31:0/1139526157 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c0039a0 con 0x7f1874158fe0 2023-10-30T13:35:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18741580b0 con 0x7f1874158fe0 2023-10-30T13:35:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.972+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18740ce830 con 0x7f1874158fe0 2023-10-30T13:35:26.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f186a7fc640 1 -- 172.21.15.31:0/1139526157 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f186c003b00 con 0x7f1874158fe0 2023-10-30T13:35:26.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f186a7fc640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18580664a0 0x7f1858068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f186a7fc640 1 -- 172.21.15.31:0/1139526157 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f186c0845d0 con 0x7f1874158fe0 2023-10-30T13:35:26.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 Processor -- start 2023-10-30T13:35:26.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f1879e72640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18580664a0 0x7f1858068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 -- start start 2023-10-30T13:35:26.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94887bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c94888510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c94892aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c9463bcc0 con 0x560c94888a50 2023-10-30T13:35:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c9469e920 con 0x560c94888130 2023-10-30T13:35:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.976+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c9469be50 con 0x560c94887810 2023-10-30T13:35:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f1879e72640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18580664a0 0x7f1858068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1870001cc0 tx=0x7f1870009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c94892aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f1879e72640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94887bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c94892aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36248/0 (socket says 172.21.15.31:36248) 2023-10-30T13:35:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 -- 172.21.15.31:0/3824717530 learned_addr learned my addr 172.21.15.31:0/3824717530 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f1879671640 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c94888510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 -- 172.21.15.31:0/3824717530 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 msgr2=0x560c94887bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94887bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 -- 172.21.15.31:0/3824717530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 msgr2=0x560c94888510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c94888510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 -- 172.21.15.31:0/3824717530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c94390890 con 0x560c94888a50 2023-10-30T13:35:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187a673640 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c94892aa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f186c0829a0 tx=0x7f186c055100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154e042314ab729f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f18577fe640 1 -- 172.21.15.31:0/3824717530 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c05d970 con 0x560c94888a50 2023-10-30T13:35:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f18577fe640 1 -- 172.21.15.31:0/3824717530 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c003db0 con 0x560c94888a50 2023-10-30T13:35:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f18577fe640 1 -- 172.21.15.31:0/3824717530 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c051d80 con 0x560c94888a50 2023-10-30T13:35:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- 172.21.15.31:0/3824717530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 msgr2=0x560c94892aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c94892aa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f186c0829a0 tx=0x7f186c055100 comp rx=0 tx=0).stop 2023-10-30T13:35:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- 172.21.15.31:0/3824717530 shutdown_connections 2023-10-30T13:35:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c94892aa0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c94888510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 --2- 172.21.15.31:0/3824717530 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94887bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- 172.21.15.31:0/3824717530 >> 172.21.15.31:0/3824717530 conn(0x560c9483dfc0 msgr2=0x560c948855d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- 172.21.15.31:0/3824717530 shutdown_connections 2023-10-30T13:35:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- 172.21.15.31:0/3824717530 wait complete. 2023-10-30T13:35:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- start start 2023-10-30T13:35:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 Processor -- start 2023-10-30T13:35:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94881b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888130 0x560c948820a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888a50 0x560c948825e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c94697740 con 0x560c94888130 2023-10-30T13:35:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c94656d10 con 0x560c94888a50 2023-10-30T13:35:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c946569b0 con 0x560c94887810 2023-10-30T13:35:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f187f45e000 10 client.0 _ll_register_callbacks cb 0x560c94832390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:35:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f1879671640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888130 0x560c948820a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.980+0000 7f1879671640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888130 0x560c948820a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36252/0 (socket says 172.21.15.31:36252) 2023-10-30T13:35:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879671640 1 -- 172.21.15.31:0/1011371401 learned_addr learned my addr 172.21.15.31:0/1011371401 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879e72640 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94881b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187a673640 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888a50 0x560c948825e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879671640 1 -- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 msgr2=0x560c94881b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879671640 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94881b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879671640 1 -- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888a50 msgr2=0x560c948825e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879671640 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888a50 0x560c948825e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879671640 1 -- 172.21.15.31:0/1011371401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18740ce830 con 0x560c94888130 2023-10-30T13:35:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879671640 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888130 0x560c948820a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f186401c420 tx=0x7f1864019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f39ec324aa0275 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-10-30T13:35:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1854ff9640 1 -- 172.21.15.31:0/1011371401 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640069e0 con 0x560c94888130 2023-10-30T13:35:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c9464b370 con 0x560c94888130 2023-10-30T13:35:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1854ff9640 1 -- 172.21.15.31:0/1011371401 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1864006b40 con 0x560c94888130 2023-10-30T13:35:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1854ff9640 1 -- 172.21.15.31:0/1011371401 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18640a4ea0 con 0x560c94888130 2023-10-30T13:35:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 10 client.4644 fetch_fsmap learned FSMap version 13 2023-10-30T13:35:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x560c94659460 con 0x560c94888130 2023-10-30T13:35:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1854ff9640 1 -- 172.21.15.31:0/1011371401 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560c9464b370 con 0x560c94888130 2023-10-30T13:35:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1854ff9640 1 -- 172.21.15.31:0/1011371401 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f1864006d00 con 0x560c94888130 2023-10-30T13:35:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 10 client.4644 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:35:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 10 client.4644 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T13:35:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 20 client.4644 populate_metadata read hostname 'smithi031' 2023-10-30T13:35:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c9489b260 0x560c9489d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 4 client.4644 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:35:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 --> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c946312d0 con 0x560c9489b260 2023-10-30T13:35:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879e72640 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c9489b260 0x560c9489d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1879e72640 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c9489b260 0x560c9489d650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac06e56fc1691e60 server_cookie=4e988f5ec97865ae in_seq=0 out_seq=0 2023-10-30T13:35:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:26.984+0000 7f1854ff9640 10 client.4644 ms_handle_connect on v2:172.21.15.33:6834/2545654386 2023-10-30T13:35:30.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f1854ff9640 1 -- 172.21.15.31:0/1011371401 <== mds.2 v2:172.21.15.33:6834/2545654386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560c946312d0 con 0x560c9489b260 2023-10-30T13:35:30.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f1854ff9640 10 client.4644 handle_command_reply: tid=0 2023-10-30T13:35:30.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 client.4644 shutdown 2023-10-30T13:35:30.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888130 msgr2=0x560c948820a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888130 0x560c948820a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f186401c420 tx=0x7f1864019180 comp rx=0 tx=0).stop 2023-10-30T13:35:30.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 shutdown_connections 2023-10-30T13:35:30.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c9489b260 0x560c9489d650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888a50 0x560c948825e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888130 0x560c948820a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 --2- 172.21.15.31:0/1011371401 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c94881b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f1854ff9640 0 client.4644 ms_handle_reset on v2:172.21.15.33:6834/2545654386 2023-10-30T13:35:30.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 >> 172.21.15.31:0/1011371401 conn(0x560c9483dfc0 msgr2=0x560c94887090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:30.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 shutdown_connections 2023-10-30T13:35:30.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 1 -- 172.21.15.31:0/1011371401 wait complete. 2023-10-30T13:35:30.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.820+0000 7f187f45e000 20 client.4644 trim_cache size 0 max 16384 2023-10-30T13:35:30.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 Processor -- start 2023-10-30T13:35:30.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 -- start start 2023-10-30T13:35:30.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c948e3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c948e4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c94351940 con 0x560c94888a50 2023-10-30T13:35:30.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c943911b0 con 0x560c94888130 2023-10-30T13:35:30.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c94691a40 con 0x560c94887810 2023-10-30T13:35:30.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f1879e72640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c948e3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36256/0 (socket says 172.21.15.31:36256) 2023-10-30T13:35:30.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f187a673640 1 -- 172.21.15.31:0/1718494816 learned_addr learned my addr 172.21.15.31:0/1718494816 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:30.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.836+0000 7f1879671640 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c948e4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1879e72640 1 -- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 msgr2=0x560c948e4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:30.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1879e72640 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c948e4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1879e72640 1 -- 172.21.15.31:0/1718494816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 msgr2=0x560c948e4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:30.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1879e72640 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1879e72640 1 -- 172.21.15.31:0/1718494816 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c9463f760 con 0x560c94887810 2023-10-30T13:35:30.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1879e72640 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c948e3dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x560c9461cb00 tx=0x7f187000c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef02ac0074cdc0f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:30.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1857fff640 1 -- 172.21.15.31:0/1718494816 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187001c040 con 0x560c94887810 2023-10-30T13:35:30.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1857fff640 1 -- 172.21.15.31:0/1718494816 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1870031030 con 0x560c94887810 2023-10-30T13:35:30.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f1857fff640 1 -- 172.21.15.31:0/1718494816 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187002c130 con 0x560c94887810 2023-10-30T13:35:30.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 -- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 msgr2=0x560c948e3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c948e3dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x560c9461cb00 tx=0x7f187000c000 comp rx=0 tx=0).stop 2023-10-30T13:35:30.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 -- 172.21.15.31:0/1718494816 shutdown_connections 2023-10-30T13:35:30.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94888130 0x560c948e4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 --2- 172.21.15.31:0/1718494816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94887810 0x560c948e3dd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 -- 172.21.15.31:0/1718494816 >> 172.21.15.31:0/1718494816 conn(0x560c9483dfc0 msgr2=0x560c9488d290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:30.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 -- 172.21.15.31:0/1718494816 shutdown_connections 2023-10-30T13:35:30.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 -- 172.21.15.31:0/1718494816 wait complete. 2023-10-30T13:35:30.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 -- start start 2023-10-30T13:35:30.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.840+0000 7f187f45e000 1 Processor -- start 2023-10-30T13:35:30.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94887810 0x560c948eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94888130 0x560c948e50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c946965d0 con 0x560c94888a50 2023-10-30T13:35:30.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c946d8ba0 con 0x560c94887810 2023-10-30T13:35:30.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c9469a2a0 con 0x560c94888130 2023-10-30T13:35:30.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 10 client.0 _ll_register_callbacks cb 0x7f186c051a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:35:30.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36262/0 (socket says 172.21.15.31:36262) 2023-10-30T13:35:30.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 -- 172.21.15.31:0/1289655007 learned_addr learned my addr 172.21.15.31:0/1289655007 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:30.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1879e72640 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94887810 0x560c948eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1879671640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94888130 0x560c948e50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 -- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94888130 msgr2=0x560c948e50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:30.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94888130 0x560c948e50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 -- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94887810 msgr2=0x560c948eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:30.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94887810 0x560c948eb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 -- 172.21.15.31:0/1289655007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187415d620 con 0x560c94888a50 2023-10-30T13:35:30.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187a673640 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e5620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f186c051b70 tx=0x7f186c059040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f58fe8979fb483c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:30.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-10-30T13:35:30.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1854ff9640 1 -- 172.21.15.31:0/1289655007 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c05d970 con 0x560c94888a50 2023-10-30T13:35:30.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1854ff9640 1 -- 172.21.15.31:0/1289655007 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f186c06e040 con 0x560c94888a50 2023-10-30T13:35:30.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c948999b0 con 0x560c94888a50 2023-10-30T13:35:30.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1854ff9640 1 -- 172.21.15.31:0/1289655007 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f186c053a10 con 0x560c94888a50 2023-10-30T13:35:30.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 10 client.4650 fetch_fsmap learned FSMap version 13 2023-10-30T13:35:30.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x560c94390cc0 con 0x560c94888a50 2023-10-30T13:35:30.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1854ff9640 1 -- 172.21.15.31:0/1289655007 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560c948999b0 con 0x560c94888a50 2023-10-30T13:35:30.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1854ff9640 1 -- 172.21.15.31:0/1289655007 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f186c005d20 con 0x560c94888a50 2023-10-30T13:35:30.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 10 client.4650 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:35:30.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 10 client.4650 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T13:35:30.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 20 client.4650 populate_metadata read hostname 'smithi031' 2023-10-30T13:35:30.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c948e8fc0 0x560c948e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 4 client.4650 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:35:30.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 --> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560c9462f6f0 con 0x560c948e8fc0 2023-10-30T13:35:30.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1879e72640 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c948e8fc0 0x560c948e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1879e72640 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c948e8fc0 0x560c948e93a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=650ce2f4cdde6812 server_cookie=552c72a05d82a1dc in_seq=0 out_seq=0 2023-10-30T13:35:30.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.844+0000 7f1854ff9640 10 client.4650 ms_handle_connect on v2:172.21.15.33:6834/2545654386 2023-10-30T13:35:30.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f1854ff9640 1 -- 172.21.15.31:0/1289655007 <== mds.2 v2:172.21.15.33:6834/2545654386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x560c9462f6f0 con 0x560c948e8fc0 2023-10-30T13:35:30.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f1854ff9640 10 client.4650 handle_command_reply: tid=0 2023-10-30T13:35:30.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 client.4650 shutdown 2023-10-30T13:35:30.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 msgr2=0x560c948e5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e5620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f186c051b70 tx=0x7f186c059040 comp rx=0 tx=0).stop 2023-10-30T13:35:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 shutdown_connections 2023-10-30T13:35:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:6834/2545654386,v1:172.21.15.33:6836/2545654386] conn(0x560c948e8fc0 0x560c948e93a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c94888a50 0x560c948e5620 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x560c94888130 0x560c948e50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 --2- 172.21.15.31:0/1289655007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c94887810 0x560c948eb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 >> 172.21.15.31:0/1289655007 conn(0x560c9483dfc0 msgr2=0x560c9488f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f1854ff9640 0 client.4650 ms_handle_reset on v2:172.21.15.33:6834/2545654386 2023-10-30T13:35:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 shutdown_connections 2023-10-30T13:35:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 1 -- 172.21.15.31:0/1289655007 wait complete. 2023-10-30T13:35:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187f45e000 20 client.4650 trim_cache size 0 max 16384 2023-10-30T13:35:30.862 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:35:30.862 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "62b651f4-6cfc-4dcb-91a0-e9335b7700dd", 2023-10-30T13:35:30.862 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2023-10-30T13:35:30.862 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4273, 2023-10-30T13:35:30.863 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-10-30T13:35:30.863 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-10-30T13:35:30.863 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-10-30T13:35:30.863 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 11.063971415999999, 2023-10-30T13:35:30.863 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 13, 2023-10-30T13:35:30.863 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 21, 2023-10-30T13:35:30.864 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 21, 2023-10-30T13:35:30.864 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 30.329765881 2023-10-30T13:35:30.864 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:35:30.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18580664a0 msgr2=0x7f1858068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18580664a0 0x7f1858068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1870001cc0 tx=0x7f1870009040 comp rx=0 tx=0).stop 2023-10-30T13:35:30.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1874158fe0 msgr2=0x7f1874155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1874158fe0 0x7f1874155630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f18740c97d0 tx=0x7f186c0115a0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 shutdown_connections 2023-10-30T13:35:30.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18580664a0 0x7f1858068960 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1874158fe0 0x7f1874155630 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18741597d0 0x7f18740ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 --2- 172.21.15.31:0/1139526157 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1860003ef0 0x7f18741ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 >> 172.21.15.31:0/1139526157 conn(0x7f18740715c0 msgr2=0x7f1874150f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:30.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 shutdown_connections 2023-10-30T13:35:30.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:30.848+0000 7f187c0d6640 1 -- 172.21.15.31:0/1139526157 wait complete. 2023-10-30T13:35:30.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-30T13:35:31.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb3e794640 1 Processor -- start 2023-10-30T13:35:31.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb3e794640 1 -- start start 2023-10-30T13:35:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb3e794640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbb381537f0 0x7fbb380701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb3e794640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb380c2aa0 con 0x7fbb3815a2e0 2023-10-30T13:35:31.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb3e794640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb380708e0 con 0x7fbb3806b2f0 2023-10-30T13:35:31.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb3e794640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb380c3ab0 con 0x7fbb381537f0 2023-10-30T13:35:31.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb37fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbb381537f0 0x7fbb380701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb37fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbb381537f0 0x7fbb380701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36264/0 (socket says 172.21.15.31:36264) 2023-10-30T13:35:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb37fff640 1 -- 172.21.15.31:0/1096117339 learned_addr learned my addr 172.21.15.31:0/1096117339 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb37fff640 1 -- 172.21.15.31:0/1096117339 >> v1:172.21.15.33:6789/0 conn(0x7fbb3806b2f0 legacy=0x7fbb3806b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb37fff640 1 -- 172.21.15.31:0/1096117339 >> v1:172.21.15.31:6789/0 conn(0x7fbb3815a2e0 legacy=0x7fbb380ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.020+0000 7fbb37fff640 1 -- 172.21.15.31:0/1096117339 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb380c1eb0 con 0x7fbb381537f0 2023-10-30T13:35:31.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb37fff640 1 --2- 172.21.15.31:0/1096117339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb381537f0 0x7fbb380701a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009870 tx=0x7fbb2801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c3d0c61673755c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb36ffd640 1 -- 172.21.15.31:0/1096117339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280a8a30 con 0x7fbb381537f0 2023-10-30T13:35:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb36ffd640 1 -- 172.21.15.31:0/1096117339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb280a8b90 con 0x7fbb381537f0 2023-10-30T13:35:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb36ffd640 1 -- 172.21.15.31:0/1096117339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280b0440 con 0x7fbb381537f0 2023-10-30T13:35:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb3e794640 1 -- 172.21.15.31:0/1096117339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb381537f0 msgr2=0x7fbb380701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb3e794640 1 --2- 172.21.15.31:0/1096117339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb381537f0 0x7fbb380701a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009870 tx=0x7fbb2801ed30 comp rx=0 tx=0).stop 2023-10-30T13:35:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb3e794640 1 -- 172.21.15.31:0/1096117339 shutdown_connections 2023-10-30T13:35:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb3e794640 1 --2- 172.21.15.31:0/1096117339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb381537f0 0x7fbb380701a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb3e794640 1 -- 172.21.15.31:0/1096117339 >> 172.21.15.31:0/1096117339 conn(0x7fbb38071fb0 msgr2=0x7fbb380723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb3e794640 1 -- 172.21.15.31:0/1096117339 shutdown_connections 2023-10-30T13:35:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.024+0000 7fbb3e794640 1 -- 172.21.15.31:0/1096117339 wait complete. 2023-10-30T13:35:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 Processor -- start 2023-10-30T13:35:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 -- start start 2023-10-30T13:35:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb3806b2f0 0x7fbb380c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbb381537f0 0x7fbb380c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb3815a2e0 0x7fbb380c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38154bb0 con 0x7fbb3815a2e0 2023-10-30T13:35:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380ca5a0 con 0x7fbb3806b2f0 2023-10-30T13:35:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38154630 con 0x7fbb381537f0 2023-10-30T13:35:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb3815a2e0 0x7fbb380c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb3815a2e0 0x7fbb380c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36278/0 (socket says 172.21.15.31:36278) 2023-10-30T13:35:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb3806b2f0 0x7fbb380c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 -- 172.21.15.31:0/98442671 learned_addr learned my addr 172.21.15.31:0/98442671 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb377fe640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbb381537f0 0x7fbb380c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 -- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbb381537f0 msgr2=0x7fbb380c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbb381537f0 0x7fbb380c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 -- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb3806b2f0 msgr2=0x7fbb380c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb3806b2f0 0x7fbb380c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 -- 172.21.15.31:0/98442671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb280094d0 con 0x7fbb3815a2e0 2023-10-30T13:35:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb3815a2e0 0x7fbb380c4440 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000c770 tx=0x7fbb3000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb34ff9640 1 -- 172.21.15.31:0/98442671 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30003b80 con 0x7fbb3815a2e0 2023-10-30T13:35:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb34ff9640 1 -- 172.21.15.31:0/98442671 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30003ce0 con 0x7fbb3815a2e0 2023-10-30T13:35:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb381548d0 con 0x7fbb3815a2e0 2023-10-30T13:35:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb34ff9640 1 -- 172.21.15.31:0/98442671 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3000aa30 con 0x7fbb3815a2e0 2023-10-30T13:35:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.028+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb380ca230 con 0x7fbb3815a2e0 2023-10-30T13:35:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb34ff9640 1 -- 172.21.15.31:0/98442671 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb3000ab90 con 0x7fbb3815a2e0 2023-10-30T13:35:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb34ff9640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbb240664a0 0x7fbb24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb34ff9640 1 -- 172.21.15.31:0/98442671 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb30083c30 con 0x7fbb3815a2e0 2023-10-30T13:35:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb37fff640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbb240664a0 0x7fbb24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 Processor -- start 2023-10-30T13:35:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 -- start start 2023-10-30T13:35:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb37fff640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbb240664a0 0x7fbb24068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbb280b2000 tx=0x7fbb280aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b082bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083130 0x561b5b083510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b08daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae36cc0 con 0x561b5b083a50 2023-10-30T13:35:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae99920 con 0x561b5b083130 2023-10-30T13:35:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.032+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae96e50 con 0x561b5b082810 2023-10-30T13:35:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b08daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b08daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36282/0 (socket says 172.21.15.31:36282) 2023-10-30T13:35:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 -- 172.21.15.31:0/4110619303 learned_addr learned my addr 172.21.15.31:0/4110619303 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb37fff640 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b082bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb377fe640 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083130 0x561b5b083510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 -- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 msgr2=0x561b5b082bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b082bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 -- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083130 msgr2=0x561b5b083510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083130 0x561b5b083510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 -- 172.21.15.31:0/4110619303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b5ab8b890 con 0x561b5b083a50 2023-10-30T13:35:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b08daa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb30054cf0 tx=0x7fbb300014b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbed011f1a31612 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb1dffb640 1 -- 172.21.15.31:0/4110619303 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30054d60 con 0x561b5b083a50 2023-10-30T13:35:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb1dffb640 1 -- 172.21.15.31:0/4110619303 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30058eb0 con 0x561b5b083a50 2023-10-30T13:35:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb1dffb640 1 -- 172.21.15.31:0/4110619303 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3005d060 con 0x561b5b083a50 2023-10-30T13:35:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- 172.21.15.31:0/4110619303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 msgr2=0x561b5b08daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b08daa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb30054cf0 tx=0x7fbb300014b0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- 172.21.15.31:0/4110619303 shutdown_connections 2023-10-30T13:35:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b08daa0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083130 0x561b5b083510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 --2- 172.21.15.31:0/4110619303 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b082bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- 172.21.15.31:0/4110619303 >> 172.21.15.31:0/4110619303 conn(0x561b5b038fc0 msgr2=0x561b5b0805d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- 172.21.15.31:0/4110619303 shutdown_connections 2023-10-30T13:35:31.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- 172.21.15.31:0/4110619303 wait complete. 2023-10-30T13:35:31.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- start start 2023-10-30T13:35:31.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 Processor -- start 2023-10-30T13:35:31.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b07cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b07d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b088c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae92740 con 0x561b5b083130 2023-10-30T13:35:31.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae51d10 con 0x561b5b083a50 2023-10-30T13:35:31.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae519b0 con 0x561b5b082810 2023-10-30T13:35:31.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb41b18000 10 client.0 _ll_register_callbacks cb 0x561b5b02d390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:35:31.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b07d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b07d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36294/0 (socket says 172.21.15.31:36294) 2023-10-30T13:35:31.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb377fe640 1 -- 172.21.15.31:0/3286418964 learned_addr learned my addr 172.21.15.31:0/3286418964 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb37fff640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b07cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.036+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b088c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb377fe640 1 -- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 msgr2=0x561b5b07cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb377fe640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b07cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb377fe640 1 -- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 msgr2=0x561b5b088c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb377fe640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b088c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb377fe640 1 -- 172.21.15.31:0/3286418964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb380ca230 con 0x561b5b083130 2023-10-30T13:35:31.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb37fff640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b07cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb377fe640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b07d490 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb381511d0 tx=0x7fbb2c001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f5d757b243b492 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-10-30T13:35:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb16ffd640 1 -- 172.21.15.31:0/3286418964 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c001b40 con 0x561b5b083130 2023-10-30T13:35:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb16ffd640 1 -- 172.21.15.31:0/3286418964 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb2c001ca0 con 0x561b5b083130 2023-10-30T13:35:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b5ae46370 con 0x561b5b083130 2023-10-30T13:35:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb16ffd640 1 -- 172.21.15.31:0/3286418964 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb2c014400 con 0x561b5b083130 2023-10-30T13:35:31.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 10 client.4665 fetch_fsmap learned FSMap version 13 2023-10-30T13:35:31.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x561b5ae54460 con 0x561b5b083130 2023-10-30T13:35:31.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb16ffd640 1 -- 172.21.15.31:0/3286418964 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b5ae46370 con 0x561b5b083130 2023-10-30T13:35:31.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb16ffd640 1 -- 172.21.15.31:0/3286418964 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fbb2c014560 con 0x561b5b083130 2023-10-30T13:35:31.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 10 client.4665 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:35:31.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 10 client.4665 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T13:35:31.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 20 client.4665 populate_metadata read hostname 'smithi031' 2023-10-30T13:35:31.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0899a0 0x561b5b089d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 4 client.4665 mds_command: new command op to 4276 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:35:31.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 --> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b5ae2c2d0 con 0x561b5b0899a0 2023-10-30T13:35:31.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb37fff640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0899a0 0x561b5b089d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb37fff640 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0899a0 0x561b5b089d80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c7d32108ccfab9d server_cookie=b65f0846d8b781c8 in_seq=0 out_seq=0 2023-10-30T13:35:31.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.040+0000 7fbb16ffd640 10 client.4665 ms_handle_connect on v2:172.21.15.33:6835/3402459191 2023-10-30T13:35:31.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb16ffd640 1 -- 172.21.15.31:0/3286418964 <== mds.1 v2:172.21.15.33:6835/3402459191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561b5ae2c2d0 con 0x561b5b0899a0 2023-10-30T13:35:31.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb16ffd640 10 client.4665 handle_command_reply: tid=0 2023-10-30T13:35:31.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 client.4665 shutdown 2023-10-30T13:35:31.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 msgr2=0x561b5b07d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b07d490 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb381511d0 tx=0x7fbb2c001130 comp rx=0 tx=0).stop 2023-10-30T13:35:31.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 shutdown_connections 2023-10-30T13:35:31.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0899a0 0x561b5b089d80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b088c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b07d490 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb16ffd640 0 client.4665 ms_handle_reset on v2:172.21.15.33:6835/3402459191 2023-10-30T13:35:31.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3286418964 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b07cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 >> 172.21.15.31:0/3286418964 conn(0x561b5b038fc0 msgr2=0x561b5b082090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 shutdown_connections 2023-10-30T13:35:31.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 1 -- 172.21.15.31:0/3286418964 wait complete. 2023-10-30T13:35:31.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.044+0000 7fbb41b18000 20 client.4665 trim_cache size 0 max 16384 2023-10-30T13:35:31.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 Processor -- start 2023-10-30T13:35:31.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- start start 2023-10-30T13:35:31.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b082810 0x561b5b046c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b083130 0x561b5b047180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5ab4c940 con 0x561b5b083a50 2023-10-30T13:35:31.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae8e0b0 con 0x561b5b082810 2023-10-30T13:35:31.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae8fb40 con 0x561b5b083130 2023-10-30T13:35:31.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b0863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36304/0 (socket says 172.21.15.31:36304) 2023-10-30T13:35:31.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 -- 172.21.15.31:0/3446784456 learned_addr learned my addr 172.21.15.31:0/3446784456 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb37fff640 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b082810 0x561b5b046c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb377fe640 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b083130 0x561b5b047180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 -- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b083130 msgr2=0x561b5b047180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b083130 0x561b5b047180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 -- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b082810 msgr2=0x561b5b046c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b082810 0x561b5b046c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 -- 172.21.15.31:0/3446784456 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b5ae3a760 con 0x561b5b083a50 2023-10-30T13:35:31.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb377fe640 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b083130 0x561b5b047180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:31.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b0863e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000a430 tx=0x7fbb30005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ed564747c1f2cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb1e7fc640 1 -- 172.21.15.31:0/3446784456 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb300510a0 con 0x561b5b083a50 2023-10-30T13:35:31.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb1e7fc640 1 -- 172.21.15.31:0/3446784456 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30058eb0 con 0x561b5b083a50 2023-10-30T13:35:31.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb1e7fc640 1 -- 172.21.15.31:0/3446784456 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30054d60 con 0x561b5b083a50 2023-10-30T13:35:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- 172.21.15.31:0/3446784456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 msgr2=0x561b5b0863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b0863e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000a430 tx=0x7fbb30005040 comp rx=0 tx=0).stop 2023-10-30T13:35:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- 172.21.15.31:0/3446784456 shutdown_connections 2023-10-30T13:35:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083a50 0x561b5b0863e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b083130 0x561b5b047180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 --2- 172.21.15.31:0/3446784456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b082810 0x561b5b046c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- 172.21.15.31:0/3446784456 >> 172.21.15.31:0/3446784456 conn(0x561b5b038fc0 msgr2=0x561b5b08b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- 172.21.15.31:0/3446784456 shutdown_connections 2023-10-30T13:35:31.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- 172.21.15.31:0/3446784456 wait complete. 2023-10-30T13:35:31.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 -- start start 2023-10-30T13:35:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.060+0000 7fbb41b18000 1 Processor -- start 2023-10-30T13:35:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b0de660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b0deba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b0df840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5aed3ba0 con 0x561b5b083130 2023-10-30T13:35:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae8ca40 con 0x561b5b083a50 2023-10-30T13:35:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b5ae514d0 con 0x561b5b082810 2023-10-30T13:35:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 10 client.0 _ll_register_callbacks cb 0x7fbb3000cf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:35:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b0deba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b0deba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36318/0 (socket says 172.21.15.31:36318) 2023-10-30T13:35:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 -- 172.21.15.31:0/2364495002 learned_addr learned my addr 172.21.15.31:0/2364495002 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb37fff640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b0de660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b0df840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 -- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 msgr2=0x561b5b0de660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b0de660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 -- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 msgr2=0x561b5b0df840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b0df840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 -- 172.21.15.31:0/2364495002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b5ab8b890 con 0x561b5b083130 2023-10-30T13:35:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb3cd31640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b0df840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb377fe640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b0deba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009010 tx=0x7fbb2c001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35dbdf9c94fc7348 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-10-30T13:35:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb16ffd640 1 -- 172.21.15.31:0/2364495002 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c014400 con 0x561b5b083130 2023-10-30T13:35:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb16ffd640 1 -- 172.21.15.31:0/2364495002 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb2c014560 con 0x561b5b083130 2023-10-30T13:35:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b5ae2a6f0 con 0x561b5b083130 2023-10-30T13:35:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb16ffd640 1 -- 172.21.15.31:0/2364495002 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb2c003600 con 0x561b5b083130 2023-10-30T13:35:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 10 client.4671 fetch_fsmap learned FSMap version 13 2023-10-30T13:35:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x561b5ab8bcc0 con 0x561b5b083130 2023-10-30T13:35:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb16ffd640 1 -- 172.21.15.31:0/2364495002 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b5ae2a6f0 con 0x561b5b083130 2023-10-30T13:35:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb16ffd640 1 -- 172.21.15.31:0/2364495002 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fbb2c003760 con 0x561b5b083130 2023-10-30T13:35:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 10 client.4671 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:35:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 10 client.4671 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T13:35:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 20 client.4671 populate_metadata read hostname 'smithi031' 2023-10-30T13:35:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0dfd80 0x561b5b0e0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 4 client.4671 mds_command: new command op to 4276 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:35:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 --> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561b5ae8b720 con 0x561b5b0dfd80 2023-10-30T13:35:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb37fff640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0dfd80 0x561b5b0e0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb16ffd640 10 client.4671 ms_handle_connect on v2:172.21.15.33:6835/3402459191 2023-10-30T13:35:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.064+0000 7fbb37fff640 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0dfd80 0x561b5b0e0160 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4219a4dfbac95181 server_cookie=c78a63e87f2437e0 in_seq=0 out_seq=0 2023-10-30T13:35:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb16ffd640 1 -- 172.21.15.31:0/2364495002 <== mds.1 v2:172.21.15.33:6835/3402459191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x561b5ae8b720 con 0x561b5b0dfd80 2023-10-30T13:35:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb16ffd640 10 client.4671 handle_command_reply: tid=0 2023-10-30T13:35:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 client.4671 shutdown 2023-10-30T13:35:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 msgr2=0x561b5b0deba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b0deba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009010 tx=0x7fbb2c001090 comp rx=0 tx=0).stop 2023-10-30T13:35:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 shutdown_connections 2023-10-30T13:35:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:6835/3402459191,v1:172.21.15.33:6837/3402459191] conn(0x561b5b0dfd80 0x561b5b0e0160 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b5b083a50 0x561b5b0df840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b5b083130 0x561b5b0deba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb16ffd640 0 client.4671 ms_handle_reset on v2:172.21.15.33:6835/3402459191 2023-10-30T13:35:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 --2- 172.21.15.31:0/2364495002 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x561b5b082810 0x561b5b0de660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 >> 172.21.15.31:0/2364495002 conn(0x561b5b038fc0 msgr2=0x561b5b089d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 shutdown_connections 2023-10-30T13:35:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 1 -- 172.21.15.31:0/2364495002 wait complete. 2023-10-30T13:35:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb41b18000 20 client.4671 trim_cache size 0 max 16384 2023-10-30T13:35:31.083 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:35:31.083 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "62b651f4-6cfc-4dcb-91a0-e9335b7700dd", 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4276, 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 16.730694634999999, 2023-10-30T13:35:31.085 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 13, 2023-10-30T13:35:31.085 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 21, 2023-10-30T13:35:31.085 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 21, 2023-10-30T13:35:31.085 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 30.548586101000001 2023-10-30T13:35:31.085 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:35:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbb240664a0 msgr2=0x7fbb24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbb240664a0 0x7fbb24068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbb280b2000 tx=0x7fbb280aa070 comp rx=0 tx=0).stop 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb3815a2e0 msgr2=0x7fbb380c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb3815a2e0 0x7fbb380c4440 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000c770 tx=0x7fbb3000cc30 comp rx=0 tx=0).stop 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 shutdown_connections 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbb240664a0 0x7fbb24068960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb3815a2e0 0x7fbb380c4440 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbb381537f0 0x7fbb380c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 --2- 172.21.15.31:0/98442671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb3806b2f0 0x7fbb380c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 >> 172.21.15.31:0/98442671 conn(0x7fbb38071fb0 msgr2=0x7fbb3806be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 shutdown_connections 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.068+0000 7fbb3e794640 1 -- 172.21.15.31:0/98442671 wait complete. 2023-10-30T13:35:31.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-30T13:35:31.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe572303640 1 Processor -- start 2023-10-30T13:35:31.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe572303640 1 -- start start 2023-10-30T13:35:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe572303640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe56c0c8b00 0x7fe56c0c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe572303640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe56c0b22a0 0x7fe56c0b3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe572303640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c0b27f0 con 0x7fe56c15cdf0 2023-10-30T13:35:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe572303640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c15d370 con 0x7fe56c0b22a0 2023-10-30T13:35:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe572303640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c15d630 con 0x7fe56c0c8b00 2023-10-30T13:35:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571b02640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe56c0b22a0 0x7fe56c0b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571b02640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe56c0b22a0 0x7fe56c0b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36320/0 (socket says 172.21.15.31:36320) 2023-10-30T13:35:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571b02640 1 -- 172.21.15.31:0/2922208122 learned_addr learned my addr 172.21.15.31:0/2922208122 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571301640 1 --2- 172.21.15.31:0/2922208122 >> v2:172.21.15.33:3300/0 conn(0x7fe56c0c8b00 0x7fe56c0c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571b02640 1 -- 172.21.15.31:0/2922208122 >> v2:172.21.15.33:3300/0 conn(0x7fe56c0c8b00 msgr2=0x7fe56c0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571b02640 1 --2- 172.21.15.31:0/2922208122 >> v2:172.21.15.33:3300/0 conn(0x7fe56c0c8b00 0x7fe56c0c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571b02640 1 -- 172.21.15.31:0/2922208122 >> v1:172.21.15.33:6789/0 conn(0x7fe56c15cdf0 legacy=0x7fe56c0c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.232+0000 7fe571b02640 1 -- 172.21.15.31:0/2922208122 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c15d910 con 0x7fe56c0b22a0 2023-10-30T13:35:31.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe571b02640 1 --2- 172.21.15.31:0/2922208122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c0b22a0 0x7fe56c0b3240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe56c06ca60 tx=0x7fe56401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3c8b88b8e7d145 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe563fff640 1 -- 172.21.15.31:0/2922208122 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5640a5860 con 0x7fe56c0b22a0 2023-10-30T13:35:31.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe563fff640 1 -- 172.21.15.31:0/2922208122 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5640a59c0 con 0x7fe56c0b22a0 2023-10-30T13:35:31.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe563fff640 1 -- 172.21.15.31:0/2922208122 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5640a5d50 con 0x7fe56c0b22a0 2023-10-30T13:35:31.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 -- 172.21.15.31:0/2922208122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c0b22a0 msgr2=0x7fe56c0b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 --2- 172.21.15.31:0/2922208122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c0b22a0 0x7fe56c0b3240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe56c06ca60 tx=0x7fe56401c9a0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 -- 172.21.15.31:0/2922208122 shutdown_connections 2023-10-30T13:35:31.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 --2- 172.21.15.31:0/2922208122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c0b22a0 0x7fe56c0b3240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 --2- 172.21.15.31:0/2922208122 >> v2:172.21.15.33:3300/0 conn(0x7fe56c0c8b00 0x7fe56c0c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 -- 172.21.15.31:0/2922208122 >> 172.21.15.31:0/2922208122 conn(0x7fe56c06bed0 msgr2=0x7fe56c06c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 -- 172.21.15.31:0/2922208122 shutdown_connections 2023-10-30T13:35:31.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.236+0000 7fe572303640 1 -- 172.21.15.31:0/2922208122 wait complete. 2023-10-30T13:35:31.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 Processor -- start 2023-10-30T13:35:31.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 -- start start 2023-10-30T13:35:31.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe56c0b22a0 0x7fe56c15de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe56c0c8b00 0x7fe56c1f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c15cdf0 0x7fe56c1f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c1f58a0 con 0x7fe56c15cdf0 2023-10-30T13:35:31.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c1f1df0 con 0x7fe56c0c8b00 2023-10-30T13:35:31.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c1f5b30 con 0x7fe56c0b22a0 2023-10-30T13:35:31.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c15cdf0 0x7fe56c1f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c15cdf0 0x7fe56c1f5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36328/0 (socket says 172.21.15.31:36328) 2023-10-30T13:35:31.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 -- 172.21.15.31:0/2744348808 learned_addr learned my addr 172.21.15.31:0/2744348808 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571301640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe56c0b22a0 0x7fe56c15de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe570b00640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe56c0c8b00 0x7fe56c1f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 -- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe56c0b22a0 msgr2=0x7fe56c15de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe56c0b22a0 0x7fe56c15de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 -- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe56c0c8b00 msgr2=0x7fe56c1f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe56c0c8b00 0x7fe56c1f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 -- 172.21.15.31:0/2744348808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564009b30 con 0x7fe56c15cdf0 2023-10-30T13:35:31.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe571b02640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c15cdf0 0x7fe56c1f5360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe5640af040 tx=0x7fe564019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe561ffb640 1 -- 172.21.15.31:0/2744348808 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5640b8670 con 0x7fe56c15cdf0 2023-10-30T13:35:31.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe561ffb640 1 -- 172.21.15.31:0/2744348808 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5640b87f0 con 0x7fe56c15cdf0 2023-10-30T13:35:31.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe561ffb640 1 -- 172.21.15.31:0/2744348808 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5640b8ba0 con 0x7fe56c15cdf0 2023-10-30T13:35:31.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c1f5e10 con 0x7fe56c15cdf0 2023-10-30T13:35:31.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.240+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c1f20b0 con 0x7fe56c15cdf0 2023-10-30T13:35:31.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe561ffb640 1 -- 172.21.15.31:0/2744348808 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5640b8d00 con 0x7fe56c15cdf0 2023-10-30T13:35:31.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe561ffb640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5500664a0 0x7fe550068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe561ffb640 1 -- 172.21.15.31:0/2744348808 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe56411f3d0 con 0x7fe56c15cdf0 2023-10-30T13:35:31.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe571301640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5500664a0 0x7fe550068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 Processor -- start 2023-10-30T13:35:31.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 -- start start 2023-10-30T13:35:31.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe571301640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5500664a0 0x7fe550068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe568009b60 tx=0x7fe568009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab6810 0x5599f5ab6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7a50 0x5599f5ac1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f586acc0 con 0x5599f5ab7130 2023-10-30T13:35:31.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f58cd920 con 0x5599f5ab7a50 2023-10-30T13:35:31.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599f58cae50 con 0x5599f5ab6810 2023-10-30T13:35:31.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.244+0000 7fe570b00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36334/0 (socket says 172.21.15.31:36334) 2023-10-30T13:35:31.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 -- 172.21.15.31:0/4199169984 learned_addr learned my addr 172.21.15.31:0/4199169984 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe571301640 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab6810 0x5599f5ab6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe571b02640 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7a50 0x5599f5ac1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 -- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab6810 msgr2=0x5599f5ab6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab6810 0x5599f5ab6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 -- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7a50 msgr2=0x5599f5ac1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7a50 0x5599f5ac1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 -- 172.21.15.31:0/4199169984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599f55bf890 con 0x5599f5ab7130 2023-10-30T13:35:31.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe571b02640 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7a50 0x5599f5ac1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:31.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe570b00640 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab7510 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe55c001530 tx=0x7fe55c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d426d28b31a8cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe54effd640 1 -- 172.21.15.31:0/4199169984 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c0052e0 con 0x5599f5ab7130 2023-10-30T13:35:31.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe54effd640 1 -- 172.21.15.31:0/4199169984 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c005440 con 0x5599f5ab7130 2023-10-30T13:35:31.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe54effd640 1 -- 172.21.15.31:0/4199169984 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c003840 con 0x5599f5ab7130 2023-10-30T13:35:31.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- 172.21.15.31:0/4199169984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 msgr2=0x5599f5ab7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab7510 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe55c001530 tx=0x7fe55c001930 comp rx=0 tx=0).stop 2023-10-30T13:35:31.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- 172.21.15.31:0/4199169984 shutdown_connections 2023-10-30T13:35:31.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7a50 0x5599f5ac1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab7510 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 --2- 172.21.15.31:0/4199169984 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab6810 0x5599f5ab6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- 172.21.15.31:0/4199169984 >> 172.21.15.31:0/4199169984 conn(0x5599f5a6cfc0 msgr2=0x5599f5ab45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- 172.21.15.31:0/4199169984 shutdown_connections 2023-10-30T13:35:31.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- 172.21.15.31:0/4199169984 wait complete. 2023-10-30T13:35:31.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- start start 2023-10-30T13:35:31.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 Processor -- start 2023-10-30T13:35:31.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5ab0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5ab15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f58c6740 con 0x5599f5ab7130 2023-10-30T13:35:31.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f5885d10 con 0x5599f5ab6810 2023-10-30T13:35:31.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599f58859b0 con 0x5599f5ab7a50 2023-10-30T13:35:31.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.248+0000 7fe5768f3000 10 client.0 _ll_register_callbacks cb 0x5599f5a61390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:35:31.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36340/0 (socket says 172.21.15.31:36340) 2023-10-30T13:35:31.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 -- 172.21.15.31:0/811647219 learned_addr learned my addr 172.21.15.31:0/811647219 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe571301640 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5ab0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe571b02640 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5ab15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 -- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 msgr2=0x5599f5ab15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5ab15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 -- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 msgr2=0x5599f5ab0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5ab0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 -- 172.21.15.31:0/811647219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55c00f040 con 0x5599f5ab7130 2023-10-30T13:35:31.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe570b00640 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab10a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x5599f58c4060 tx=0x7fe55c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1c9f3acfc50dd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe547fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-10-30T13:35:31.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe547fff640 1 -- 172.21.15.31:0/811647219 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c007410 con 0x5599f5ab7130 2023-10-30T13:35:31.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe547fff640 1 -- 172.21.15.31:0/811647219 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe55c007570 con 0x5599f5ab7130 2023-10-30T13:35:31.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599f587a370 con 0x5599f5ab7130 2023-10-30T13:35:31.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe547fff640 1 -- 172.21.15.31:0/811647219 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe55c0067e0 con 0x5599f5ab7130 2023-10-30T13:35:31.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 10 client.4683 fetch_fsmap learned FSMap version 13 2023-10-30T13:35:31.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x5599f5888460 con 0x5599f5ab7130 2023-10-30T13:35:31.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe547fff640 1 -- 172.21.15.31:0/811647219 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5599f587a370 con 0x5599f5ab7130 2023-10-30T13:35:31.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe547fff640 1 -- 172.21.15.31:0/811647219 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fe55c006940 con 0x5599f5ab7130 2023-10-30T13:35:31.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 10 client.4683 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:35:31.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 10 client.4683 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-30T13:35:31.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 20 client.4683 populate_metadata read hostname 'smithi031' 2023-10-30T13:35:31.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5aca050 0x5599f5acc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 4 client.4683 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:35:31.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 --> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599f58602d0 con 0x5599f5aca050 2023-10-30T13:35:31.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe571301640 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5aca050 0x5599f5acc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe571301640 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5aca050 0x5599f5acc440 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8593ac432e033d server_cookie=3c11c967f73530fa in_seq=0 out_seq=0 2023-10-30T13:35:31.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.252+0000 7fe547fff640 10 client.4683 ms_handle_connect on v2:172.21.15.33:6838/2419808617 2023-10-30T13:35:31.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe547fff640 1 -- 172.21.15.31:0/811647219 <== mds.0 v2:172.21.15.33:6838/2419808617 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5599f58602d0 con 0x5599f5aca050 2023-10-30T13:35:31.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe547fff640 10 client.4683 handle_command_reply: tid=0 2023-10-30T13:35:31.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 client.4683 shutdown 2023-10-30T13:35:31.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 msgr2=0x5599f5ab10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab10a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x5599f58c4060 tx=0x7fe55c000bd0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 shutdown_connections 2023-10-30T13:35:31.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5aca050 0x5599f5acc440 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5ab15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5ab10a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 --2- 172.21.15.31:0/811647219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5ab0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe547fff640 0 client.4683 ms_handle_reset on v2:172.21.15.33:6838/2419808617 2023-10-30T13:35:31.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 >> 172.21.15.31:0/811647219 conn(0x5599f5a6cfc0 msgr2=0x5599f5ab6090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 shutdown_connections 2023-10-30T13:35:31.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 1 -- 172.21.15.31:0/811647219 wait complete. 2023-10-30T13:35:31.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.256+0000 7fe5768f3000 20 client.4683 trim_cache size 0 max 16384 2023-10-30T13:35:31.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 Processor -- start 2023-10-30T13:35:31.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- start start 2023-10-30T13:35:31.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab6810 0x5599f5a72b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7130 0x5599f5a7b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5a7ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f5580940 con 0x5599f5ab6810 2023-10-30T13:35:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f586e4a0 con 0x5599f5ab7130 2023-10-30T13:35:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599f58c3b40 con 0x5599f5ab7a50 2023-10-30T13:35:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab6810 0x5599f5a72b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab6810 0x5599f5a72b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36348/0 (socket says 172.21.15.31:36348) 2023-10-30T13:35:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 -- 172.21.15.31:0/3576832384 learned_addr learned my addr 172.21.15.31:0/3576832384 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571b02640 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5a7ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe570b00640 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7130 0x5599f5a7b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 -- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 msgr2=0x5599f5a7ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5a7ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 -- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7130 msgr2=0x5599f5a7b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7130 0x5599f5a7b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 -- 172.21.15.31:0/3576832384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599f5ac8d40 con 0x5599f5ab6810 2023-10-30T13:35:31.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe571301640 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab6810 0x5599f5a72b80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5599f587cb40 tx=0x7fe568018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538a762f8081d485 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe54f7fe640 1 -- 172.21.15.31:0/3576832384 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56801c040 con 0x5599f5ab6810 2023-10-30T13:35:31.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe54f7fe640 1 -- 172.21.15.31:0/3576832384 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5680043d0 con 0x5599f5ab6810 2023-10-30T13:35:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe54f7fe640 1 -- 172.21.15.31:0/3576832384 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56802aea0 con 0x5599f5ab6810 2023-10-30T13:35:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- 172.21.15.31:0/3576832384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab6810 msgr2=0x5599f5a72b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab6810 0x5599f5a72b80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5599f587cb40 tx=0x7fe568018000 comp rx=0 tx=0).stop 2023-10-30T13:35:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- 172.21.15.31:0/3576832384 shutdown_connections 2023-10-30T13:35:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5a7ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab7130 0x5599f5a7b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 --2- 172.21.15.31:0/3576832384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab6810 0x5599f5a72b80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- 172.21.15.31:0/3576832384 >> 172.21.15.31:0/3576832384 conn(0x5599f5a6cfc0 msgr2=0x5599f5a7a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- 172.21.15.31:0/3576832384 shutdown_connections 2023-10-30T13:35:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.272+0000 7fe5768f3000 1 -- 172.21.15.31:0/3576832384 wait complete. 2023-10-30T13:35:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 -- start start 2023-10-30T13:35:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 Processor -- start 2023-10-30T13:35:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5abb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5b17e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5b18390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f58746c0 con 0x5599f5ab7130 2023-10-30T13:35:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f5907ba0 con 0x5599f5ab6810 2023-10-30T13:35:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599f58854d0 con 0x5599f5ab7a50 2023-10-30T13:35:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 10 client.0 _ll_register_callbacks cb 0x7fe55c007e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:35:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5b17e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5b17e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36350/0 (socket says 172.21.15.31:36350) 2023-10-30T13:35:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 -- 172.21.15.31:0/942183044 learned_addr learned my addr 172.21.15.31:0/942183044 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe571301640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5abb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe571b02640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5b18390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 -- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 msgr2=0x5599f5b18390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5b18390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 -- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 msgr2=0x5599f5abb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5abb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 -- 172.21.15.31:0/942183044 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe568004040 con 0x5599f5ab7130 2023-10-30T13:35:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe571b02640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5b18390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe571301640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5abb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe570b00640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5b17e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe55c000c10 tx=0x7fe55c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee8282d3bad5432 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe547fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-10-30T13:35:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe547fff640 1 -- 172.21.15.31:0/942183044 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c006fe0 con 0x5599f5ab7130 2023-10-30T13:35:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599f5ac8740 con 0x5599f5ab7130 2023-10-30T13:35:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe547fff640 1 -- 172.21.15.31:0/942183044 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe55c003600 con 0x5599f5ab7130 2023-10-30T13:35:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe547fff640 1 -- 172.21.15.31:0/942183044 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe55c006d60 con 0x5599f5ab7130 2023-10-30T13:35:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 10 client.4689 fetch_fsmap learned FSMap version 13 2023-10-30T13:35:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x5599f586e760 con 0x5599f5ab7130 2023-10-30T13:35:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe547fff640 1 -- 172.21.15.31:0/942183044 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5599f5ac8740 con 0x5599f5ab7130 2023-10-30T13:35:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe547fff640 1 -- 172.21.15.31:0/942183044 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fe55c0037c0 con 0x5599f5ab7130 2023-10-30T13:35:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 10 client.4689 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:35:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 10 client.4689 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-30T13:35:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 20 client.4689 populate_metadata read hostname 'smithi031' 2023-10-30T13:35:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5b188d0 0x5599f5b16c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 4 client.4689 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:35:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.276+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 --> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5599f585e6f0 con 0x5599f5b188d0 2023-10-30T13:35:31.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe571301640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5b188d0 0x5599f5b16c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe571301640 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5b188d0 0x5599f5b16c30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c8e92ef968dcd9 server_cookie=25d8cf9ae2e8bc47 in_seq=0 out_seq=0 2023-10-30T13:35:31.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe547fff640 10 client.4689 ms_handle_connect on v2:172.21.15.33:6838/2419808617 2023-10-30T13:35:31.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe547fff640 1 -- 172.21.15.31:0/942183044 <== mds.0 v2:172.21.15.33:6838/2419808617 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5599f585e6f0 con 0x5599f5b188d0 2023-10-30T13:35:31.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe547fff640 10 client.4689 handle_command_reply: tid=0 2023-10-30T13:35:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 client.4689 shutdown 2023-10-30T13:35:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 msgr2=0x5599f5b17e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5b17e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe55c000c10 tx=0x7fe55c009040 comp rx=0 tx=0).stop 2023-10-30T13:35:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 shutdown_connections 2023-10-30T13:35:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:6838/2419808617,v1:172.21.15.33:6839/2419808617] conn(0x5599f5b188d0 0x5599f5b16c30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x5599f5ab7a50 0x5599f5b18390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599f5ab7130 0x5599f5b17e50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 --2- 172.21.15.31:0/942183044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599f5ab6810 0x5599f5abb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe547fff640 0 client.4689 ms_handle_reset on v2:172.21.15.33:6838/2419808617 2023-10-30T13:35:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 >> 172.21.15.31:0/942183044 conn(0x5599f5a6cfc0 msgr2=0x5599f5a7a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 shutdown_connections 2023-10-30T13:35:31.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 1 -- 172.21.15.31:0/942183044 wait complete. 2023-10-30T13:35:31.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe5768f3000 20 client.4689 trim_cache size 0 max 16384 2023-10-30T13:35:31.297 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:35:31.297 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "62b651f4-6cfc-4dcb-91a0-e9335b7700dd", 2023-10-30T13:35:31.297 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2023-10-30T13:35:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4282, 2023-10-30T13:35:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-10-30T13:35:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-10-30T13:35:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-10-30T13:35:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 23.009305261000002, 2023-10-30T13:35:31.299 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 13, 2023-10-30T13:35:31.299 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 21, 2023-10-30T13:35:31.299 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 21, 2023-10-30T13:35:31.299 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 30.759791269000001 2023-10-30T13:35:31.299 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:35:31.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5500664a0 msgr2=0x7fe550068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5500664a0 0x7fe550068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe568009b60 tx=0x7fe568009190 comp rx=0 tx=0).stop 2023-10-30T13:35:31.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c15cdf0 msgr2=0x7fe56c1f5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c15cdf0 0x7fe56c1f5360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe5640af040 tx=0x7fe564019740 comp rx=0 tx=0).stop 2023-10-30T13:35:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 shutdown_connections 2023-10-30T13:35:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5500664a0 0x7fe550068960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe56c15cdf0 0x7fe56c1f5360 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe56c0c8b00 0x7fe56c1f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 --2- 172.21.15.31:0/2744348808 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe56c0b22a0 0x7fe56c15de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 >> 172.21.15.31:0/2744348808 conn(0x7fe56c06bed0 msgr2=0x7fe56c0c6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 shutdown_connections 2023-10-30T13:35:31.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.280+0000 7fe572303640 1 -- 172.21.15.31:0/2744348808 wait complete. 2023-10-30T13:35:31.304 INFO:teuthology.run_tasks:Running task kclient... 2023-10-30T13:35:31.325 INFO:tasks.kclient:Mounting kernel clients... 2023-10-30T13:35:31.326 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-30T13:35:31.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:31.327 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2023-10-30T13:35:31.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:31.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2023-10-30T13:35:31.402 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2023-10-30T13:35:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:35:31.404 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-30T13:35:31.404 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-30T13:35:31.405 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:31.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-30T13:35:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.584+0000 7f3dfc691640 1 Processor -- start 2023-10-30T13:35:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.584+0000 7f3dfc691640 1 -- start start 2023-10-30T13:35:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.584+0000 7f3dfc691640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3df4153c80 0x7f3df415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.584+0000 7f3dfc691640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c42a0 0x7f3df40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.584+0000 7f3dfc691640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df4159f20 con 0x7f3df40c4a10 2023-10-30T13:35:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.584+0000 7f3dfc691640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df415dec0 con 0x7f3df4153c80 2023-10-30T13:35:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.584+0000 7f3dfc691640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df41599a0 con 0x7f3df40c42a0 2023-10-30T13:35:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3df9c2c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f3df40c4a10 0x7f3df415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55082/0 (socket says 172.21.15.31:55082) 2023-10-30T13:35:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3df9c2c640 1 -- 172.21.15.31:0/3096950826 learned_addr learned my addr 172.21.15.31:0/3096950826 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3dfac2e640 1 --2- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c42a0 0x7f3df40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3dfa42d640 1 --2- 172.21.15.31:0/3096950826 >> v2:172.21.15.33:3300/0 conn(0x7f3df4153c80 0x7f3df415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3df942b640 1 -- 172.21.15.31:0/3096950826 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773814560 0 0) 0x7f3df4159f20 con 0x7f3df40c4a10 2023-10-30T13:35:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3df942b640 1 -- 172.21.15.31:0/3096950826 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de0003530 con 0x7f3df40c4a10 2023-10-30T13:35:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3df942b640 1 -- 172.21.15.31:0/3096950826 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861683207 0 0) 0x7f3de0003530 con 0x7f3df40c4a10 2023-10-30T13:35:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3dfa42d640 1 -- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c42a0 msgr2=0x7f3df40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3dfa42d640 1 --2- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c42a0 0x7f3df40ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3dfa42d640 1 -- 172.21.15.31:0/3096950826 >> v1:172.21.15.31:6789/0 conn(0x7f3df40c4a10 legacy=0x7f3df415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3dfa42d640 1 -- 172.21.15.31:0/3096950826 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df415d620 con 0x7f3df4153c80 2023-10-30T13:35:31.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.588+0000 7f3dfa42d640 1 --2- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df4153c80 0x7f3df415cf90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3df00097c0 tx=0x7f3df001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5d6b67c4f74eb24 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3df942b640 1 -- 172.21.15.31:0/3096950826 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de4001680 con 0x7f3df4153c80 2023-10-30T13:35:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3df942b640 1 -- 172.21.15.31:0/3096950826 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df00a85d0 con 0x7f3df4153c80 2023-10-30T13:35:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3df942b640 1 -- 172.21.15.31:0/3096950826 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df00a8960 con 0x7f3df4153c80 2023-10-30T13:35:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df4153c80 msgr2=0x7f3df415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 --2- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df4153c80 0x7f3df415cf90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3df00097c0 tx=0x7f3df001eb90 comp rx=0 tx=0).stop 2023-10-30T13:35:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- 172.21.15.31:0/3096950826 shutdown_connections 2023-10-30T13:35:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 --2- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c42a0 0x7f3df40ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 --2- 172.21.15.31:0/3096950826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df4153c80 0x7f3df415cf90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- 172.21.15.31:0/3096950826 >> 172.21.15.31:0/3096950826 conn(0x7f3df40715c0 msgr2=0x7f3df40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- 172.21.15.31:0/3096950826 shutdown_connections 2023-10-30T13:35:31.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- 172.21.15.31:0/3096950826 wait complete. 2023-10-30T13:35:31.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 Processor -- start 2023-10-30T13:35:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- start start 2023-10-30T13:35:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df40c42a0 0x7f3df41e3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c4a10 0x7f3df4149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df4153c80 0x7f3df414a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df4151880 con 0x7f3df4153c80 2023-10-30T13:35:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df41525d0 con 0x7f3df40c42a0 2023-10-30T13:35:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfc691640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df414fa40 con 0x7f3df40c4a10 2023-10-30T13:35:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfac2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df4153c80 0x7f3df414a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfac2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df4153c80 0x7f3df414a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36366/0 (socket says 172.21.15.31:36366) 2023-10-30T13:35:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3dfac2e640 1 -- 172.21.15.31:0/361362127 learned_addr learned my addr 172.21.15.31:0/361362127 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.592+0000 7f3df9c2c640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c4a10 0x7f3df4149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfa42d640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df40c42a0 0x7f3df41e3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfac2e640 1 -- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c4a10 msgr2=0x7f3df4149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfac2e640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c4a10 0x7f3df4149b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfac2e640 1 -- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df40c42a0 msgr2=0x7f3df41e3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfac2e640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df40c42a0 0x7f3df41e3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfac2e640 1 -- 172.21.15.31:0/361362127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0009420 con 0x7f3df4153c80 2023-10-30T13:35:31.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3df9c2c640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c4a10 0x7f3df4149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:31.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfac2e640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df4153c80 0x7f3df414a0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00efc0 tx=0x7f3dec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3deaffd640 1 -- 172.21.15.31:0/361362127 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec014480 con 0x7f3df4153c80 2023-10-30T13:35:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3deaffd640 1 -- 172.21.15.31:0/361362127 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec00f040 con 0x7f3df4153c80 2023-10-30T13:35:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3deaffd640 1 -- 172.21.15.31:0/361362127 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec014820 con 0x7f3df4153c80 2023-10-30T13:35:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df4151af0 con 0x7f3df4153c80 2023-10-30T13:35:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df41520b0 con 0x7f3df4153c80 2023-10-30T13:35:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df404e170 con 0x7f3df4153c80 2023-10-30T13:35:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3deaffd640 1 -- 172.21.15.31:0/361362127 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3dec014980 con 0x7f3df4153c80 2023-10-30T13:35:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3deaffd640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3dd80664a0 0x7f3dd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.596+0000 7f3deaffd640 1 -- 172.21.15.31:0/361362127 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dec084a00 con 0x7f3df4153c80 2023-10-30T13:35:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.600+0000 7f3dfa42d640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3dd80664a0 0x7f3dd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.600+0000 7f3dfa42d640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3dd80664a0 0x7f3dd8068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3df00b2000 tx=0x7f3df0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.600+0000 7f3deaffd640 1 -- 172.21.15.31:0/361362127 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3dec051de0 con 0x7f3df4153c80 2023-10-30T13:35:31.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.796+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3df4073d50 con 0x7f3df4153c80 2023-10-30T13:35:31.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.796+0000 7f3deaffd640 1 -- 172.21.15.31:0/361362127 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3dec018070 con 0x7f3df4153c80 2023-10-30T13:35:31.802 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-30T13:35:31.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3dd80664a0 msgr2=0x7f3dd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3dd80664a0 0x7f3dd8068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3df00b2000 tx=0x7f3df0008040 comp rx=0 tx=0).stop 2023-10-30T13:35:31.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df4153c80 msgr2=0x7f3df414a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df4153c80 0x7f3df414a0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00efc0 tx=0x7f3dec00c490 comp rx=0 tx=0).stop 2023-10-30T13:35:31.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 shutdown_connections 2023-10-30T13:35:31.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3dd80664a0 0x7f3dd8068960 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df4153c80 0x7f3df414a0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3df40c4a10 0x7f3df4149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 --2- 172.21.15.31:0/361362127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3df40c42a0 0x7f3df41e3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 >> 172.21.15.31:0/361362127 conn(0x7f3df40715c0 msgr2=0x7f3df40c5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 shutdown_connections 2023-10-30T13:35:31.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:31.800+0000 7f3dfc691640 1 -- 172.21.15.31:0/361362127 wait complete. 2023-10-30T13:35:31.823 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-30T13:35:31.823 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-30T13:35:31.823 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2023-10-30T13:35:31.824 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-30T13:35:31.824 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:31.824 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-30T13:35:31.824 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-30T13:35:31.825 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-30T13:35:31.825 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-30T13:35:31.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:31.825 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2023-10-30T13:35:31.835 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-30T13:35:31.836 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-30T13:35:31.836 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2023-10-30T13:35:31.836 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:35:31.836 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2023-10-30T13:35:31.837 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:35:31.837 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:35:31.837 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2023-10-30T13:35:31.837 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2023-10-30T13:35:31.837 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:35:31.837 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2023-10-30T13:35:31.838 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2023-10-30T13:35:31.838 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-30T13:35:31.838 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2023-10-30T13:35:31.838 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2023-10-30T13:35:31.838 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-10-30T13:35:31.839 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42392sec preferred_lft 42392sec 2023-10-30T13:35:31.839 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2023-10-30T13:35:31.839 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:35:31.839 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:35:31.839 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2023-10-30T13:35:31.840 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2023-10-30T13:35:31.840 INFO:teuthology.orchestra.run.smithi031.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-30T13:35:31.840 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 02:42:bc:01:8c:1d brd ff:ff:ff:ff:ff:ff 2023-10-30T13:35:31.840 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-30T13:35:31.840 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:35:31.841 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-30T13:35:31.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:31.842 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-10-30T13:35:31.842 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-10-30T13:35:31.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2023-10-30T13:35:31.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2023-10-30T13:35:31.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2023-10-30T13:35:31.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-30T13:35:31.844 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-10-30T13:35:31.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:31.957 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-30T13:35:31.978 INFO:teuthology.orchestra.run.smithi031.stdout:1 2023-10-30T13:35:31.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:31.980 DEBUG:teuthology.orchestra.run.smithi031:> route 2023-10-30T13:35:32.052 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2023-10-30T13:35:32.053 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-30T13:35:32.053 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-10-30T13:35:32.053 INFO:teuthology.orchestra.run.smithi031.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-30T13:35:32.053 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-10-30T13:35:32.053 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-30T13:35:32.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.054 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-10-30T13:35:32.054 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-10-30T13:35:32.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-30T13:35:32.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-30T13:35:32.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-30T13:35:32.055 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-10-30T13:35:32.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.150 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2023-10-30T13:35:32.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.160 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2023-10-30T13:35:32.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.212 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-10-30T13:35:32.212 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-10-30T13:35:32.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T13:35:32.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-30T13:35:32.213 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-10-30T13:35:32.294 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-30T13:35:32.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.295 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-10-30T13:35:32.295 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-10-30T13:35:32.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-30T13:35:32.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-30T13:35:32.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-30T13:35:32.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-30T13:35:32.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-30T13:35:32.297 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-10-30T13:35:32.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.461 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-10-30T13:35:32.462 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-10-30T13:35:32.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2023-10-30T13:35:32.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2023-10-30T13:35:32.463 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-10-30T13:35:32.505 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T13:35:32.506 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:32.555 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-30T13:35:32.555 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T13:35:32.555 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:32.606 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-30T13:35:32.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:35:32.712 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:35:32.712 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:35:32.712 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2023-10-30T13:35:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:35:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:35:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2023-10-30T13:35:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.33:3300,172.21.15.33:3301" 2023-10-30T13:35:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T13:35:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:35:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:35:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2023-10-30T13:35:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.33:3300,172.21.15.33:3301" 2023-10-30T13:35:32.827 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T13:35:32.827 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:6789,172.21.15.33:6789,172.21.15.33:6790" 2023-10-30T13:35:32.827 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.31:6789,172.21.15.33:6789,172.21.15.33:6790:/ 2023-10-30T13:35:32.827 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=62b651f4-6cfc-4dcb-91a0-e9335b7700dd" will pass to kernel 2023-10-30T13:35:32.830 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-30T13:35:32.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:32.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.860 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:35:32.946 INFO:teuthology.orchestra.run.smithi031.stdout:client4698 2023-10-30T13:35:32.946 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:35:32.946 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:35:32.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:32.999 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:35:33.072 INFO:teuthology.orchestra.run.smithi031.stdout:client4698 2023-10-30T13:35:33.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:33.073 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:35:33.149 INFO:teuthology.orchestra.run.smithi031.stdout:client4698 2023-10-30T13:35:33.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:35:33.150 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/62b651f4-6cfc-4dcb-91a0-e9335b7700dd.client4698/status') 2023-10-30T13:35:33.223 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.4698 (3)192.168.0.1:0/290785035 2023-10-30T13:35:33.223 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2023-10-30T13:35:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2023-10-30T13:35:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2023-10-30T13:35:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:68 bytes copied, 5.3595e-05 s, 1.3 MB/s 2023-10-30T13:35:33.225 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-30T13:35:33.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:33.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350fb65640 1 Processor -- start 2023-10-30T13:35:33.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350fb65640 1 -- start start 2023-10-30T13:35:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350fb65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 0x7f350815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350fb65640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f35080c42a0 0x7f35080ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350fb65640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3508159f20 con 0x7f3508153c80 2023-10-30T13:35:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350fb65640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350815dec0 con 0x7f35080c42a0 2023-10-30T13:35:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350fb65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35081599a0 con 0x7f35080c4a10 2023-10-30T13:35:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350d901640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f3508153c80 0x7f350815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55094/0 (socket says 172.21.15.31:55094) 2023-10-30T13:35:33.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350d901640 1 -- 172.21.15.31:0/3765891180 learned_addr learned my addr 172.21.15.31:0/3765891180 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:33.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350e102640 1 --2- 172.21.15.31:0/3765891180 >> v2:172.21.15.33:3300/0 conn(0x7f35080c42a0 0x7f35080ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350d100640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 0x7f350815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811310572 0 0) 0x7f3508159f20 con 0x7f3508153c80 2023-10-30T13:35:33.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4003530 con 0x7f3508153c80 2023-10-30T13:35:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3175665398 0 0) 0x7f34f4003530 con 0x7f3508153c80 2023-10-30T13:35:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 msgr2=0x7f350815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 0x7f350815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 >> v2:172.21.15.33:3300/0 conn(0x7f35080c42a0 msgr2=0x7f35080ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 --2- 172.21.15.31:0/3765891180 >> v2:172.21.15.33:3300/0 conn(0x7f35080c42a0 0x7f35080ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350815d620 con 0x7f3508153c80 2023-10-30T13:35:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f3504001680 con 0x7f3508153c80 2023-10-30T13:35:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 >> v1:172.21.15.31:6789/0 conn(0x7f3508153c80 legacy=0x7f350815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 0x7f34f4003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34f4003f00 0x7f34f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4004d10 0x7f34f4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350815dec0 con 0x7f34f4003f00 2023-10-30T13:35:33.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35081599a0 con 0x7f34f4004d10 2023-10-30T13:35:33.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.408+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f400b9c0 con 0x7f35080c4a10 2023-10-30T13:35:33.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d100640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34f4003f00 0x7f34f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d901640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 0x7f34f4003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350e102640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4004d10 0x7f34f4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d100640 1 -- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 msgr2=0x7f34f4003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:33.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d100640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 0x7f34f4003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d100640 1 -- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4004d10 msgr2=0x7f34f4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d100640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4004d10 0x7f34f4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d100640 1 -- 172.21.15.31:0/3765891180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f4003530 con 0x7f34f4003f00 2023-10-30T13:35:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d901640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c4a10 0x7f34f4003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350d100640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34f4003f00 0x7f34f40047d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009ab0 tx=0x7f34f801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d234d25ecaeed492 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80a5620 con 0x7f34f4003f00 2023-10-30T13:35:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f80a5780 con 0x7f34f4003f00 2023-10-30T13:35:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350c8ff640 1 -- 172.21.15.31:0/3765891180 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80a5b30 con 0x7f34f4003f00 2023-10-30T13:35:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350fb65640 1 -- 172.21.15.31:0/3765891180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34f4003f00 msgr2=0x7f34f40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.412+0000 7f350fb65640 1 --2- 172.21.15.31:0/3765891180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34f4003f00 0x7f34f40047d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009ab0 tx=0x7f34f801ca30 comp rx=0 tx=0).stop 2023-10-30T13:35:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350e102640 1 -- 172.21.15.31:0/3765891180 reap_dead start 2023-10-30T13:35:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- 172.21.15.31:0/3765891180 shutdown_connections 2023-10-30T13:35:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- 172.21.15.31:0/3765891180 >> 172.21.15.31:0/3765891180 conn(0x7f35080715c0 msgr2=0x7f35080719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- 172.21.15.31:0/3765891180 shutdown_connections 2023-10-30T13:35:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- 172.21.15.31:0/3765891180 wait complete. 2023-10-30T13:35:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 Processor -- start 2023-10-30T13:35:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- start start 2023-10-30T13:35:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4003f00 0x7f350815b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f350814fc90 0x7f350815c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c74e0 0x7f35081450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35080ce140 con 0x7f350814fc90 2023-10-30T13:35:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350815d000 con 0x7f34f4003f00 2023-10-30T13:35:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350fb65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350815b350 con 0x7f35080c74e0 2023-10-30T13:35:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f350814fc90 0x7f350815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f350814fc90 0x7f350815c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36382/0 (socket says 172.21.15.31:36382) 2023-10-30T13:35:33.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 -- 172.21.15.31:0/358309091 learned_addr learned my addr 172.21.15.31:0/358309091 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:33.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350e102640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c74e0 0x7f35081450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d901640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4003f00 0x7f350815b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 -- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c74e0 msgr2=0x7f35081450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c74e0 0x7f35081450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 -- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4003f00 msgr2=0x7f350815b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4003f00 0x7f350815b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 -- 172.21.15.31:0/358309091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35080cdb80 con 0x7f350814fc90 2023-10-30T13:35:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.416+0000 7f350d100640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f350814fc90 0x7f350815c590 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009ab0 tx=0x7f34f4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f34fe7fc640 1 -- 172.21.15.31:0/358309091 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80bc070 con 0x7f350814fc90 2023-10-30T13:35:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f34fe7fc640 1 -- 172.21.15.31:0/358309091 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f8004020 con 0x7f350814fc90 2023-10-30T13:35:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f34fe7fc640 1 -- 172.21.15.31:0/358309091 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8004390 con 0x7f350814fc90 2023-10-30T13:35:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35080cde60 con 0x7f350814fc90 2023-10-30T13:35:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350815afe0 con 0x7f350814fc90 2023-10-30T13:35:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350815ab20 con 0x7f350814fc90 2023-10-30T13:35:33.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f34fe7fc640 1 -- 172.21.15.31:0/358309091 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34f80b4040 con 0x7f350814fc90 2023-10-30T13:35:33.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f34fe7fc640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34ec0664d0 0x7f34ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f34fe7fc640 1 -- 172.21.15.31:0/358309091 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34f811df50 con 0x7f350814fc90 2023-10-30T13:35:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.420+0000 7f350d901640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34ec0664d0 0x7f34ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.424+0000 7f350d901640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34ec0664d0 0x7f34ec068990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3504005ae0 tx=0x7f3504009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.424+0000 7f34fe7fc640 1 -- 172.21.15.31:0/358309091 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34f80eb450 con 0x7f350814fc90 2023-10-30T13:35:33.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.584+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f350804e170 con 0x7f350814fc90 2023-10-30T13:35:33.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.584+0000 7f34fe7fc640 1 -- 172.21.15.31:0/358309091 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f34f80eea70 con 0x7f350814fc90 2023-10-30T13:35:33.589 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:33.589 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34ec0664d0 msgr2=0x7f34ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34ec0664d0 0x7f34ec068990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3504005ae0 tx=0x7f3504009040 comp rx=0 tx=0).stop 2023-10-30T13:35:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f350814fc90 msgr2=0x7f350815c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f350814fc90 0x7f350815c590 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009ab0 tx=0x7f34f4005800 comp rx=0 tx=0).stop 2023-10-30T13:35:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 shutdown_connections 2023-10-30T13:35:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34ec0664d0 0x7f34ec068990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35080c74e0 0x7f35081450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f350814fc90 0x7f350815c590 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 --2- 172.21.15.31:0/358309091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34f4003f00 0x7f350815b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 >> 172.21.15.31:0/358309091 conn(0x7f35080715c0 msgr2=0x7f35080c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:33.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 shutdown_connections 2023-10-30T13:35:33.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.588+0000 7f350fb65640 1 -- 172.21.15.31:0/358309091 wait complete. 2023-10-30T13:35:33.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:35:05.304460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:35:06.652927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:35:33.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:35:33.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.752+0000 7ffbd8d01640 1 Processor -- start 2023-10-30T13:35:33.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.752+0000 7ffbd8d01640 1 -- start start 2023-10-30T13:35:33.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.752+0000 7ffbd8d01640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ffbd4153ba0 0x7ffbd40cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.752+0000 7ffbd8d01640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffbd40c4810 0x7ffbd415a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd8d01640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd4153f80 con 0x7ffbd40c4410 2023-10-30T13:35:33.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd8d01640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd4159df0 con 0x7ffbd40c4810 2023-10-30T13:35:33.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd8d01640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd415a7c0 con 0x7ffbd4153ba0 2023-10-30T13:35:33.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffbd40c4810 0x7ffbd415a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffbd40c4810 0x7ffbd415a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36388/0 (socket says 172.21.15.31:36388) 2023-10-30T13:35:33.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/3092033541 learned_addr learned my addr 172.21.15.31:0/3092033541 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:33.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd259c640 1 --2- 172.21.15.31:0/3092033541 >> v2:172.21.15.33:3300/0 conn(0x7ffbd4153ba0 0x7ffbd40cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/3092033541 >> v2:172.21.15.33:3300/0 conn(0x7ffbd4153ba0 msgr2=0x7ffbd40cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:33.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 --2- 172.21.15.31:0/3092033541 >> v2:172.21.15.33:3300/0 conn(0x7ffbd4153ba0 0x7ffbd40cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/3092033541 >> v1:172.21.15.33:6789/0 conn(0x7ffbd40c4410 legacy=0x7ffbd415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:33.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/3092033541 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd41597c0 con 0x7ffbd40c4810 2023-10-30T13:35:33.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd2d9d640 1 --2- 172.21.15.31:0/3092033541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd40c4810 0x7ffbd415a280 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40d0170 tx=0x7ffbcc01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bff6059462211ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd1d9b640 1 -- 172.21.15.31:0/3092033541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0a5790 con 0x7ffbd40c4810 2023-10-30T13:35:33.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd1d9b640 1 -- 172.21.15.31:0/3092033541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbcc0a58f0 con 0x7ffbd40c4810 2023-10-30T13:35:33.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.756+0000 7ffbd1d9b640 1 -- 172.21.15.31:0/3092033541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0a5c80 con 0x7ffbd40c4810 2023-10-30T13:35:33.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- 172.21.15.31:0/3092033541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd40c4810 msgr2=0x7ffbd415a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/3092033541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd40c4810 0x7ffbd415a280 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40d0170 tx=0x7ffbcc01c8d0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- 172.21.15.31:0/3092033541 shutdown_connections 2023-10-30T13:35:33.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/3092033541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd40c4810 0x7ffbd415a280 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/3092033541 >> v2:172.21.15.33:3300/0 conn(0x7ffbd4153ba0 0x7ffbd40cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- 172.21.15.31:0/3092033541 >> 172.21.15.31:0/3092033541 conn(0x7ffbd4071790 msgr2=0x7ffbd4071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:33.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- 172.21.15.31:0/3092033541 shutdown_connections 2023-10-30T13:35:33.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- 172.21.15.31:0/3092033541 wait complete. 2023-10-30T13:35:33.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 Processor -- start 2023-10-30T13:35:33.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- start start 2023-10-30T13:35:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbd40c4410 0x7ffbd415be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbd40c4810 0x7ffbd4150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd4153ba0 0x7ffbd4150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd4152540 con 0x7ffbd4153ba0 2023-10-30T13:35:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd41f51f0 con 0x7ffbd40c4410 2023-10-30T13:35:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd8d01640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd41f5d70 con 0x7ffbd40c4810 2023-10-30T13:35:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd4153ba0 0x7ffbd4150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd4153ba0 0x7ffbd4150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36400/0 (socket says 172.21.15.31:36400) 2023-10-30T13:35:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/496348816 learned_addr learned my addr 172.21.15.31:0/496348816 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbd259c640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbd40c4410 0x7ffbd415be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.760+0000 7ffbcbfff640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbd40c4810 0x7ffbd4150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbd40c4810 msgr2=0x7ffbd4150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd2d9d640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbd40c4810 0x7ffbd4150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbd40c4410 msgr2=0x7ffbd415be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd2d9d640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbd40c4410 0x7ffbd415be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd2d9d640 1 -- 172.21.15.31:0/496348816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc009b30 con 0x7ffbd4153ba0 2023-10-30T13:35:33.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd2d9d640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd4153ba0 0x7ffbd4150840 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0af040 tx=0x7ffbcc019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbcaffd640 1 -- 172.21.15.31:0/496348816 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0b8670 con 0x7ffbd4153ba0 2023-10-30T13:35:33.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbcaffd640 1 -- 172.21.15.31:0/496348816 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbcc0b87d0 con 0x7ffbd4153ba0 2023-10-30T13:35:33.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbcaffd640 1 -- 172.21.15.31:0/496348816 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0b8b60 con 0x7ffbd4153ba0 2023-10-30T13:35:33.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd414e030 con 0x7ffbd4153ba0 2023-10-30T13:35:33.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd41f5440 con 0x7ffbd4153ba0 2023-10-30T13:35:33.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd41f5ee0 con 0x7ffbd4153ba0 2023-10-30T13:35:33.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbcaffd640 1 -- 172.21.15.31:0/496348816 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffbcc0b8cc0 con 0x7ffbd4153ba0 2023-10-30T13:35:33.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbcaffd640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbb40664a0 0x7ffbb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbcaffd640 1 -- 172.21.15.31:0/496348816 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbcc11f3f0 con 0x7ffbd4153ba0 2023-10-30T13:35:33.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.764+0000 7ffbd259c640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbb40664a0 0x7ffbb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.768+0000 7ffbd259c640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbb40664a0 0x7ffbb4068960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40079c0 tx=0x7ffbc4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.768+0000 7ffbcaffd640 1 -- 172.21.15.31:0/496348816 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffbcc0eca60 con 0x7ffbd4153ba0 2023-10-30T13:35:33.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.924+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffbd404e170 con 0x7ffbd4153ba0 2023-10-30T13:35:33.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.924+0000 7ffbcaffd640 1 -- 172.21.15.31:0/496348816 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffbcc0f0080 con 0x7ffbd4153ba0 2023-10-30T13:35:33.930 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:35:33.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbb40664a0 msgr2=0x7ffbb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbb40664a0 0x7ffbb4068960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40079c0 tx=0x7ffbc4008040 comp rx=0 tx=0).stop 2023-10-30T13:35:33.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd4153ba0 msgr2=0x7ffbd4150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd4153ba0 0x7ffbd4150840 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0af040 tx=0x7ffbcc019670 comp rx=0 tx=0).stop 2023-10-30T13:35:33.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 shutdown_connections 2023-10-30T13:35:33.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbb40664a0 0x7ffbb4068960 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd4153ba0 0x7ffbd4150840 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbd40c4810 0x7ffbd4150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 --2- 172.21.15.31:0/496348816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbd40c4410 0x7ffbd415be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 >> 172.21.15.31:0/496348816 conn(0x7ffbd4071790 msgr2=0x7ffbd414f690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:33.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 shutdown_connections 2023-10-30T13:35:33.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:33.928+0000 7ffbd8d01640 1 -- 172.21.15.31:0/496348816 wait complete. 2023-10-30T13:35:33.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:35:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.092+0000 7f8f00db5640 1 Processor -- start 2023-10-30T13:35:34.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.092+0000 7f8f00db5640 1 -- start start 2023-10-30T13:35:34.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8f00db5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 0x7f8efc0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8f00db5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc15d0a0 con 0x7f8efc071cb0 2023-10-30T13:35:34.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8f00db5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc15d380 con 0x7f8efc153300 2023-10-30T13:35:34.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8f00db5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0cca10 con 0x7f8efc0718d0 2023-10-30T13:35:34.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efaffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8efc071cb0 0x7f8efc15a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55096/0 (socket says 172.21.15.31:55096) 2023-10-30T13:35:34.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efaffd640 1 -- 172.21.15.31:0/1776166780 learned_addr learned my addr 172.21.15.31:0/1776166780 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:34.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efb7fe640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 0x7f8efc0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069793251 0 0) 0x7f8efc15d0a0 con 0x7f8efc071cb0 2023-10-30T13:35:34.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee4003530 con 0x7f8efc071cb0 2023-10-30T13:35:34.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054227160 0 0) 0x7f8efc15d380 con 0x7f8efc153300 2023-10-30T13:35:34.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc15d0a0 con 0x7f8efc153300 2023-10-30T13:35:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 758580702 0 0) 0x7f8ee4003530 con 0x7f8efc071cb0 2023-10-30T13:35:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 msgr2=0x7f8efc0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 0x7f8efc0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 >> v1:172.21.15.33:6789/0 conn(0x7f8efc153300 legacy=0x7f8efc0cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc15d660 con 0x7f8efc071cb0 2023-10-30T13:35:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f8ef0001680 con 0x7f8efc071cb0 2023-10-30T13:35:34.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 >> v1:172.21.15.31:6789/0 conn(0x7f8efc071cb0 legacy=0x7f8efc15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 0x7f8ee4003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee40040a0 0x7f8ee40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ee4004d10 0x7f8ee4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0cca10 con 0x7f8ee4004d10 2023-10-30T13:35:34.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee400ba70 con 0x7f8ee40040a0 2023-10-30T13:35:34.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee400bc30 con 0x7f8efc0718d0 2023-10-30T13:35:34.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efbfff640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ee4004d10 0x7f8ee4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efb7fe640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 0x7f8ee4003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efaffd640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee40040a0 0x7f8ee40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efbfff640 1 -- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 msgr2=0x7f8ee4003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:34.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efbfff640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8efc0718d0 0x7f8ee4003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efbfff640 1 -- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee40040a0 msgr2=0x7f8ee40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:34.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efbfff640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee40040a0 0x7f8ee40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efbfff640 1 -- 172.21.15.31:0/1776166780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee4003530 con 0x7f8ee4004d10 2023-10-30T13:35:34.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.096+0000 7f8efaffd640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee40040a0 0x7f8ee40047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8efbfff640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ee4004d10 0x7f8ee4005190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4002040 tx=0x7f8ef401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df63774878f0256 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4003bd0 con 0x7f8ee4004d10 2023-10-30T13:35:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef4003d50 con 0x7f8ee4004d10 2023-10-30T13:35:34.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8efa7fc640 1 -- 172.21.15.31:0/1776166780 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4005310 con 0x7f8ee4004d10 2023-10-30T13:35:34.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8f00db5640 1 -- 172.21.15.31:0/1776166780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ee4004d10 msgr2=0x7f8ee4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8f00db5640 1 --2- 172.21.15.31:0/1776166780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ee4004d10 0x7f8ee4005190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4002040 tx=0x7f8ef401c8e0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8efbfff640 1 -- 172.21.15.31:0/1776166780 reap_dead start 2023-10-30T13:35:34.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8f00db5640 1 -- 172.21.15.31:0/1776166780 shutdown_connections 2023-10-30T13:35:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8f00db5640 1 -- 172.21.15.31:0/1776166780 >> 172.21.15.31:0/1776166780 conn(0x7f8efc06ab00 msgr2=0x7f8efc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8f00db5640 1 -- 172.21.15.31:0/1776166780 shutdown_connections 2023-10-30T13:35:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.100+0000 7f8f00db5640 1 -- 172.21.15.31:0/1776166780 wait complete. 2023-10-30T13:35:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 Processor -- start 2023-10-30T13:35:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 -- start start 2023-10-30T13:35:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ee4004d10 0x7f8efc148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8efc14faa0 0x7f8efc1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8efc158ce0 0x7f8efc1eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc14e950 con 0x7f8efc14faa0 2023-10-30T13:35:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc1f2a60 con 0x7f8efc158ce0 2023-10-30T13:35:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc1591d0 con 0x7f8ee4004d10 2023-10-30T13:35:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8efc14faa0 0x7f8efc1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8efc14faa0 0x7f8efc1ee590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36424/0 (socket says 172.21.15.31:36424) 2023-10-30T13:35:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 -- 172.21.15.31:0/2936178615 learned_addr learned my addr 172.21.15.31:0/2936178615 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efb7fe640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ee4004d10 0x7f8efc148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efbfff640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8efc158ce0 0x7f8efc1eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 -- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ee4004d10 msgr2=0x7f8efc148900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ee4004d10 0x7f8efc148900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 -- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8efc158ce0 msgr2=0x7f8efc1eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8efc158ce0 0x7f8efc1eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 -- 172.21.15.31:0/2936178615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc1f1ca0 con 0x7f8efc14faa0 2023-10-30T13:35:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8efaffd640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8efc14faa0 0x7f8efc1ee590 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000e7f0 tx=0x7f8ef000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8ee3fff640 1 -- 172.21.15.31:0/2936178615 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef000ee40 con 0x7f8efc14faa0 2023-10-30T13:35:34.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8ee3fff640 1 -- 172.21.15.31:0/2936178615 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef000c8d0 con 0x7f8efc14faa0 2023-10-30T13:35:34.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8ee3fff640 1 -- 172.21.15.31:0/2936178615 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef000cc60 con 0x7f8efc14faa0 2023-10-30T13:35:34.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc1f1f80 con 0x7f8efc14faa0 2023-10-30T13:35:34.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc1f2540 con 0x7f8efc14faa0 2023-10-30T13:35:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.104+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc04e170 con 0x7f8efc14faa0 2023-10-30T13:35:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.108+0000 7f8ee3fff640 1 -- 172.21.15.31:0/2936178615 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ef001d460 con 0x7f8efc14faa0 2023-10-30T13:35:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.108+0000 7f8ee3fff640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ee80664a0 0x7f8ee8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.108+0000 7f8ee3fff640 1 -- 172.21.15.31:0/2936178615 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ef0083d80 con 0x7f8efc14faa0 2023-10-30T13:35:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.108+0000 7f8efb7fe640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ee80664a0 0x7f8ee8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.108+0000 7f8efb7fe640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ee80664a0 0x7f8ee8068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009be0 tx=0x7f8eec009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.112+0000 7f8ee3fff640 1 -- 172.21.15.31:0/2936178615 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ef00511f0 con 0x7f8efc14faa0 2023-10-30T13:35:34.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.268+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8efc0c9f80 con 0x7f8efc14faa0 2023-10-30T13:35:34.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.268+0000 7f8ee3fff640 1 -- 172.21.15.31:0/2936178615 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8ef0054810 con 0x7f8efc14faa0 2023-10-30T13:35:34.273 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-10-30T13:35:34.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ee80664a0 msgr2=0x7f8ee8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ee80664a0 0x7f8ee8068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009be0 tx=0x7f8eec009190 comp rx=0 tx=0).stop 2023-10-30T13:35:34.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8efc14faa0 msgr2=0x7f8efc1ee590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8efc14faa0 0x7f8efc1ee590 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000e7f0 tx=0x7f8ef000ebf0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 shutdown_connections 2023-10-30T13:35:34.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8ee80664a0 0x7f8ee8068960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8efc158ce0 0x7f8efc1eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8efc14faa0 0x7f8efc1ee590 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 --2- 172.21.15.31:0/2936178615 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ee4004d10 0x7f8efc148900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 >> 172.21.15.31:0/2936178615 conn(0x7f8efc06ab00 msgr2=0x7f8efc15c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 shutdown_connections 2023-10-30T13:35:34.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.272+0000 7f8f00db5640 1 -- 172.21.15.31:0/2936178615 wait complete. 2023-10-30T13:35:34.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:35:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe62db1d640 1 Processor -- start 2023-10-30T13:35:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe62db1d640 1 -- start start 2023-10-30T13:35:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe62db1d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe628153c40 0x7fe62806d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe62db1d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe628071ac0 0x7fe62815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe62db1d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62815cf10 con 0x7fe62806dfc0 2023-10-30T13:35:34.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe62db1d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62806d8d0 con 0x7fe628153c40 2023-10-30T13:35:34.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe62db1d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6280722b0 con 0x7fe628071ac0 2023-10-30T13:35:34.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe626ffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe62806dfc0 0x7fe628069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55098/0 (socket says 172.21.15.31:55098) 2023-10-30T13:35:34.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe6277fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe628153c40 0x7fe62806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe626ffd640 1 -- 172.21.15.31:0/3509632378 learned_addr learned my addr 172.21.15.31:0/3509632378 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:34.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe627fff640 1 --2- 172.21.15.31:0/3509632378 >> v2:172.21.15.33:3300/0 conn(0x7fe628071ac0 0x7fe62815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe6277fe640 1 -- 172.21.15.31:0/3509632378 >> v2:172.21.15.33:3300/0 conn(0x7fe628071ac0 msgr2=0x7fe62815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe6277fe640 1 --2- 172.21.15.31:0/3509632378 >> v2:172.21.15.33:3300/0 conn(0x7fe628071ac0 0x7fe62815d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe6277fe640 1 -- 172.21.15.31:0/3509632378 >> v1:172.21.15.31:6789/0 conn(0x7fe62806dfc0 legacy=0x7fe628069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe6277fe640 1 -- 172.21.15.31:0/3509632378 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe628072590 con 0x7fe628153c40 2023-10-30T13:35:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe6267fc640 1 -- 172.21.15.31:0/3509632378 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193715335 0 0) 0x7fe62815cf10 con 0x7fe62806dfc0 2023-10-30T13:35:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.436+0000 7fe627fff640 1 --2- 172.21.15.31:0/3509632378 >> v2:172.21.15.33:3300/0 conn(0x7fe628071ac0 0x7fe62815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe6277fe640 1 --2- 172.21.15.31:0/3509632378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe628153c40 0x7fe62806d240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6280c9ce0 tx=0x7fe61801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779bc412485de7c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe6267fc640 1 -- 172.21.15.31:0/3509632378 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6180a7070 con 0x7fe628153c40 2023-10-30T13:35:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe6267fc640 1 -- 172.21.15.31:0/3509632378 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6180ab430 con 0x7fe628153c40 2023-10-30T13:35:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe6267fc640 1 -- 172.21.15.31:0/3509632378 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6180ab7c0 con 0x7fe628153c40 2023-10-30T13:35:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 -- 172.21.15.31:0/3509632378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe628153c40 msgr2=0x7fe62806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 --2- 172.21.15.31:0/3509632378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe628153c40 0x7fe62806d240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6280c9ce0 tx=0x7fe61801c430 comp rx=0 tx=0).stop 2023-10-30T13:35:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 -- 172.21.15.31:0/3509632378 shutdown_connections 2023-10-30T13:35:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 --2- 172.21.15.31:0/3509632378 >> v2:172.21.15.33:3300/0 conn(0x7fe628071ac0 0x7fe62815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 --2- 172.21.15.31:0/3509632378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe628153c40 0x7fe62806d240 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 -- 172.21.15.31:0/3509632378 >> 172.21.15.31:0/3509632378 conn(0x7fe62806afa0 msgr2=0x7fe62806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 -- 172.21.15.31:0/3509632378 shutdown_connections 2023-10-30T13:35:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.440+0000 7fe62db1d640 1 -- 172.21.15.31:0/3509632378 wait complete. 2023-10-30T13:35:34.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 Processor -- start 2023-10-30T13:35:34.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 -- start start 2023-10-30T13:35:34.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe62806dfc0 0x7fe62813e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe628071ac0 0x7fe6281f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe628153c40 0x7fe6281f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6281f60d0 con 0x7fe62806dfc0 2023-10-30T13:35:34.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6281527a0 con 0x7fe628153c40 2023-10-30T13:35:34.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62814e470 con 0x7fe628071ac0 2023-10-30T13:35:34.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe62806dfc0 0x7fe62813e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe62806dfc0 0x7fe62813e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36442/0 (socket says 172.21.15.31:36442) 2023-10-30T13:35:34.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 -- 172.21.15.31:0/1594269012 learned_addr learned my addr 172.21.15.31:0/1594269012 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:34.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe626ffd640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe628071ac0 0x7fe6281f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe627fff640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe628153c40 0x7fe6281f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 -- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe628071ac0 msgr2=0x7fe6281f1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe628071ac0 0x7fe6281f1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 -- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe628153c40 msgr2=0x7fe6281f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe628153c40 0x7fe6281f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 -- 172.21.15.31:0/1594269012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe618009420 con 0x7fe62806dfc0 2023-10-30T13:35:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe6277fe640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe62806dfc0 0x7fe62813e660 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe618019040 tx=0x7fe61801ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe60ffff640 1 -- 172.21.15.31:0/1594269012 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6180b5810 con 0x7fe62806dfc0 2023-10-30T13:35:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe60ffff640 1 -- 172.21.15.31:0/1594269012 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6180b5990 con 0x7fe62806dfc0 2023-10-30T13:35:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.444+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62813d980 con 0x7fe62806dfc0 2023-10-30T13:35:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.448+0000 7fe60ffff640 1 -- 172.21.15.31:0/1594269012 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6180b5d40 con 0x7fe62806dfc0 2023-10-30T13:35:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.448+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6281f55f0 con 0x7fe62806dfc0 2023-10-30T13:35:34.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.448+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62804e170 con 0x7fe62806dfc0 2023-10-30T13:35:34.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.448+0000 7fe60ffff640 1 -- 172.21.15.31:0/1594269012 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe6180b4390 con 0x7fe62806dfc0 2023-10-30T13:35:34.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.448+0000 7fe60ffff640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6140664a0 0x7fe614068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.448+0000 7fe60ffff640 1 -- 172.21.15.31:0/1594269012 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe61811e660 con 0x7fe62806dfc0 2023-10-30T13:35:34.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.448+0000 7fe626ffd640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6140664a0 0x7fe614068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.452+0000 7fe626ffd640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6140664a0 0x7fe614068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe62813ef20 tx=0x7fe61c007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.452+0000 7fe60ffff640 1 -- 172.21.15.31:0/1594269012 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe6180ebb80 con 0x7fe62806dfc0 2023-10-30T13:35:34.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe62806dda0 con 0x7fe62806dfc0 2023-10-30T13:35:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe60ffff640 1 -- 172.21.15.31:0/1594269012 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe618006e20 con 0x7fe62806dfc0 2023-10-30T13:35:34.615 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-10-30T13:35:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6140664a0 msgr2=0x7fe614068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6140664a0 0x7fe614068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe62813ef20 tx=0x7fe61c007b80 comp rx=0 tx=0).stop 2023-10-30T13:35:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe62806dfc0 msgr2=0x7fe62813e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe62806dfc0 0x7fe62813e660 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe618019040 tx=0x7fe61801ac70 comp rx=0 tx=0).stop 2023-10-30T13:35:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 shutdown_connections 2023-10-30T13:35:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6140664a0 0x7fe614068960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe628153c40 0x7fe6281f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe628071ac0 0x7fe6281f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 --2- 172.21.15.31:0/1594269012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe62806dfc0 0x7fe62813e660 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 >> 172.21.15.31:0/1594269012 conn(0x7fe62806afa0 msgr2=0x7fe62806b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 shutdown_connections 2023-10-30T13:35:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.612+0000 7fe62db1d640 1 -- 172.21.15.31:0/1594269012 wait complete. 2023-10-30T13:35:34.637 INFO:tasks.mds_thrash:mds thrasher using random seed: 1698672934 2023-10-30T13:35:34.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:34.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.776+0000 7f63ccfee640 1 Processor -- start 2023-10-30T13:35:34.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.776+0000 7f63ccfee640 1 -- start start 2023-10-30T13:35:34.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.776+0000 7f63ccfee640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f63c80c4a10 0x7f63c815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.776+0000 7f63ccfee640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c8159f20 con 0x7f63c8153c80 2023-10-30T13:35:34.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.776+0000 7f63ccfee640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c815d620 con 0x7f63c80c42a0 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.776+0000 7f63ccfee640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c815dec0 con 0x7f63c80c4a10 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63c659c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f63c8153c80 0x7f63c815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43370/0 (socket says 172.21.15.31:43370) 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63c659c640 1 -- 172.21.15.31:0/2460906547 learned_addr learned my addr 172.21.15.31:0/2460906547 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63bffff640 1 --2- 172.21.15.31:0/2460906547 >> v2:172.21.15.31:3300/0 conn(0x7f63c80c4a10 0x7f63c815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63c5d9b640 1 -- 172.21.15.31:0/2460906547 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957060639 0 0) 0x7f63c8159f20 con 0x7f63c8153c80 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63c5d9b640 1 -- 172.21.15.31:0/2460906547 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4003530 con 0x7f63c8153c80 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63bffff640 1 -- 172.21.15.31:0/2460906547 >> v1:172.21.15.33:6789/0 conn(0x7f63c80c42a0 legacy=0x7f63c80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63bffff640 1 -- 172.21.15.31:0/2460906547 >> v1:172.21.15.31:6789/0 conn(0x7f63c8153c80 legacy=0x7f63c815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63bffff640 1 -- 172.21.15.31:0/2460906547 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c815d900 con 0x7f63c80c4a10 2023-10-30T13:35:34.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63bffff640 1 --2- 172.21.15.31:0/2460906547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c4a10 0x7f63c815cf90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f63b00097c0 tx=0x7f63b001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6940851670c2b036 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63c5d9b640 1 -- 172.21.15.31:0/2460906547 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00a8970 con 0x7f63c80c4a10 2023-10-30T13:35:34.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63c5d9b640 1 -- 172.21.15.31:0/2460906547 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b00a8ad0 con 0x7f63c80c4a10 2023-10-30T13:35:34.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.780+0000 7f63c5d9b640 1 -- 172.21.15.31:0/2460906547 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00a8e60 con 0x7f63c80c4a10 2023-10-30T13:35:34.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- 172.21.15.31:0/2460906547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c4a10 msgr2=0x7f63c815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 --2- 172.21.15.31:0/2460906547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c4a10 0x7f63c815cf90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f63b00097c0 tx=0x7f63b001ec70 comp rx=0 tx=0).stop 2023-10-30T13:35:34.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- 172.21.15.31:0/2460906547 shutdown_connections 2023-10-30T13:35:34.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 --2- 172.21.15.31:0/2460906547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c4a10 0x7f63c815cf90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- 172.21.15.31:0/2460906547 >> 172.21.15.31:0/2460906547 conn(0x7f63c80715c0 msgr2=0x7f63c80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- 172.21.15.31:0/2460906547 shutdown_connections 2023-10-30T13:35:34.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- 172.21.15.31:0/2460906547 wait complete. 2023-10-30T13:35:34.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 Processor -- start 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- start start 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c42a0 0x7f63c8158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63c80c4a10 0x7f63c8158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f63c8153c80 0x7f63c81f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cf850 con 0x7f63c80c42a0 2023-10-30T13:35:34.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cfb30 con 0x7f63c80c4a10 2023-10-30T13:35:34.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63ccfee640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8154490 con 0x7f63c8153c80 2023-10-30T13:35:34.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c42a0 0x7f63c8158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c42a0 0x7f63c8158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51274/0 (socket says 172.21.15.31:51274) 2023-10-30T13:35:34.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 -- 172.21.15.31:0/152440671 learned_addr learned my addr 172.21.15.31:0/152440671 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:34.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63bffff640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63c80c4a10 0x7f63c8158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c6d9d640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f63c8153c80 0x7f63c81f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 -- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f63c8153c80 msgr2=0x7f63c81f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f63c8153c80 0x7f63c81f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 -- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63c80c4a10 msgr2=0x7f63c8158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63c80c4a10 0x7f63c8158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.784+0000 7f63c659c640 1 -- 172.21.15.31:0/152440671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b0009420 con 0x7f63c80c42a0 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63c659c640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c42a0 0x7f63c8158350 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f63b800c420 tx=0x7f63b800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63beffd640 1 -- 172.21.15.31:0/152440671 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8005cf0 con 0x7f63c80c42a0 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63beffd640 1 -- 172.21.15.31:0/152440671 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8005e50 con 0x7f63c80c42a0 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63beffd640 1 -- 172.21.15.31:0/152440671 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80072d0 con 0x7f63c80c42a0 2023-10-30T13:35:34.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c8154770 con 0x7f63c80c42a0 2023-10-30T13:35:34.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c814fc40 con 0x7f63c80c42a0 2023-10-30T13:35:34.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c81542f0 con 0x7f63c80c42a0 2023-10-30T13:35:34.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63beffd640 1 -- 172.21.15.31:0/152440671 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63b8007430 con 0x7f63c80c42a0 2023-10-30T13:35:34.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63beffd640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f63a80664a0 0x7f63a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.788+0000 7f63beffd640 1 -- 172.21.15.31:0/152440671 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63b8083370 con 0x7f63c80c42a0 2023-10-30T13:35:34.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.792+0000 7f63bffff640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f63a80664a0 0x7f63a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.792+0000 7f63bffff640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f63a80664a0 0x7f63a8068960 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f63b00b2000 tx=0x7f63b0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.792+0000 7f63beffd640 1 -- 172.21.15.31:0/152440671 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f63c81542f0 con 0x7f63c80c42a0 2023-10-30T13:35:34.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.952+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63c81542f0 con 0x7f63c80c42a0 2023-10-30T13:35:34.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.952+0000 7f63beffd640 1 -- 172.21.15.31:0/152440671 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f63c81542f0 con 0x7f63c80c42a0 2023-10-30T13:35:34.957 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:34.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f63a80664a0 msgr2=0x7f63a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f63a80664a0 0x7f63a8068960 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f63b00b2000 tx=0x7f63b0008040 comp rx=0 tx=0).stop 2023-10-30T13:35:34.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c42a0 msgr2=0x7f63c8158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c42a0 0x7f63c8158350 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f63b800c420 tx=0x7f63b800c8e0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 shutdown_connections 2023-10-30T13:35:34.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f63a80664a0 0x7f63a8068960 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f63c8153c80 0x7f63c81f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63c80c4a10 0x7f63c8158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 --2- 172.21.15.31:0/152440671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c80c42a0 0x7f63c8158350 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 >> 172.21.15.31:0/152440671 conn(0x7f63c80715c0 msgr2=0x7f63c8152510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 shutdown_connections 2023-10-30T13:35:34.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:34.956+0000 7f63ccfee640 1 -- 172.21.15.31:0/152440671 wait complete. 2023-10-30T13:35:34.979 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:35:05.304460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:35:06.652927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:35:34.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:35:35.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.120+0000 7f6f56b80640 1 Processor -- start 2023-10-30T13:35:35.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.120+0000 7f6f56b80640 1 -- start start 2023-10-30T13:35:35.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f56b80640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6f500c1020 0x7f6f50072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f56b80640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f501581f0 con 0x7f6f500725c0 2023-10-30T13:35:35.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f56b80640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f50151e00 con 0x7f6f50151a20 2023-10-30T13:35:35.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f56b80640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f501576d0 con 0x7f6f500c1020 2023-10-30T13:35:35.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f5511d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f6f500725c0 0x7f6f500c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43384/0 (socket says 172.21.15.31:43384) 2023-10-30T13:35:35.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f5511d640 1 -- 172.21.15.31:0/2708442365 learned_addr learned my addr 172.21.15.31:0/2708442365 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:35.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2708442365 >> v2:172.21.15.31:3300/0 conn(0x7f6f500c1020 0x7f6f50072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f4ffff640 1 -- 172.21.15.31:0/2708442365 >> v1:172.21.15.33:6789/0 conn(0x7f6f50151a20 legacy=0x7f6f500cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f4ffff640 1 -- 172.21.15.31:0/2708442365 >> v1:172.21.15.31:6789/0 conn(0x7f6f500725c0 legacy=0x7f6f500c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.124+0000 7f6f4ffff640 1 -- 172.21.15.31:0/2708442365 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f500c19d0 con 0x7f6f500c1020 2023-10-30T13:35:35.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2708442365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500c1020 0x7f6f50072080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0097c0 tx=0x7f6f3c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0e4c72b7435ee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/2708442365 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c0a8970 con 0x7f6f500c1020 2023-10-30T13:35:35.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/2708442365 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3c0a8ad0 con 0x7f6f500c1020 2023-10-30T13:35:35.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/2708442365 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c0a8e60 con 0x7f6f500c1020 2023-10-30T13:35:35.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 -- 172.21.15.31:0/2708442365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500c1020 msgr2=0x7f6f50072080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2708442365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500c1020 0x7f6f50072080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0097c0 tx=0x7f6f3c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:35:35.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 -- 172.21.15.31:0/2708442365 shutdown_connections 2023-10-30T13:35:35.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2708442365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500c1020 0x7f6f50072080 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 -- 172.21.15.31:0/2708442365 >> 172.21.15.31:0/2708442365 conn(0x7f6f5006da40 msgr2=0x7f6f5006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 -- 172.21.15.31:0/2708442365 shutdown_connections 2023-10-30T13:35:35.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 -- 172.21.15.31:0/2708442365 wait complete. 2023-10-30T13:35:35.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 Processor -- start 2023-10-30T13:35:35.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.128+0000 7f6f56b80640 1 -- start start 2023-10-30T13:35:35.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500725c0 0x7f6f501ed950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f500c1020 0x7f6f50146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f50151a20 0x7f6f50154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501ed4c0 con 0x7f6f500725c0 2023-10-30T13:35:35.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501ecee0 con 0x7f6f500c1020 2023-10-30T13:35:35.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501547a0 con 0x7f6f50151a20 2023-10-30T13:35:35.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500725c0 0x7f6f501ed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500725c0 0x7f6f501ed950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51292/0 (socket says 172.21.15.31:51292) 2023-10-30T13:35:35.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 -- 172.21.15.31:0/2832253602 learned_addr learned my addr 172.21.15.31:0/2832253602 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:35.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f500c1020 0x7f6f50146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5511d640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f50151a20 0x7f6f50154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 -- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f50151a20 msgr2=0x7f6f50154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f50151a20 0x7f6f50154260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 -- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f500c1020 msgr2=0x7f6f50146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f500c1020 0x7f6f50146c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 -- 172.21.15.31:0/2832253602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c009420 con 0x7f6f500725c0 2023-10-30T13:35:35.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f5491c640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500725c0 0x7f6f501ed950 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400c530 tx=0x7f6f4400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f4d7fa640 1 -- 172.21.15.31:0/2832253602 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44005220 con 0x7f6f500725c0 2023-10-30T13:35:35.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f4d7fa640 1 -- 172.21.15.31:0/2832253602 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f44005380 con 0x7f6f500725c0 2023-10-30T13:35:35.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f4d7fa640 1 -- 172.21.15.31:0/2832253602 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44007270 con 0x7f6f500725c0 2023-10-30T13:35:35.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f50154bd0 con 0x7f6f500725c0 2023-10-30T13:35:35.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f50152480 con 0x7f6f500725c0 2023-10-30T13:35:35.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5004e170 con 0x7f6f500725c0 2023-10-30T13:35:35.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.132+0000 7f6f4d7fa640 1 -- 172.21.15.31:0/2832253602 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6f440073d0 con 0x7f6f500725c0 2023-10-30T13:35:35.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.136+0000 7f6f4d7fa640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f340664a0 0x7f6f34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.136+0000 7f6f4d7fa640 1 -- 172.21.15.31:0/2832253602 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f440837d0 con 0x7f6f500725c0 2023-10-30T13:35:35.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.136+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f340664a0 0x7f6f34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.136+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f340664a0 0x7f6f34068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0b2000 tx=0x7f6f3c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.140+0000 7f6f4d7fa640 1 -- 172.21.15.31:0/2832253602 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6f44050c40 con 0x7f6f500725c0 2023-10-30T13:35:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f50144650 con 0x7f6f500725c0 2023-10-30T13:35:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f4d7fa640 1 -- 172.21.15.31:0/2832253602 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f44054260 con 0x7f6f500725c0 2023-10-30T13:35:35.300 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:35:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f340664a0 msgr2=0x7f6f34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f340664a0 0x7f6f34068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0b2000 tx=0x7f6f3c008040 comp rx=0 tx=0).stop 2023-10-30T13:35:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500725c0 msgr2=0x7f6f501ed950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500725c0 0x7f6f501ed950 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400c530 tx=0x7f6f4400c9f0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 shutdown_connections 2023-10-30T13:35:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f340664a0 0x7f6f34068960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f50151a20 0x7f6f50154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f500c1020 0x7f6f50146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 --2- 172.21.15.31:0/2832253602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f500725c0 0x7f6f501ed950 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 >> 172.21.15.31:0/2832253602 conn(0x7f6f5006da40 msgr2=0x7f6f50149560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 shutdown_connections 2023-10-30T13:35:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.296+0000 7f6f56b80640 1 -- 172.21.15.31:0/2832253602 wait complete. 2023-10-30T13:35:35.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:35:35.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f889d572640 1 Processor -- start 2023-10-30T13:35:35.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f889d572640 1 -- start start 2023-10-30T13:35:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f889d572640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8898153ba0 0x7f88980cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f889d572640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f88980c4810 0x7f889815a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f889d572640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8898153f80 con 0x7f88980c4410 2023-10-30T13:35:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f889d572640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8898159df0 con 0x7f88980c4810 2023-10-30T13:35:35.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f889d572640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889815a7c0 con 0x7f8898153ba0 2023-10-30T13:35:35.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f88977fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f88980c4810 0x7f889815a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f888ffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f88980c4410 0x7f889815cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43398/0 (socket says 172.21.15.31:43398) 2023-10-30T13:35:35.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f888ffff640 1 -- 172.21.15.31:0/845975504 learned_addr learned my addr 172.21.15.31:0/845975504 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:35.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f8896ffd640 1 --2- 172.21.15.31:0/845975504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8898153ba0 0x7f88980cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f88977fe640 1 -- 172.21.15.31:0/845975504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8898153ba0 msgr2=0x7f88980cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:35.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f88977fe640 1 --2- 172.21.15.31:0/845975504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8898153ba0 0x7f88980cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f88977fe640 1 -- 172.21.15.31:0/845975504 >> v1:172.21.15.31:6789/0 conn(0x7f88980c4410 legacy=0x7f889815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f88977fe640 1 -- 172.21.15.31:0/845975504 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88981597c0 con 0x7f88980c4810 2023-10-30T13:35:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.464+0000 7f88967fc640 1 -- 172.21.15.31:0/845975504 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133691012 0 0) 0x7f8898153f80 con 0x7f88980c4410 2023-10-30T13:35:35.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f88977fe640 1 --2- 172.21.15.31:0/845975504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88980c4810 0x7f889815a280 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8890009d90 tx=0x7f889001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad48279e033789f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f88967fc640 1 -- 172.21.15.31:0/845975504 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900a5950 con 0x7f88980c4810 2023-10-30T13:35:35.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f88967fc640 1 -- 172.21.15.31:0/845975504 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88900a5ab0 con 0x7f88980c4810 2023-10-30T13:35:35.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f88967fc640 1 -- 172.21.15.31:0/845975504 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900a5e40 con 0x7f88980c4810 2023-10-30T13:35:35.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 -- 172.21.15.31:0/845975504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88980c4810 msgr2=0x7f889815a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 --2- 172.21.15.31:0/845975504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88980c4810 0x7f889815a280 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8890009d90 tx=0x7f889001c890 comp rx=0 tx=0).stop 2023-10-30T13:35:35.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 -- 172.21.15.31:0/845975504 shutdown_connections 2023-10-30T13:35:35.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 --2- 172.21.15.31:0/845975504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88980c4810 0x7f889815a280 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 --2- 172.21.15.31:0/845975504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8898153ba0 0x7f88980cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 -- 172.21.15.31:0/845975504 >> 172.21.15.31:0/845975504 conn(0x7f8898071790 msgr2=0x7f8898071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 -- 172.21.15.31:0/845975504 shutdown_connections 2023-10-30T13:35:35.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.468+0000 7f889d572640 1 -- 172.21.15.31:0/845975504 wait complete. 2023-10-30T13:35:35.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 Processor -- start 2023-10-30T13:35:35.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 -- start start 2023-10-30T13:35:35.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88980c4410 0x7f88980ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88980c4810 0x7f889815a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8898153ba0 0x7f88981f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88980c8410 con 0x7f8898153ba0 2023-10-30T13:35:35.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88980c93a0 con 0x7f88980c4810 2023-10-30T13:35:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889814f080 con 0x7f88980c4410 2023-10-30T13:35:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8898153ba0 0x7f88981f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8898153ba0 0x7f88981f52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51316/0 (socket says 172.21.15.31:51316) 2023-10-30T13:35:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 -- 172.21.15.31:0/1627395284 learned_addr learned my addr 172.21.15.31:0/1627395284 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f8896ffd640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88980c4410 0x7f88980ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f888ffff640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88980c4810 0x7f889815a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 -- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88980c4410 msgr2=0x7f88980ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88980c4410 0x7f88980ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 -- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88980c4810 msgr2=0x7f889815a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88980c4810 0x7f889815a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 -- 172.21.15.31:0/1627395284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890009a70 con 0x7f8898153ba0 2023-10-30T13:35:35.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f88977fe640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8898153ba0 0x7f88981f52f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f88900b0040 tx=0x7f8890019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f888f7fe640 1 -- 172.21.15.31:0/1627395284 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900add90 con 0x7f8898153ba0 2023-10-30T13:35:35.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f888f7fe640 1 -- 172.21.15.31:0/1627395284 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890009450 con 0x7f8898153ba0 2023-10-30T13:35:35.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f888f7fe640 1 -- 172.21.15.31:0/1627395284 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900b9690 con 0x7f8898153ba0 2023-10-30T13:35:35.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889815b3c0 con 0x7f8898153ba0 2023-10-30T13:35:35.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88980c8a70 con 0x7f8898153ba0 2023-10-30T13:35:35.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.472+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88981f5010 con 0x7f8898153ba0 2023-10-30T13:35:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.476+0000 7f888f7fe640 1 -- 172.21.15.31:0/1627395284 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88900b97f0 con 0x7f8898153ba0 2023-10-30T13:35:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.476+0000 7f888f7fe640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88780664a0 0x7f8878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.476+0000 7f888f7fe640 1 -- 172.21.15.31:0/1627395284 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f889011f870 con 0x7f8898153ba0 2023-10-30T13:35:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.476+0000 7f8896ffd640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88780664a0 0x7f8878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.476+0000 7f8896ffd640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88780664a0 0x7f8878068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8888009a50 tx=0x7f8888008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.480+0000 7f888f7fe640 1 -- 172.21.15.31:0/1627395284 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88981f5010 con 0x7f8898153ba0 2023-10-30T13:35:35.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88981f5010 con 0x7f8898153ba0 2023-10-30T13:35:35.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f888f7fe640 1 -- 172.21.15.31:0/1627395284 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88981f5010 con 0x7f8898153ba0 2023-10-30T13:35:35.640 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-10-30T13:35:35.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88780664a0 msgr2=0x7f8878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88780664a0 0x7f8878068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8888009a50 tx=0x7f8888008040 comp rx=0 tx=0).stop 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8898153ba0 msgr2=0x7f88981f52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8898153ba0 0x7f88981f52f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f88900b0040 tx=0x7f8890019630 comp rx=0 tx=0).stop 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 shutdown_connections 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88780664a0 0x7f8878068960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8898153ba0 0x7f88981f52f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88980c4810 0x7f889815a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 --2- 172.21.15.31:0/1627395284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88980c4410 0x7f88980ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 >> 172.21.15.31:0/1627395284 conn(0x7f8898071790 msgr2=0x7f8898071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 shutdown_connections 2023-10-30T13:35:35.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.636+0000 7f889d572640 1 -- 172.21.15.31:0/1627395284 wait complete. 2023-10-30T13:35:35.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:35:35.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea61017640 1 Processor -- start 2023-10-30T13:35:35.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea61017640 1 -- start start 2023-10-30T13:35:35.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea61017640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fea5c151a20 0x7fea5c0cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea61017640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c1581f0 con 0x7fea5c0c1020 2023-10-30T13:35:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea61017640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c0cb9f0 con 0x7fea5c0c74a0 2023-10-30T13:35:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea61017640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c1576d0 con 0x7fea5c151a20 2023-10-30T13:35:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea5a59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fea5c0c1020 0x7fea5c072080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43408/0 (socket says 172.21.15.31:43408) 2023-10-30T13:35:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea5a59c640 1 -- 172.21.15.31:0/1474198778 learned_addr learned my addr 172.21.15.31:0/1474198778 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:35.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea5ad9d640 1 --2- 172.21.15.31:0/1474198778 >> v2:172.21.15.33:3300/0 conn(0x7fea5c151a20 0x7fea5c0cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290885470 0 0) 0x7fea5c1581f0 con 0x7fea5c0c1020 2023-10-30T13:35:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.804+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea44003530 con 0x7fea5c0c1020 2023-10-30T13:35:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165085478 0 0) 0x7fea44003530 con 0x7fea5c0c1020 2023-10-30T13:35:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 >> v2:172.21.15.33:3300/0 conn(0x7fea5c151a20 msgr2=0x7fea5c0cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 --2- 172.21.15.31:0/1474198778 >> v2:172.21.15.33:3300/0 conn(0x7fea5c151a20 0x7fea5c0cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 >> v1:172.21.15.33:6789/0 conn(0x7fea5c0c74a0 legacy=0x7fea5c161f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c151e00 con 0x7fea5c0c1020 2023-10-30T13:35:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fea50001680 con 0x7fea5c0c1020 2023-10-30T13:35:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 >> v1:172.21.15.31:6789/0 conn(0x7fea5c0c1020 legacy=0x7fea5c072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea44003980 0x7fea44003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea44004380 0x7fea44004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44005050 0x7fea4400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4400bed0 con 0x7fea44003980 2023-10-30T13:35:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4400c040 con 0x7fea44004380 2023-10-30T13:35:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4400c200 con 0x7fea44005050 2023-10-30T13:35:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5ad9d640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea44003980 0x7fea44003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5a59c640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea44004380 0x7fea44004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5b59e640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44005050 0x7fea4400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5ad9d640 1 -- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44005050 msgr2=0x7fea4400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5ad9d640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44005050 0x7fea4400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5ad9d640 1 -- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea44004380 msgr2=0x7fea44004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5ad9d640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea44004380 0x7fea44004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5ad9d640 1 -- 172.21.15.31:0/1474198778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44003530 con 0x7fea44003980 2023-10-30T13:35:35.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5b59e640 1 -- 172.21.15.31:0/1474198778 reap_dead start 2023-10-30T13:35:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.808+0000 7fea5ad9d640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea44003980 0x7fea44003d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009ae0 tx=0x7fea4c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbf01dec32e12c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c0a5660 con 0x7fea44003980 2023-10-30T13:35:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c0a57c0 con 0x7fea44003980 2023-10-30T13:35:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea59d9b640 1 -- 172.21.15.31:0/1474198778 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c0a5b70 con 0x7fea44003980 2023-10-30T13:35:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- 172.21.15.31:0/1474198778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea44003980 msgr2=0x7fea44003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea44003980 0x7fea44003d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009ae0 tx=0x7fea4c01ca70 comp rx=0 tx=0).stop 2023-10-30T13:35:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- 172.21.15.31:0/1474198778 shutdown_connections 2023-10-30T13:35:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 --2- 172.21.15.31:0/1474198778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea44003980 0x7fea44003d60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- 172.21.15.31:0/1474198778 >> 172.21.15.31:0/1474198778 conn(0x7fea5c06da40 msgr2=0x7fea5c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- 172.21.15.31:0/1474198778 shutdown_connections 2023-10-30T13:35:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- 172.21.15.31:0/1474198778 wait complete. 2023-10-30T13:35:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 Processor -- start 2023-10-30T13:35:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- start start 2023-10-30T13:35:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44003980 0x7fea5c0c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea5c071120 0x7fea5c150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea5c150c50 0x7fea5c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c0afb40 con 0x7fea5c150c50 2023-10-30T13:35:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c1510d0 con 0x7fea5c071120 2023-10-30T13:35:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.812+0000 7fea61017640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c1dd7e0 con 0x7fea44003980 2023-10-30T13:35:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea5c150c50 0x7fea5c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea5c150c50 0x7fea5c0adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51330/0 (socket says 172.21.15.31:51330) 2023-10-30T13:35:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5ad9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44003980 0x7fea5c0c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 -- 172.21.15.31:0/2403041983 learned_addr learned my addr 172.21.15.31:0/2403041983 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5a59c640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea5c071120 0x7fea5c150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 -- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44003980 msgr2=0x7fea5c0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44003980 0x7fea5c0c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 -- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea5c071120 msgr2=0x7fea5c150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea5c071120 0x7fea5c150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 -- 172.21.15.31:0/2403041983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c152350 con 0x7fea5c150c50 2023-10-30T13:35:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5a59c640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea5c071120 0x7fea5c150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea5b59e640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea5c150c50 0x7fea5c0adef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea540059b0 tx=0x7fea5400f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea437fe640 1 -- 172.21.15.31:0/2403041983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5400fda0 con 0x7fea5c150c50 2023-10-30T13:35:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea437fe640 1 -- 172.21.15.31:0/2403041983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea54015410 con 0x7fea5c150c50 2023-10-30T13:35:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea437fe640 1 -- 172.21.15.31:0/2403041983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea540157c0 con 0x7fea5c150c50 2023-10-30T13:35:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5c152b60 con 0x7fea5c150c50 2023-10-30T13:35:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c1525a0 con 0x7fea5c150c50 2023-10-30T13:35:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c04e170 con 0x7fea5c150c50 2023-10-30T13:35:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.816+0000 7fea437fe640 1 -- 172.21.15.31:0/2403041983 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea54015920 con 0x7fea5c150c50 2023-10-30T13:35:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.820+0000 7fea437fe640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea480664d0 0x7fea48068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.820+0000 7fea437fe640 1 -- 172.21.15.31:0/2403041983 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea54087300 con 0x7fea5c150c50 2023-10-30T13:35:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.820+0000 7fea5ad9d640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea480664d0 0x7fea48068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.820+0000 7fea5ad9d640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea480664d0 0x7fea48068990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0af000 tx=0x7fea4c0046d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.820+0000 7fea437fe640 1 -- 172.21.15.31:0/2403041983 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea54058200 con 0x7fea5c150c50 2023-10-30T13:35:35.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fea5c0cafe0 con 0x7fea5c150c50 2023-10-30T13:35:35.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea437fe640 1 -- 172.21.15.31:0/2403041983 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fea54057d90 con 0x7fea5c150c50 2023-10-30T13:35:35.984 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-10-30T13:35:35.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea480664d0 msgr2=0x7fea48068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea480664d0 0x7fea48068990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0af000 tx=0x7fea4c0046d0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea5c150c50 msgr2=0x7fea5c0adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea5c150c50 0x7fea5c0adef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea540059b0 tx=0x7fea5400f540 comp rx=0 tx=0).stop 2023-10-30T13:35:35.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 shutdown_connections 2023-10-30T13:35:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea480664d0 0x7fea48068990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea5c150c50 0x7fea5c0adef0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea5c071120 0x7fea5c150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 --2- 172.21.15.31:0/2403041983 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea44003980 0x7fea5c0c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 >> 172.21.15.31:0/2403041983 conn(0x7fea5c06da40 msgr2=0x7fea5c06de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 shutdown_connections 2023-10-30T13:35:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:35.980+0000 7fea61017640 1 -- 172.21.15.31:0/2403041983 wait complete. 2023-10-30T13:35:36.005 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-30T13:35:36.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:36.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f265d06b640 1 Processor -- start 2023-10-30T13:35:36.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f265d06b640 1 -- start start 2023-10-30T13:35:36.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f265d06b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2658153c80 0x7f265815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f265d06b640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f26580c4a10 0x7f26580c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f265d06b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2658159f20 con 0x7f265815a3b0 2023-10-30T13:35:36.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f265d06b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265815dec0 con 0x7f2658153c80 2023-10-30T13:35:36.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f265d06b640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26581599a0 con 0x7f26580c4a10 2023-10-30T13:35:36.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f2656d9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2658153c80 0x7f265815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f2656d9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2658153c80 0x7f265815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51332/0 (socket says 172.21.15.31:51332) 2023-10-30T13:35:36.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f2656d9d640 1 -- 172.21.15.31:0/2230262288 learned_addr learned my addr 172.21.15.31:0/2230262288 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:36.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f264ffff640 1 --2- 172.21.15.31:0/2230262288 >> v2:172.21.15.33:3300/0 conn(0x7f26580c4a10 0x7f26580c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f2656d9d640 1 -- 172.21.15.31:0/2230262288 >> v2:172.21.15.33:3300/0 conn(0x7f26580c4a10 msgr2=0x7f26580c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:36.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f2656d9d640 1 --2- 172.21.15.31:0/2230262288 >> v2:172.21.15.33:3300/0 conn(0x7f26580c4a10 0x7f26580c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f2656d9d640 1 -- 172.21.15.31:0/2230262288 >> v1:172.21.15.33:6789/0 conn(0x7f265815a3b0 legacy=0x7f26580ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.144+0000 7f2656d9d640 1 -- 172.21.15.31:0/2230262288 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26580c47e0 con 0x7f2658153c80 2023-10-30T13:35:36.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f2656d9d640 1 --2- 172.21.15.31:0/2230262288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2658153c80 0x7f265815cf90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f265806be50 tx=0x7f264801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bfec87b388bdd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265659c640 1 -- 172.21.15.31:0/2230262288 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26480a89f0 con 0x7f2658153c80 2023-10-30T13:35:36.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265659c640 1 -- 172.21.15.31:0/2230262288 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26480a8b50 con 0x7f2658153c80 2023-10-30T13:35:36.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265659c640 1 -- 172.21.15.31:0/2230262288 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26480b0440 con 0x7f2658153c80 2023-10-30T13:35:36.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 -- 172.21.15.31:0/2230262288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2658153c80 msgr2=0x7f265815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 --2- 172.21.15.31:0/2230262288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2658153c80 0x7f265815cf90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f265806be50 tx=0x7f264801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 -- 172.21.15.31:0/2230262288 shutdown_connections 2023-10-30T13:35:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 --2- 172.21.15.31:0/2230262288 >> v2:172.21.15.33:3300/0 conn(0x7f26580c4a10 0x7f26580c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 --2- 172.21.15.31:0/2230262288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2658153c80 0x7f265815cf90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 -- 172.21.15.31:0/2230262288 >> 172.21.15.31:0/2230262288 conn(0x7f26580715c0 msgr2=0x7f26580719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 -- 172.21.15.31:0/2230262288 shutdown_connections 2023-10-30T13:35:36.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.148+0000 7f265d06b640 1 -- 172.21.15.31:0/2230262288 wait complete. 2023-10-30T13:35:36.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 Processor -- start 2023-10-30T13:35:36.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 -- start start 2023-10-30T13:35:36.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26580c4a10 0x7f2658152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2658153c80 0x7f26581f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f265815a3b0 0x7f2658145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26581531f0 con 0x7f265815a3b0 2023-10-30T13:35:36.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265815ab60 con 0x7f26580c4a10 2023-10-30T13:35:36.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265815ae00 con 0x7f2658153c80 2023-10-30T13:35:36.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f265815a3b0 0x7f2658145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f265815a3b0 0x7f2658145a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51342/0 (socket says 172.21.15.31:51342) 2023-10-30T13:35:36.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 -- 172.21.15.31:0/1331795983 learned_addr learned my addr 172.21.15.31:0/1331795983 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:36.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f2656d9d640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26580c4a10 0x7f2658152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f264ffff640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2658153c80 0x7f26581f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 -- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2658153c80 msgr2=0x7f26581f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2658153c80 0x7f26581f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 -- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26580c4a10 msgr2=0x7f2658152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26580c4a10 0x7f2658152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 -- 172.21.15.31:0/1331795983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480094d0 con 0x7f265815a3b0 2023-10-30T13:35:36.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265759e640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f265815a3b0 0x7f2658145a60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f265000c730 tx=0x7f265000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f264f7fe640 1 -- 172.21.15.31:0/1331795983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650003b80 con 0x7f265815a3b0 2023-10-30T13:35:36.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f264f7fe640 1 -- 172.21.15.31:0/1331795983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2650003ce0 con 0x7f265815a3b0 2023-10-30T13:35:36.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26581f2cb0 con 0x7f265815a3b0 2023-10-30T13:35:36.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f264f7fe640 1 -- 172.21.15.31:0/1331795983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265000aa30 con 0x7f265815a3b0 2023-10-30T13:35:36.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.152+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265815c820 con 0x7f265815a3b0 2023-10-30T13:35:36.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.156+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26581529d0 con 0x7f265815a3b0 2023-10-30T13:35:36.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.156+0000 7f264f7fe640 1 -- 172.21.15.31:0/1331795983 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f265000ab90 con 0x7f265815a3b0 2023-10-30T13:35:36.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.156+0000 7f264f7fe640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26380664a0 0x7f2638068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.156+0000 7f264f7fe640 1 -- 172.21.15.31:0/1331795983 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2650083ba0 con 0x7f265815a3b0 2023-10-30T13:35:36.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.156+0000 7f2656d9d640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26380664a0 0x7f2638068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.160+0000 7f2656d9d640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26380664a0 0x7f2638068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f26480b2000 tx=0x7f2648008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.160+0000 7f264f7fe640 1 -- 172.21.15.31:0/1331795983 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26581529d0 con 0x7f265815a3b0 2023-10-30T13:35:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.352+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26581529d0 con 0x7f265815a3b0 2023-10-30T13:35:36.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.352+0000 7f264f7fe640 1 -- 172.21.15.31:0/1331795983 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7f26581529d0 con 0x7f265815a3b0 2023-10-30T13:35:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:36.358 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:35:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26380664a0 msgr2=0x7f2638068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26380664a0 0x7f2638068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f26480b2000 tx=0x7f2648008040 comp rx=0 tx=0).stop 2023-10-30T13:35:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f265815a3b0 msgr2=0x7f2658145a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f265815a3b0 0x7f2658145a60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f265000c730 tx=0x7f265000cbf0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 shutdown_connections 2023-10-30T13:35:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26380664a0 0x7f2638068960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f265815a3b0 0x7f2658145a60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2658153c80 0x7f26581f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 --2- 172.21.15.31:0/1331795983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26580c4a10 0x7f2658152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 >> 172.21.15.31:0/1331795983 conn(0x7f26580715c0 msgr2=0x7f26580caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 shutdown_connections 2023-10-30T13:35:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.356+0000 7f265d06b640 1 -- 172.21.15.31:0/1331795983 wait complete. 2023-10-30T13:35:36.378 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-30T13:35:36.379 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-30T13:35:36.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:35:36.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.516+0000 7fb197e6b640 1 Processor -- start 2023-10-30T13:35:36.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.516+0000 7fb197e6b640 1 -- start start 2023-10-30T13:35:36.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb197e6b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb19006b760 0x7fb19015e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb197e6b640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb19006bb40 0x7fb190157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb197e6b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190071e00 con 0x7fb19006c420 2023-10-30T13:35:36.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb197e6b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb190070020 con 0x7fb19006b760 2023-10-30T13:35:36.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb197e6b640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19015eb90 con 0x7fb19006bb40 2023-10-30T13:35:36.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195c07640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb19006b760 0x7fb19015e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195c07640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb19006b760 0x7fb19015e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51344/0 (socket says 172.21.15.31:51344) 2023-10-30T13:35:36.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195c07640 1 -- 172.21.15.31:0/3099744322 learned_addr learned my addr 172.21.15.31:0/3099744322 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:36.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195406640 1 --2- 172.21.15.31:0/3099744322 >> v2:172.21.15.33:3300/0 conn(0x7fb19006bb40 0x7fb190157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195c07640 1 -- 172.21.15.31:0/3099744322 >> v2:172.21.15.33:3300/0 conn(0x7fb19006bb40 msgr2=0x7fb190157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:36.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195c07640 1 --2- 172.21.15.31:0/3099744322 >> v2:172.21.15.33:3300/0 conn(0x7fb19006bb40 0x7fb190157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195c07640 1 -- 172.21.15.31:0/3099744322 >> v1:172.21.15.33:6789/0 conn(0x7fb19006c420 legacy=0x7fb1900c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.520+0000 7fb195c07640 1 -- 172.21.15.31:0/3099744322 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190070f10 con 0x7fb19006b760 2023-10-30T13:35:36.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb195c07640 1 --2- 172.21.15.31:0/3099744322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006b760 0x7fb19015e650 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb19006fa40 tx=0x7fb18c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bd4429ac97ed97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb194c05640 1 -- 172.21.15.31:0/3099744322 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c0aa430 con 0x7fb19006b760 2023-10-30T13:35:36.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb194c05640 1 -- 172.21.15.31:0/3099744322 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18c0aa590 con 0x7fb19006b760 2023-10-30T13:35:36.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb194c05640 1 -- 172.21.15.31:0/3099744322 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c0aa920 con 0x7fb19006b760 2023-10-30T13:35:36.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 -- 172.21.15.31:0/3099744322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006b760 msgr2=0x7fb19015e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 --2- 172.21.15.31:0/3099744322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006b760 0x7fb19015e650 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb19006fa40 tx=0x7fb18c01c4d0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 -- 172.21.15.31:0/3099744322 shutdown_connections 2023-10-30T13:35:36.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 --2- 172.21.15.31:0/3099744322 >> v2:172.21.15.33:3300/0 conn(0x7fb19006bb40 0x7fb190157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 --2- 172.21.15.31:0/3099744322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006b760 0x7fb19015e650 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 -- 172.21.15.31:0/3099744322 >> 172.21.15.31:0/3099744322 conn(0x7fb1900cac70 msgr2=0x7fb1900cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 -- 172.21.15.31:0/3099744322 shutdown_connections 2023-10-30T13:35:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 -- 172.21.15.31:0/3099744322 wait complete. 2023-10-30T13:35:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 Processor -- start 2023-10-30T13:35:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.524+0000 7fb197e6b640 1 -- start start 2023-10-30T13:35:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb19006b760 0x7fb1901513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006bb40 0x7fb1900c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb19006c420 0x7fb19014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1901479a0 con 0x7fb19006bb40 2023-10-30T13:35:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900caa40 con 0x7fb19006c420 2023-10-30T13:35:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19014ed30 con 0x7fb19006b760 2023-10-30T13:35:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006bb40 0x7fb1900c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006bb40 0x7fb1900c9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51346/0 (socket says 172.21.15.31:51346) 2023-10-30T13:35:36.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 -- 172.21.15.31:0/1745807560 learned_addr learned my addr 172.21.15.31:0/1745807560 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:36.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195c07640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb19006b760 0x7fb1901513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb196408640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb19006c420 0x7fb19014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 -- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb19006b760 msgr2=0x7fb1901513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb19006b760 0x7fb1901513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 -- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb19006c420 msgr2=0x7fb19014ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb19006c420 0x7fb19014ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 -- 172.21.15.31:0/1745807560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c0094d0 con 0x7fb19006bb40 2023-10-30T13:35:36.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb195406640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006bb40 0x7fb1900c9dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb18000d7d0 tx=0x7fb18000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb1867fc640 1 -- 172.21.15.31:0/1745807560 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180018070 con 0x7fb19006bb40 2023-10-30T13:35:36.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb1867fc640 1 -- 172.21.15.31:0/1745807560 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb180004020 con 0x7fb19006bb40 2023-10-30T13:35:36.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb1867fc640 1 -- 172.21.15.31:0/1745807560 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1800041d0 con 0x7fb19006bb40 2023-10-30T13:35:36.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19014f010 con 0x7fb19006bb40 2023-10-30T13:35:36.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19014d5b0 con 0x7fb19006bb40 2023-10-30T13:35:36.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19004e170 con 0x7fb19006bb40 2023-10-30T13:35:36.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.528+0000 7fb1867fc640 1 -- 172.21.15.31:0/1745807560 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb180004330 con 0x7fb19006bb40 2023-10-30T13:35:36.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.532+0000 7fb1867fc640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1740664d0 0x7fb174068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.532+0000 7fb1867fc640 1 -- 172.21.15.31:0/1745807560 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb180083ce0 con 0x7fb19006bb40 2023-10-30T13:35:36.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.532+0000 7fb195c07640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1740664d0 0x7fb174068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.532+0000 7fb195c07640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1740664d0 0x7fb174068990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0ae000 tx=0x7fb18c0b5040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.536+0000 7fb1867fc640 1 -- 172.21.15.31:0/1745807560 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1800510c0 con 0x7fb19006bb40 2023-10-30T13:35:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.688+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb19014d430 con 0x7fb1740664d0 2023-10-30T13:35:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.708+0000 7fb1867fc640 1 -- 172.21.15.31:0/1745807560 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264699 (secure 0 0 0) 0x7fb18c0aa960 con 0x7fb1740664d0 2023-10-30T13:35:36.715 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:35:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1740664d0 msgr2=0x7fb174068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1740664d0 0x7fb174068990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0ae000 tx=0x7fb18c0b5040 comp rx=0 tx=0).stop 2023-10-30T13:35:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006bb40 msgr2=0x7fb1900c9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006bb40 0x7fb1900c9dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb18000d7d0 tx=0x7fb18000dc90 comp rx=0 tx=0).stop 2023-10-30T13:35:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 shutdown_connections 2023-10-30T13:35:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1740664d0 0x7fb174068990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb19006c420 0x7fb19014ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb19006bb40 0x7fb1900c9dd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 --2- 172.21.15.31:0/1745807560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb19006b760 0x7fb1901513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 >> 172.21.15.31:0/1745807560 conn(0x7fb1900cac70 msgr2=0x7fb1900c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 shutdown_connections 2023-10-30T13:35:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.712+0000 7fb197e6b640 1 -- 172.21.15.31:0/1745807560 wait complete. 2023-10-30T13:35:36.735 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T13:35:35.166199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596106,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":287,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219080,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755640,"statfs":{"total":773094113280,"available":772869775360,"internally_reserved":0,"allocated":3022848,"data_stored":1775020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3328,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005026"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994380+0000","last_change":"2023-10-30T13:35:07.792742+0000","last_active":"2023-10-30T13:35:10.994380+0000","last_peered":"2023-10-30T13:35:10.994380+0000","last_clean":"2023-10-30T13:35:10.994380+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:35:10.994380+0000","last_undegraded":"2023-10-30T13:35:10.994380+0000","last_fullsized":"2023-10-30T13:35:10.994380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:02.805042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744019+0000","last_change":"2023-10-30T13:35:07.755187+0000","last_active":"2023-10-30T13:35:08.744019+0000","last_peered":"2023-10-30T13:35:08.744019+0000","last_clean":"2023-10-30T13:35:08.744019+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:35:08.744019+0000","last_undegraded":"2023-10-30T13:35:08.744019+0000","last_fullsized":"2023-10-30T13:35:08.744019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:22.764808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298537+0000","last_change":"2023-10-30T13:35:07.754357+0000","last_active":"2023-10-30T13:35:09.298537+0000","last_peered":"2023-10-30T13:35:09.298537+0000","last_clean":"2023-10-30T13:35:09.298537+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:35:09.298537+0000","last_undegraded":"2023-10-30T13:35:09.298537+0000","last_fullsized":"2023-10-30T13:35:09.298537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:23.201416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748146+0000","last_change":"2023-10-30T13:35:07.792466+0000","last_active":"2023-10-30T13:35:08.748146+0000","last_peered":"2023-10-30T13:35:08.748146+0000","last_clean":"2023-10-30T13:35:08.748146+0000","last_became_active":"2023-10-30T13:35:07.789749+0000","last_became_peered":"2023-10-30T13:35:07.789749+0000","last_unstale":"2023-10-30T13:35:08.748146+0000","last_undegraded":"2023-10-30T13:35:08.748146+0000","last_fullsized":"2023-10-30T13:35:08.748146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:46.527032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993635+0000","last_change":"2023-10-30T13:35:07.793403+0000","last_active":"2023-10-30T13:35:10.993635+0000","last_peered":"2023-10-30T13:35:10.993635+0000","last_clean":"2023-10-30T13:35:10.993635+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:35:10.993635+0000","last_undegraded":"2023-10-30T13:35:10.993635+0000","last_fullsized":"2023-10-30T13:35:10.993635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:27.304652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747210+0000","last_change":"2023-10-30T13:35:07.792285+0000","last_active":"2023-10-30T13:35:08.747210+0000","last_peered":"2023-10-30T13:35:08.747210+0000","last_clean":"2023-10-30T13:35:08.747210+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:35:08.747210+0000","last_undegraded":"2023-10-30T13:35:08.747210+0000","last_fullsized":"2023-10-30T13:35:08.747210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:31.537183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993398+0000","last_change":"2023-10-30T13:35:07.776386+0000","last_active":"2023-10-30T13:35:10.993398+0000","last_peered":"2023-10-30T13:35:10.993398+0000","last_clean":"2023-10-30T13:35:10.993398+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:35:10.993398+0000","last_undegraded":"2023-10-30T13:35:10.993398+0000","last_fullsized":"2023-10-30T13:35:10.993398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:27.492389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531107+0000","last_change":"2023-10-30T13:35:07.790487+0000","last_active":"2023-10-30T13:35:09.531107+0000","last_peered":"2023-10-30T13:35:09.531107+0000","last_clean":"2023-10-30T13:35:09.531107+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:35:09.531107+0000","last_undegraded":"2023-10-30T13:35:09.531107+0000","last_fullsized":"2023-10-30T13:35:09.531107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:27.384592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994435+0000","last_change":"2023-10-30T13:35:07.804038+0000","last_active":"2023-10-30T13:35:10.994435+0000","last_peered":"2023-10-30T13:35:10.994435+0000","last_clean":"2023-10-30T13:35:10.994435+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:35:10.994435+0000","last_undegraded":"2023-10-30T13:35:10.994435+0000","last_fullsized":"2023-10-30T13:35:10.994435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:09.137404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993722+0000","last_change":"2023-10-30T13:35:07.800657+0000","last_active":"2023-10-30T13:35:10.993722+0000","last_peered":"2023-10-30T13:35:10.993722+0000","last_clean":"2023-10-30T13:35:10.993722+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:35:10.993722+0000","last_undegraded":"2023-10-30T13:35:10.993722+0000","last_fullsized":"2023-10-30T13:35:10.993722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:12.080911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993171+0000","last_change":"2023-10-30T13:35:07.802245+0000","last_active":"2023-10-30T13:35:10.993171+0000","last_peered":"2023-10-30T13:35:10.993171+0000","last_clean":"2023-10-30T13:35:10.993171+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:35:10.993171+0000","last_undegraded":"2023-10-30T13:35:10.993171+0000","last_fullsized":"2023-10-30T13:35:10.993171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:58.462586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750176+0000","last_change":"2023-10-30T13:35:07.783940+0000","last_active":"2023-10-30T13:35:08.750176+0000","last_peered":"2023-10-30T13:35:08.750176+0000","last_clean":"2023-10-30T13:35:08.750176+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:35:08.750176+0000","last_undegraded":"2023-10-30T13:35:08.750176+0000","last_fullsized":"2023-10-30T13:35:08.750176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:07.873653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531433+0000","last_change":"2023-10-30T13:35:07.780349+0000","last_active":"2023-10-30T13:35:09.531433+0000","last_peered":"2023-10-30T13:35:09.531433+0000","last_clean":"2023-10-30T13:35:09.531433+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:35:09.531433+0000","last_undegraded":"2023-10-30T13:35:09.531433+0000","last_fullsized":"2023-10-30T13:35:09.531433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:15.619776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298339+0000","last_change":"2023-10-30T13:35:07.750959+0000","last_active":"2023-10-30T13:35:09.298339+0000","last_peered":"2023-10-30T13:35:09.298339+0000","last_clean":"2023-10-30T13:35:09.298339+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:35:09.298339+0000","last_undegraded":"2023-10-30T13:35:09.298339+0000","last_fullsized":"2023-10-30T13:35:09.298339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:56.853163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.768466+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:27.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748523+0000","last_change":"2023-10-30T13:35:07.789918+0000","last_active":"2023-10-30T13:35:08.748523+0000","last_peered":"2023-10-30T13:35:08.748523+0000","last_clean":"2023-10-30T13:35:08.748523+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:35:08.748523+0000","last_undegraded":"2023-10-30T13:35:08.748523+0000","last_fullsized":"2023-10-30T13:35:08.748523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:53.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749638+0000","last_change":"2023-10-30T13:35:07.796429+0000","last_active":"2023-10-30T13:35:08.749638+0000","last_peered":"2023-10-30T13:35:08.749638+0000","last_clean":"2023-10-30T13:35:08.749638+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:35:08.749638+0000","last_undegraded":"2023-10-30T13:35:08.749638+0000","last_fullsized":"2023-10-30T13:35:08.749638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:15.022866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489395+0000","last_change":"2023-10-30T13:35:07.780131+0000","last_active":"2023-10-30T13:35:10.489395+0000","last_peered":"2023-10-30T13:35:10.489395+0000","last_clean":"2023-10-30T13:35:10.489395+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:35:10.489395+0000","last_undegraded":"2023-10-30T13:35:10.489395+0000","last_fullsized":"2023-10-30T13:35:10.489395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:04.046338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994792+0000","last_change":"2023-10-30T13:35:07.791464+0000","last_active":"2023-10-30T13:35:10.994792+0000","last_peered":"2023-10-30T13:35:10.994792+0000","last_clean":"2023-10-30T13:35:10.994792+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:35:10.994792+0000","last_undegraded":"2023-10-30T13:35:10.994792+0000","last_fullsized":"2023-10-30T13:35:10.994792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993329+0000","last_change":"2023-10-30T13:35:07.783079+0000","last_active":"2023-10-30T13:35:10.993329+0000","last_peered":"2023-10-30T13:35:10.993329+0000","last_clean":"2023-10-30T13:35:10.993329+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:35:10.993329+0000","last_undegraded":"2023-10-30T13:35:10.993329+0000","last_fullsized":"2023-10-30T13:35:10.993329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:43:32.360152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298481+0000","last_change":"2023-10-30T13:35:07.794666+0000","last_active":"2023-10-30T13:35:09.298481+0000","last_peered":"2023-10-30T13:35:09.298481+0000","last_clean":"2023-10-30T13:35:09.298481+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:35:09.298481+0000","last_undegraded":"2023-10-30T13:35:09.298481+0000","last_fullsized":"2023-10-30T13:35:09.298481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:52.189022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747346+0000","last_change":"2023-10-30T13:35:07.796601+0000","last_active":"2023-10-30T13:35:08.747346+0000","last_peered":"2023-10-30T13:35:08.747346+0000","last_clean":"2023-10-30T13:35:08.747346+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:35:08.747346+0000","last_undegraded":"2023-10-30T13:35:08.747346+0000","last_fullsized":"2023-10-30T13:35:08.747346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:07.572028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531135+0000","last_change":"2023-10-30T13:35:07.767965+0000","last_active":"2023-10-30T13:35:09.531135+0000","last_peered":"2023-10-30T13:35:09.531135+0000","last_clean":"2023-10-30T13:35:09.531135+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:35:09.531135+0000","last_undegraded":"2023-10-30T13:35:09.531135+0000","last_fullsized":"2023-10-30T13:35:09.531135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:15.544113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748017+0000","last_change":"2023-10-30T13:35:07.772029+0000","last_active":"2023-10-30T13:35:08.748017+0000","last_peered":"2023-10-30T13:35:08.748017+0000","last_clean":"2023-10-30T13:35:08.748017+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:35:08.748017+0000","last_undegraded":"2023-10-30T13:35:08.748017+0000","last_fullsized":"2023-10-30T13:35:08.748017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:13.305147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531716+0000","last_change":"2023-10-30T13:35:07.758274+0000","last_active":"2023-10-30T13:35:09.531716+0000","last_peered":"2023-10-30T13:35:09.531716+0000","last_clean":"2023-10-30T13:35:09.531716+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:35:09.531716+0000","last_undegraded":"2023-10-30T13:35:09.531716+0000","last_fullsized":"2023-10-30T13:35:09.531716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:15.191933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994099+0000","last_change":"2023-10-30T13:35:07.803220+0000","last_active":"2023-10-30T13:35:10.994099+0000","last_peered":"2023-10-30T13:35:10.994099+0000","last_clean":"2023-10-30T13:35:10.994099+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:35:10.994099+0000","last_undegraded":"2023-10-30T13:35:10.994099+0000","last_fullsized":"2023-10-30T13:35:10.994099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:45.165258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743680+0000","last_change":"2023-10-30T13:35:07.753288+0000","last_active":"2023-10-30T13:35:08.743680+0000","last_peered":"2023-10-30T13:35:08.743680+0000","last_clean":"2023-10-30T13:35:08.743680+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:35:08.743680+0000","last_undegraded":"2023-10-30T13:35:08.743680+0000","last_fullsized":"2023-10-30T13:35:08.743680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:43.815379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531511+0000","last_change":"2023-10-30T13:35:07.767931+0000","last_active":"2023-10-30T13:35:09.531511+0000","last_peered":"2023-10-30T13:35:09.531511+0000","last_clean":"2023-10-30T13:35:09.531511+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:35:09.531511+0000","last_undegraded":"2023-10-30T13:35:09.531511+0000","last_fullsized":"2023-10-30T13:35:09.531511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:43.830918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298592+0000","last_change":"2023-10-30T13:35:07.753695+0000","last_active":"2023-10-30T13:35:09.298592+0000","last_peered":"2023-10-30T13:35:09.298592+0000","last_clean":"2023-10-30T13:35:09.298592+0000","last_became_active":"2023-10-30T13:35:07.753358+0000","last_became_peered":"2023-10-30T13:35:07.753358+0000","last_unstale":"2023-10-30T13:35:09.298592+0000","last_undegraded":"2023-10-30T13:35:09.298592+0000","last_fullsized":"2023-10-30T13:35:09.298592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:12.840727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747580+0000","last_change":"2023-10-30T13:35:07.790236+0000","last_active":"2023-10-30T13:35:08.747580+0000","last_peered":"2023-10-30T13:35:08.747580+0000","last_clean":"2023-10-30T13:35:08.747580+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:35:08.747580+0000","last_undegraded":"2023-10-30T13:35:08.747580+0000","last_fullsized":"2023-10-30T13:35:08.747580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:22.029281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489368+0000","last_change":"2023-10-30T13:35:07.790720+0000","last_active":"2023-10-30T13:35:10.489368+0000","last_peered":"2023-10-30T13:35:10.489368+0000","last_clean":"2023-10-30T13:35:10.489368+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:35:10.489368+0000","last_undegraded":"2023-10-30T13:35:10.489368+0000","last_fullsized":"2023-10-30T13:35:10.489368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:16.691133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992429+0000","last_change":"2023-10-30T13:35:07.780576+0000","last_active":"2023-10-30T13:35:10.992429+0000","last_peered":"2023-10-30T13:35:10.992429+0000","last_clean":"2023-10-30T13:35:10.992429+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:35:10.992429+0000","last_undegraded":"2023-10-30T13:35:10.992429+0000","last_fullsized":"2023-10-30T13:35:10.992429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:20.704354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749626+0000","last_change":"2023-10-30T13:35:07.755919+0000","last_active":"2023-10-30T13:35:08.749626+0000","last_peered":"2023-10-30T13:35:08.749626+0000","last_clean":"2023-10-30T13:35:08.749626+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:35:08.749626+0000","last_undegraded":"2023-10-30T13:35:08.749626+0000","last_fullsized":"2023-10-30T13:35:08.749626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:01.978170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.360079+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:35:19.360079+0000","last_peered":"2023-10-30T13:35:19.360079+0000","last_clean":"2023-10-30T13:35:19.360079+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:35:19.360079+0000","last_undegraded":"2023-10-30T13:35:19.360079+0000","last_fullsized":"2023-10-30T13:35:19.360079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":880,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298837+0000","last_change":"2023-10-30T13:35:07.762428+0000","last_active":"2023-10-30T13:35:09.298837+0000","last_peered":"2023-10-30T13:35:09.298837+0000","last_clean":"2023-10-30T13:35:09.298837+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:35:09.298837+0000","last_undegraded":"2023-10-30T13:35:09.298837+0000","last_fullsized":"2023-10-30T13:35:09.298837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:14.191396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804153+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:35:24.804153+0000","last_peered":"2023-10-30T13:35:24.804153+0000","last_clean":"2023-10-30T13:35:24.804153+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:35:24.804153+0000","last_undegraded":"2023-10-30T13:35:24.804153+0000","last_fullsized":"2023-10-30T13:35:24.804153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.803715+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:35:24.803715+0000","last_peered":"2023-10-30T13:35:24.803715+0000","last_clean":"2023-10-30T13:35:24.803715+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:35:24.803715+0000","last_undegraded":"2023-10-30T13:35:24.803715+0000","last_fullsized":"2023-10-30T13:35:24.803715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531211+0000","last_change":"2023-10-30T13:35:07.751131+0000","last_active":"2023-10-30T13:35:09.531211+0000","last_peered":"2023-10-30T13:35:09.531211+0000","last_clean":"2023-10-30T13:35:09.531211+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:35:09.531211+0000","last_undegraded":"2023-10-30T13:35:09.531211+0000","last_fullsized":"2023-10-30T13:35:09.531211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:55:46.387299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748359+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:35:08.748359+0000","last_peered":"2023-10-30T13:35:08.748359+0000","last_clean":"2023-10-30T13:35:08.748359+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:35:08.748359+0000","last_undegraded":"2023-10-30T13:35:08.748359+0000","last_fullsized":"2023-10-30T13:35:08.748359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743438+0000","last_change":"2023-10-30T13:35:07.750392+0000","last_active":"2023-10-30T13:35:08.743438+0000","last_peered":"2023-10-30T13:35:08.743438+0000","last_clean":"2023-10-30T13:35:08.743438+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:35:08.743438+0000","last_undegraded":"2023-10-30T13:35:08.743438+0000","last_fullsized":"2023-10-30T13:35:08.743438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:33.554754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.359410+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:35:19.359410+0000","last_peered":"2023-10-30T13:35:19.359410+0000","last_clean":"2023-10-30T13:35:19.359410+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:35:19.359410+0000","last_undegraded":"2023-10-30T13:35:19.359410+0000","last_fullsized":"2023-10-30T13:35:19.359410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749206+0000","last_change":"2023-10-30T13:35:07.797513+0000","last_active":"2023-10-30T13:35:08.749206+0000","last_peered":"2023-10-30T13:35:08.749206+0000","last_clean":"2023-10-30T13:35:08.749206+0000","last_became_active":"2023-10-30T13:35:07.796748+0000","last_became_peered":"2023-10-30T13:35:07.796748+0000","last_unstale":"2023-10-30T13:35:08.749206+0000","last_undegraded":"2023-10-30T13:35:08.749206+0000","last_fullsized":"2023-10-30T13:35:08.749206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:41.838836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.351941+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:35:19.351941+0000","last_peered":"2023-10-30T13:35:19.351941+0000","last_clean":"2023-10-30T13:35:19.351941+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:35:19.351941+0000","last_undegraded":"2023-10-30T13:35:19.351941+0000","last_fullsized":"2023-10-30T13:35:19.351941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298743+0000","last_change":"2023-10-30T13:35:07.789983+0000","last_active":"2023-10-30T13:35:09.298743+0000","last_peered":"2023-10-30T13:35:09.298743+0000","last_clean":"2023-10-30T13:35:09.298743+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:35:09.298743+0000","last_undegraded":"2023-10-30T13:35:09.298743+0000","last_fullsized":"2023-10-30T13:35:09.298743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:03.304499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.290855+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:35:13.290855+0000","last_peered":"2023-10-30T13:35:13.290855+0000","last_clean":"2023-10-30T13:35:13.290855+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:35:13.290855+0000","last_undegraded":"2023-10-30T13:35:13.290855+0000","last_fullsized":"2023-10-30T13:35:13.290855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750279+0000","last_change":"2023-10-30T13:35:07.794930+0000","last_active":"2023-10-30T13:35:08.750279+0000","last_peered":"2023-10-30T13:35:08.750279+0000","last_clean":"2023-10-30T13:35:08.750279+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:35:08.750279+0000","last_undegraded":"2023-10-30T13:35:08.750279+0000","last_fullsized":"2023-10-30T13:35:08.750279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:19.259009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:32.855435+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:35:32.855435+0000","last_peered":"2023-10-30T13:35:32.855435+0000","last_clean":"2023-10-30T13:35:32.855435+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:35:32.855435+0000","last_undegraded":"2023-10-30T13:35:32.855435+0000","last_fullsized":"2023-10-30T13:35:32.855435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1870,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993842+0000","last_change":"2023-10-30T13:35:07.791923+0000","last_active":"2023-10-30T13:35:10.993842+0000","last_peered":"2023-10-30T13:35:10.993842+0000","last_clean":"2023-10-30T13:35:10.993842+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:35:10.993842+0000","last_undegraded":"2023-10-30T13:35:10.993842+0000","last_fullsized":"2023-10-30T13:35:10.993842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:33.080768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.801219+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:35:24.801219+0000","last_peered":"2023-10-30T13:35:24.801219+0000","last_clean":"2023-10-30T13:35:24.801219+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:35:24.801219+0000","last_undegraded":"2023-10-30T13:35:24.801219+0000","last_fullsized":"2023-10-30T13:35:24.801219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298364+0000","last_change":"2023-10-30T13:35:07.794790+0000","last_active":"2023-10-30T13:35:09.298364+0000","last_peered":"2023-10-30T13:35:09.298364+0000","last_clean":"2023-10-30T13:35:09.298364+0000","last_became_active":"2023-10-30T13:35:07.794477+0000","last_became_peered":"2023-10-30T13:35:07.794477+0000","last_unstale":"2023-10-30T13:35:09.298364+0000","last_undegraded":"2023-10-30T13:35:09.298364+0000","last_fullsized":"2023-10-30T13:35:09.298364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:45.565940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.354274+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:35:19.354274+0000","last_peered":"2023-10-30T13:35:19.354274+0000","last_clean":"2023-10-30T13:35:19.354274+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:35:19.354274+0000","last_undegraded":"2023-10-30T13:35:19.354274+0000","last_fullsized":"2023-10-30T13:35:19.354274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993612+0000","last_change":"2023-10-30T13:35:07.797247+0000","last_active":"2023-10-30T13:35:10.993612+0000","last_peered":"2023-10-30T13:35:10.993612+0000","last_clean":"2023-10-30T13:35:10.993612+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:35:10.993612+0000","last_undegraded":"2023-10-30T13:35:10.993612+0000","last_fullsized":"2023-10-30T13:35:10.993612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:57:02.556658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.803960+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:35:24.803960+0000","last_peered":"2023-10-30T13:35:24.803960+0000","last_clean":"2023-10-30T13:35:24.803960+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:35:24.803960+0000","last_undegraded":"2023-10-30T13:35:24.803960+0000","last_fullsized":"2023-10-30T13:35:24.803960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993402+0000","last_change":"2023-10-30T13:35:07.773683+0000","last_active":"2023-10-30T13:35:10.993402+0000","last_peered":"2023-10-30T13:35:10.993402+0000","last_clean":"2023-10-30T13:35:10.993402+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:35:10.993402+0000","last_undegraded":"2023-10-30T13:35:10.993402+0000","last_fullsized":"2023-10-30T13:35:10.993402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:30:03.553514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804386+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:35:24.804386+0000","last_peered":"2023-10-30T13:35:24.804386+0000","last_clean":"2023-10-30T13:35:24.804386+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:35:24.804386+0000","last_undegraded":"2023-10-30T13:35:24.804386+0000","last_fullsized":"2023-10-30T13:35:24.804386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993760+0000","last_change":"2023-10-30T13:35:07.791266+0000","last_active":"2023-10-30T13:35:10.993760+0000","last_peered":"2023-10-30T13:35:10.993760+0000","last_clean":"2023-10-30T13:35:10.993760+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:35:10.993760+0000","last_undegraded":"2023-10-30T13:35:10.993760+0000","last_fullsized":"2023-10-30T13:35:10.993760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:20.086559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291409+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:35:13.291409+0000","last_peered":"2023-10-30T13:35:13.291409+0000","last_clean":"2023-10-30T13:35:13.291409+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:35:13.291409+0000","last_undegraded":"2023-10-30T13:35:13.291409+0000","last_fullsized":"2023-10-30T13:35:13.291409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994310+0000","last_change":"2023-10-30T13:35:07.794747+0000","last_active":"2023-10-30T13:35:10.994310+0000","last_peered":"2023-10-30T13:35:10.994310+0000","last_clean":"2023-10-30T13:35:10.994310+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:35:10.994310+0000","last_undegraded":"2023-10-30T13:35:10.994310+0000","last_fullsized":"2023-10-30T13:35:10.994310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:40.026054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.287725+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:35:13.287725+0000","last_peered":"2023-10-30T13:35:13.287725+0000","last_clean":"2023-10-30T13:35:13.287725+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:35:13.287725+0000","last_undegraded":"2023-10-30T13:35:13.287725+0000","last_fullsized":"2023-10-30T13:35:13.287725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994800+0000","last_change":"2023-10-30T13:35:07.793255+0000","last_active":"2023-10-30T13:35:10.994800+0000","last_peered":"2023-10-30T13:35:10.994800+0000","last_clean":"2023-10-30T13:35:10.994800+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:35:10.994800+0000","last_undegraded":"2023-10-30T13:35:10.994800+0000","last_fullsized":"2023-10-30T13:35:10.994800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:56.491002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236788+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:35:09.236788+0000","last_peered":"2023-10-30T13:35:09.236788+0000","last_clean":"2023-10-30T13:35:09.236788+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:35:09.236788+0000","last_undegraded":"2023-10-30T13:35:09.236788+0000","last_fullsized":"2023-10-30T13:35:09.236788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748659+0000","last_change":"2023-10-30T13:35:07.791920+0000","last_active":"2023-10-30T13:35:08.748659+0000","last_peered":"2023-10-30T13:35:08.748659+0000","last_clean":"2023-10-30T13:35:08.748659+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:35:08.748659+0000","last_undegraded":"2023-10-30T13:35:08.748659+0000","last_fullsized":"2023-10-30T13:35:08.748659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:04.379349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.800093+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:35:24.800093+0000","last_peered":"2023-10-30T13:35:24.800093+0000","last_clean":"2023-10-30T13:35:24.800093+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:35:24.800093+0000","last_undegraded":"2023-10-30T13:35:24.800093+0000","last_fullsized":"2023-10-30T13:35:24.800093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994214+0000","last_change":"2023-10-30T13:35:07.803211+0000","last_active":"2023-10-30T13:35:10.994214+0000","last_peered":"2023-10-30T13:35:10.994214+0000","last_clean":"2023-10-30T13:35:10.994214+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:35:10.994214+0000","last_undegraded":"2023-10-30T13:35:10.994214+0000","last_fullsized":"2023-10-30T13:35:10.994214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:55.371993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768067+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:35:07.768067+0000","last_peered":"2023-10-30T13:35:07.768067+0000","last_clean":"2023-10-30T13:35:07.768067+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:35:07.768067+0000","last_undegraded":"2023-10-30T13:35:07.768067+0000","last_fullsized":"2023-10-30T13:35:07.768067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.353148+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:35:19.353148+0000","last_peered":"2023-10-30T13:35:19.353148+0000","last_clean":"2023-10-30T13:35:19.353148+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:35:19.353148+0000","last_undegraded":"2023-10-30T13:35:19.353148+0000","last_fullsized":"2023-10-30T13:35:19.353148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531304+0000","last_change":"2023-10-30T13:35:07.792082+0000","last_active":"2023-10-30T13:35:09.531304+0000","last_peered":"2023-10-30T13:35:09.531304+0000","last_clean":"2023-10-30T13:35:09.531304+0000","last_became_active":"2023-10-30T13:35:07.791831+0000","last_became_peered":"2023-10-30T13:35:07.791831+0000","last_unstale":"2023-10-30T13:35:09.531304+0000","last_undegraded":"2023-10-30T13:35:09.531304+0000","last_fullsized":"2023-10-30T13:35:09.531304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:47.863235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799794+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:35:24.799794+0000","last_peered":"2023-10-30T13:35:24.799794+0000","last_clean":"2023-10-30T13:35:24.799794+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:35:24.799794+0000","last_undegraded":"2023-10-30T13:35:24.799794+0000","last_fullsized":"2023-10-30T13:35:24.799794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298228+0000","last_change":"2023-10-30T13:35:07.760132+0000","last_active":"2023-10-30T13:35:09.298228+0000","last_peered":"2023-10-30T13:35:09.298228+0000","last_clean":"2023-10-30T13:35:09.298228+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:35:09.298228+0000","last_undegraded":"2023-10-30T13:35:09.298228+0000","last_fullsized":"2023-10-30T13:35:09.298228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:14.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799584+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:35:24.799584+0000","last_peered":"2023-10-30T13:35:24.799584+0000","last_clean":"2023-10-30T13:35:24.799584+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:35:24.799584+0000","last_undegraded":"2023-10-30T13:35:24.799584+0000","last_fullsized":"2023-10-30T13:35:24.799584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.750958+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:53.697186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747960+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:35:08.747960+0000","last_peered":"2023-10-30T13:35:08.747960+0000","last_clean":"2023-10-30T13:35:08.747960+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:35:08.747960+0000","last_undegraded":"2023-10-30T13:35:08.747960+0000","last_fullsized":"2023-10-30T13:35:08.747960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750179+0000","last_change":"2023-10-30T13:35:07.768640+0000","last_active":"2023-10-30T13:35:08.750179+0000","last_peered":"2023-10-30T13:35:08.750179+0000","last_clean":"2023-10-30T13:35:08.750179+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:35:08.750179+0000","last_undegraded":"2023-10-30T13:35:08.750179+0000","last_fullsized":"2023-10-30T13:35:08.750179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:26.852057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743612+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:35:08.743612+0000","last_peered":"2023-10-30T13:35:08.743612+0000","last_clean":"2023-10-30T13:35:08.743612+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:35:08.743612+0000","last_undegraded":"2023-10-30T13:35:08.743612+0000","last_fullsized":"2023-10-30T13:35:08.743612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749093+0000","last_change":"2023-10-30T13:35:07.805535+0000","last_active":"2023-10-30T13:35:08.749093+0000","last_peered":"2023-10-30T13:35:08.749093+0000","last_clean":"2023-10-30T13:35:08.749093+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:35:08.749093+0000","last_undegraded":"2023-10-30T13:35:08.749093+0000","last_fullsized":"2023-10-30T13:35:08.749093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:06.549889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994784+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:35:10.994784+0000","last_peered":"2023-10-30T13:35:10.994784+0000","last_clean":"2023-10-30T13:35:10.994784+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:35:10.994784+0000","last_undegraded":"2023-10-30T13:35:10.994784+0000","last_fullsized":"2023-10-30T13:35:10.994784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743474+0000","last_change":"2023-10-30T13:35:07.754171+0000","last_active":"2023-10-30T13:35:08.743474+0000","last_peered":"2023-10-30T13:35:08.743474+0000","last_clean":"2023-10-30T13:35:08.743474+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:35:08.743474+0000","last_undegraded":"2023-10-30T13:35:08.743474+0000","last_fullsized":"2023-10-30T13:35:08.743474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:50.849695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.801594+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:35:24.801594+0000","last_peered":"2023-10-30T13:35:24.801594+0000","last_clean":"2023-10-30T13:35:24.801594+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:35:24.801594+0000","last_undegraded":"2023-10-30T13:35:24.801594+0000","last_fullsized":"2023-10-30T13:35:24.801594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531754+0000","last_change":"2023-10-30T13:35:07.794804+0000","last_active":"2023-10-30T13:35:09.531754+0000","last_peered":"2023-10-30T13:35:09.531754+0000","last_clean":"2023-10-30T13:35:09.531754+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:35:09.531754+0000","last_undegraded":"2023-10-30T13:35:09.531754+0000","last_fullsized":"2023-10-30T13:35:09.531754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:00.913592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.355614+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:35:19.355614+0000","last_peered":"2023-10-30T13:35:19.355614+0000","last_clean":"2023-10-30T13:35:19.355614+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:35:19.355614+0000","last_undegraded":"2023-10-30T13:35:19.355614+0000","last_fullsized":"2023-10-30T13:35:19.355614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298809+0000","last_change":"2023-10-30T13:35:07.752077+0000","last_active":"2023-10-30T13:35:09.298809+0000","last_peered":"2023-10-30T13:35:09.298809+0000","last_clean":"2023-10-30T13:35:09.298809+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:35:09.298809+0000","last_undegraded":"2023-10-30T13:35:09.298809+0000","last_fullsized":"2023-10-30T13:35:09.298809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:07.661975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.350893+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:35:19.350893+0000","last_peered":"2023-10-30T13:35:19.350893+0000","last_clean":"2023-10-30T13:35:19.350893+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:35:19.350893+0000","last_undegraded":"2023-10-30T13:35:19.350893+0000","last_fullsized":"2023-10-30T13:35:19.350893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748158+0000","last_change":"2023-10-30T13:35:07.759389+0000","last_active":"2023-10-30T13:35:08.748158+0000","last_peered":"2023-10-30T13:35:08.748158+0000","last_clean":"2023-10-30T13:35:08.748158+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:35:08.748158+0000","last_undegraded":"2023-10-30T13:35:08.748158+0000","last_fullsized":"2023-10-30T13:35:08.748158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:15.536615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.351639+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:35:19.351639+0000","last_peered":"2023-10-30T13:35:19.351639+0000","last_clean":"2023-10-30T13:35:19.351639+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:35:19.351639+0000","last_undegraded":"2023-10-30T13:35:19.351639+0000","last_fullsized":"2023-10-30T13:35:19.351639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531288+0000","last_change":"2023-10-30T13:35:07.753673+0000","last_active":"2023-10-30T13:35:09.531288+0000","last_peered":"2023-10-30T13:35:09.531288+0000","last_clean":"2023-10-30T13:35:09.531288+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:35:09.531288+0000","last_undegraded":"2023-10-30T13:35:09.531288+0000","last_fullsized":"2023-10-30T13:35:09.531288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:27:18.430183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236727+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:35:09.236727+0000","last_peered":"2023-10-30T13:35:09.236727+0000","last_clean":"2023-10-30T13:35:09.236727+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:35:09.236727+0000","last_undegraded":"2023-10-30T13:35:09.236727+0000","last_fullsized":"2023-10-30T13:35:09.236727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531465+0000","last_change":"2023-10-30T13:35:07.755302+0000","last_active":"2023-10-30T13:35:09.531465+0000","last_peered":"2023-10-30T13:35:09.531465+0000","last_clean":"2023-10-30T13:35:09.531465+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:35:09.531465+0000","last_undegraded":"2023-10-30T13:35:09.531465+0000","last_fullsized":"2023-10-30T13:35:09.531465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:31.543587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.352360+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:35:19.352360+0000","last_peered":"2023-10-30T13:35:19.352360+0000","last_clean":"2023-10-30T13:35:19.352360+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:35:19.352360+0000","last_undegraded":"2023-10-30T13:35:19.352360+0000","last_fullsized":"2023-10-30T13:35:19.352360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992766+0000","last_change":"2023-10-30T13:35:07.780297+0000","last_active":"2023-10-30T13:35:10.992766+0000","last_peered":"2023-10-30T13:35:10.992766+0000","last_clean":"2023-10-30T13:35:10.992766+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:35:10.992766+0000","last_undegraded":"2023-10-30T13:35:10.992766+0000","last_fullsized":"2023-10-30T13:35:10.992766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:48.971885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748946+0000","last_change":"2023-10-30T13:35:07.806203+0000","last_active":"2023-10-30T13:35:08.748946+0000","last_peered":"2023-10-30T13:35:08.748946+0000","last_clean":"2023-10-30T13:35:08.748946+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:35:08.748946+0000","last_undegraded":"2023-10-30T13:35:08.748946+0000","last_fullsized":"2023-10-30T13:35:08.748946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:34.175218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294221+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:35:13.294221+0000","last_peered":"2023-10-30T13:35:13.294221+0000","last_clean":"2023-10-30T13:35:13.294221+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:35:13.294221+0000","last_undegraded":"2023-10-30T13:35:13.294221+0000","last_fullsized":"2023-10-30T13:35:13.294221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748564+0000","last_change":"2023-10-30T13:35:07.791319+0000","last_active":"2023-10-30T13:35:08.748564+0000","last_peered":"2023-10-30T13:35:08.748564+0000","last_clean":"2023-10-30T13:35:08.748564+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:35:08.748564+0000","last_undegraded":"2023-10-30T13:35:08.748564+0000","last_fullsized":"2023-10-30T13:35:08.748564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:42.560856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799715+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:35:24.799715+0000","last_peered":"2023-10-30T13:35:24.799715+0000","last_clean":"2023-10-30T13:35:24.799715+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:35:24.799715+0000","last_undegraded":"2023-10-30T13:35:24.799715+0000","last_fullsized":"2023-10-30T13:35:24.799715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298499+0000","last_change":"2023-10-30T13:35:07.790084+0000","last_active":"2023-10-30T13:35:09.298499+0000","last_peered":"2023-10-30T13:35:09.298499+0000","last_clean":"2023-10-30T13:35:09.298499+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:35:09.298499+0000","last_undegraded":"2023-10-30T13:35:09.298499+0000","last_fullsized":"2023-10-30T13:35:09.298499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:45.193072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291592+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:35:13.291592+0000","last_peered":"2023-10-30T13:35:13.291592+0000","last_clean":"2023-10-30T13:35:13.291592+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:35:13.291592+0000","last_undegraded":"2023-10-30T13:35:13.291592+0000","last_fullsized":"2023-10-30T13:35:13.291592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531695+0000","last_change":"2023-10-30T13:35:07.782341+0000","last_active":"2023-10-30T13:35:09.531695+0000","last_peered":"2023-10-30T13:35:09.531695+0000","last_clean":"2023-10-30T13:35:09.531695+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:35:09.531695+0000","last_undegraded":"2023-10-30T13:35:09.531695+0000","last_fullsized":"2023-10-30T13:35:09.531695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:19:06.392683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804103+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:35:24.804103+0000","last_peered":"2023-10-30T13:35:24.804103+0000","last_clean":"2023-10-30T13:35:24.804103+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:35:24.804103+0000","last_undegraded":"2023-10-30T13:35:24.804103+0000","last_fullsized":"2023-10-30T13:35:24.804103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5738,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":664793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509452,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":75691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509452,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":73669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":73795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]}]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":74229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]}]},{"osd":4,"up_from":14,"seq":60129542155,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":73565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":663879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:35:36.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa428f3e640 1 Processor -- start 2023-10-30T13:35:36.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa428f3e640 1 -- start start 2023-10-30T13:35:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa428f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa424153c80 0x7fa42415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa428f3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa4240c4a10 0x7fa4240c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa428f3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa424159f20 con 0x7fa42415a3b0 2023-10-30T13:35:36.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa428f3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42415dec0 con 0x7fa4240c4a10 2023-10-30T13:35:36.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa428f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4241599a0 con 0x7fa424153c80 2023-10-30T13:35:36.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa421d9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa4240c4a10 0x7fa4240c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa421d9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa4240c4a10 0x7fa4240c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51350/0 (socket says 172.21.15.31:51350) 2023-10-30T13:35:36.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa421d9b640 1 -- 172.21.15.31:0/4082137536 learned_addr learned my addr 172.21.15.31:0/4082137536 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa42259c640 1 --2- 172.21.15.31:0/4082137536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa424153c80 0x7fa42415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa421d9b640 1 -- 172.21.15.31:0/4082137536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa424153c80 msgr2=0x7fa42415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa421d9b640 1 --2- 172.21.15.31:0/4082137536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa424153c80 0x7fa42415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa421d9b640 1 -- 172.21.15.31:0/4082137536 >> v1:172.21.15.33:6789/0 conn(0x7fa42415a3b0 legacy=0x7fa4240ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.920+0000 7fa421d9b640 1 -- 172.21.15.31:0/4082137536 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4240c47e0 con 0x7fa4240c4a10 2023-10-30T13:35:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa421d9b640 1 --2- 172.21.15.31:0/4082137536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4240c4a10 0x7fa4240c42a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa418009870 tx=0x7fa41801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e42df6b97e1736 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa42159a640 1 -- 172.21.15.31:0/4082137536 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4180a8a30 con 0x7fa4240c4a10 2023-10-30T13:35:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa42159a640 1 -- 172.21.15.31:0/4082137536 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4180a8b90 con 0x7fa4240c4a10 2023-10-30T13:35:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa42159a640 1 -- 172.21.15.31:0/4082137536 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4180b0440 con 0x7fa4240c4a10 2023-10-30T13:35:36.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 -- 172.21.15.31:0/4082137536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4240c4a10 msgr2=0x7fa4240c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 --2- 172.21.15.31:0/4082137536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4240c4a10 0x7fa4240c42a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa418009870 tx=0x7fa41801ed30 comp rx=0 tx=0).stop 2023-10-30T13:35:36.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 -- 172.21.15.31:0/4082137536 shutdown_connections 2023-10-30T13:35:36.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 --2- 172.21.15.31:0/4082137536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4240c4a10 0x7fa4240c42a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 --2- 172.21.15.31:0/4082137536 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa424153c80 0x7fa42415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 -- 172.21.15.31:0/4082137536 >> 172.21.15.31:0/4082137536 conn(0x7fa4240715c0 msgr2=0x7fa4240719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:36.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 -- 172.21.15.31:0/4082137536 shutdown_connections 2023-10-30T13:35:36.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.924+0000 7fa428f3e640 1 -- 172.21.15.31:0/4082137536 wait complete. 2023-10-30T13:35:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 Processor -- start 2023-10-30T13:35:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 -- start start 2023-10-30T13:35:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4240c4a10 0x7fa4241ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424153c80 0x7fa42415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa42415a3b0 0x7fa42415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4241eef10 con 0x7fa424153c80 2023-10-30T13:35:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa424151d10 con 0x7fa4240c4a10 2023-10-30T13:35:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa424151ff0 con 0x7fa42415a3b0 2023-10-30T13:35:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424153c80 0x7fa42415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424153c80 0x7fa42415ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51360/0 (socket says 172.21.15.31:51360) 2023-10-30T13:35:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 -- 172.21.15.31:0/1192378572 learned_addr learned my addr 172.21.15.31:0/1192378572 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa422d9d640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa42415a3b0 0x7fa42415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa42259c640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4240c4a10 0x7fa4241ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 -- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa42415a3b0 msgr2=0x7fa42415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa42415a3b0 0x7fa42415ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 -- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4240c4a10 msgr2=0x7fa4241ef3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4240c4a10 0x7fa4241ef3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 -- 172.21.15.31:0/1192378572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4180094d0 con 0x7fa424153c80 2023-10-30T13:35:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa421d9b640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424153c80 0x7fa42415ab60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa4180097c0 tx=0x7fa41801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa412ffd640 1 -- 172.21.15.31:0/1192378572 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4180bf070 con 0x7fa424153c80 2023-10-30T13:35:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa412ffd640 1 -- 172.21.15.31:0/1192378572 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4180b1c10 con 0x7fa424153c80 2023-10-30T13:35:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa412ffd640 1 -- 172.21.15.31:0/1192378572 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4180b9410 con 0x7fa424153c80 2023-10-30T13:35:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4241ee8c0 con 0x7fa424153c80 2023-10-30T13:35:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.928+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4241522d0 con 0x7fa424153c80 2023-10-30T13:35:36.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.932+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4241ef080 con 0x7fa424153c80 2023-10-30T13:35:36.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.932+0000 7fa412ffd640 1 -- 172.21.15.31:0/1192378572 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4180b7040 con 0x7fa424153c80 2023-10-30T13:35:36.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.932+0000 7fa412ffd640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4040664a0 0x7fa404068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:36.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.932+0000 7fa412ffd640 1 -- 172.21.15.31:0/1192378572 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa418121300 con 0x7fa424153c80 2023-10-30T13:35:36.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.932+0000 7fa42259c640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4040664a0 0x7fa404068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:36.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.932+0000 7fa42259c640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4040664a0 0x7fa404068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa41400b9d0 tx=0x7fa414008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:36.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:36.936+0000 7fa412ffd640 1 -- 172.21.15.31:0/1192378572 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4180ee650 con 0x7fa424153c80 2023-10-30T13:35:37.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.088+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4241eed70 con 0x7fa4040664a0 2023-10-30T13:35:37.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.108+0000 7fa412ffd640 1 -- 172.21.15.31:0/1192378572 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264699 (secure 0 0 0) 0x7fa41400b670 con 0x7fa4040664a0 2023-10-30T13:35:37.116 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:37.118 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:35:37.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4040664a0 msgr2=0x7fa404068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4040664a0 0x7fa404068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa41400b9d0 tx=0x7fa414008040 comp rx=0 tx=0).stop 2023-10-30T13:35:37.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424153c80 msgr2=0x7fa42415ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424153c80 0x7fa42415ab60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa4180097c0 tx=0x7fa41801bad0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 shutdown_connections 2023-10-30T13:35:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4040664a0 0x7fa404068960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa42415a3b0 0x7fa42415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424153c80 0x7fa42415ab60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 --2- 172.21.15.31:0/1192378572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4240c4a10 0x7fa4241ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 >> 172.21.15.31:0/1192378572 conn(0x7fa4240715c0 msgr2=0x7fa4240cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 shutdown_connections 2023-10-30T13:35:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.112+0000 7fa428f3e640 1 -- 172.21.15.31:0/1192378572 wait complete. 2023-10-30T13:35:37.137 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T13:35:35.166199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596106,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":287,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219080,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755640,"statfs":{"total":773094113280,"available":772869775360,"internally_reserved":0,"allocated":3022848,"data_stored":1775020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3328,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005026"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994380+0000","last_change":"2023-10-30T13:35:07.792742+0000","last_active":"2023-10-30T13:35:10.994380+0000","last_peered":"2023-10-30T13:35:10.994380+0000","last_clean":"2023-10-30T13:35:10.994380+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:35:10.994380+0000","last_undegraded":"2023-10-30T13:35:10.994380+0000","last_fullsized":"2023-10-30T13:35:10.994380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:02.805042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744019+0000","last_change":"2023-10-30T13:35:07.755187+0000","last_active":"2023-10-30T13:35:08.744019+0000","last_peered":"2023-10-30T13:35:08.744019+0000","last_clean":"2023-10-30T13:35:08.744019+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:35:08.744019+0000","last_undegraded":"2023-10-30T13:35:08.744019+0000","last_fullsized":"2023-10-30T13:35:08.744019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:22.764808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298537+0000","last_change":"2023-10-30T13:35:07.754357+0000","last_active":"2023-10-30T13:35:09.298537+0000","last_peered":"2023-10-30T13:35:09.298537+0000","last_clean":"2023-10-30T13:35:09.298537+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:35:09.298537+0000","last_undegraded":"2023-10-30T13:35:09.298537+0000","last_fullsized":"2023-10-30T13:35:09.298537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:23.201416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748146+0000","last_change":"2023-10-30T13:35:07.792466+0000","last_active":"2023-10-30T13:35:08.748146+0000","last_peered":"2023-10-30T13:35:08.748146+0000","last_clean":"2023-10-30T13:35:08.748146+0000","last_became_active":"2023-10-30T13:35:07.789749+0000","last_became_peered":"2023-10-30T13:35:07.789749+0000","last_unstale":"2023-10-30T13:35:08.748146+0000","last_undegraded":"2023-10-30T13:35:08.748146+0000","last_fullsized":"2023-10-30T13:35:08.748146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:46.527032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993635+0000","last_change":"2023-10-30T13:35:07.793403+0000","last_active":"2023-10-30T13:35:10.993635+0000","last_peered":"2023-10-30T13:35:10.993635+0000","last_clean":"2023-10-30T13:35:10.993635+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:35:10.993635+0000","last_undegraded":"2023-10-30T13:35:10.993635+0000","last_fullsized":"2023-10-30T13:35:10.993635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:27.304652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747210+0000","last_change":"2023-10-30T13:35:07.792285+0000","last_active":"2023-10-30T13:35:08.747210+0000","last_peered":"2023-10-30T13:35:08.747210+0000","last_clean":"2023-10-30T13:35:08.747210+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:35:08.747210+0000","last_undegraded":"2023-10-30T13:35:08.747210+0000","last_fullsized":"2023-10-30T13:35:08.747210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:31.537183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993398+0000","last_change":"2023-10-30T13:35:07.776386+0000","last_active":"2023-10-30T13:35:10.993398+0000","last_peered":"2023-10-30T13:35:10.993398+0000","last_clean":"2023-10-30T13:35:10.993398+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:35:10.993398+0000","last_undegraded":"2023-10-30T13:35:10.993398+0000","last_fullsized":"2023-10-30T13:35:10.993398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:27.492389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531107+0000","last_change":"2023-10-30T13:35:07.790487+0000","last_active":"2023-10-30T13:35:09.531107+0000","last_peered":"2023-10-30T13:35:09.531107+0000","last_clean":"2023-10-30T13:35:09.531107+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:35:09.531107+0000","last_undegraded":"2023-10-30T13:35:09.531107+0000","last_fullsized":"2023-10-30T13:35:09.531107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:27.384592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994435+0000","last_change":"2023-10-30T13:35:07.804038+0000","last_active":"2023-10-30T13:35:10.994435+0000","last_peered":"2023-10-30T13:35:10.994435+0000","last_clean":"2023-10-30T13:35:10.994435+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:35:10.994435+0000","last_undegraded":"2023-10-30T13:35:10.994435+0000","last_fullsized":"2023-10-30T13:35:10.994435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:09.137404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993722+0000","last_change":"2023-10-30T13:35:07.800657+0000","last_active":"2023-10-30T13:35:10.993722+0000","last_peered":"2023-10-30T13:35:10.993722+0000","last_clean":"2023-10-30T13:35:10.993722+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:35:10.993722+0000","last_undegraded":"2023-10-30T13:35:10.993722+0000","last_fullsized":"2023-10-30T13:35:10.993722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:12.080911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993171+0000","last_change":"2023-10-30T13:35:07.802245+0000","last_active":"2023-10-30T13:35:10.993171+0000","last_peered":"2023-10-30T13:35:10.993171+0000","last_clean":"2023-10-30T13:35:10.993171+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:35:10.993171+0000","last_undegraded":"2023-10-30T13:35:10.993171+0000","last_fullsized":"2023-10-30T13:35:10.993171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:58.462586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750176+0000","last_change":"2023-10-30T13:35:07.783940+0000","last_active":"2023-10-30T13:35:08.750176+0000","last_peered":"2023-10-30T13:35:08.750176+0000","last_clean":"2023-10-30T13:35:08.750176+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:35:08.750176+0000","last_undegraded":"2023-10-30T13:35:08.750176+0000","last_fullsized":"2023-10-30T13:35:08.750176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:07.873653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531433+0000","last_change":"2023-10-30T13:35:07.780349+0000","last_active":"2023-10-30T13:35:09.531433+0000","last_peered":"2023-10-30T13:35:09.531433+0000","last_clean":"2023-10-30T13:35:09.531433+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:35:09.531433+0000","last_undegraded":"2023-10-30T13:35:09.531433+0000","last_fullsized":"2023-10-30T13:35:09.531433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:15.619776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298339+0000","last_change":"2023-10-30T13:35:07.750959+0000","last_active":"2023-10-30T13:35:09.298339+0000","last_peered":"2023-10-30T13:35:09.298339+0000","last_clean":"2023-10-30T13:35:09.298339+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:35:09.298339+0000","last_undegraded":"2023-10-30T13:35:09.298339+0000","last_fullsized":"2023-10-30T13:35:09.298339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:56.853163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.768466+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:27.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748523+0000","last_change":"2023-10-30T13:35:07.789918+0000","last_active":"2023-10-30T13:35:08.748523+0000","last_peered":"2023-10-30T13:35:08.748523+0000","last_clean":"2023-10-30T13:35:08.748523+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:35:08.748523+0000","last_undegraded":"2023-10-30T13:35:08.748523+0000","last_fullsized":"2023-10-30T13:35:08.748523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:53.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749638+0000","last_change":"2023-10-30T13:35:07.796429+0000","last_active":"2023-10-30T13:35:08.749638+0000","last_peered":"2023-10-30T13:35:08.749638+0000","last_clean":"2023-10-30T13:35:08.749638+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:35:08.749638+0000","last_undegraded":"2023-10-30T13:35:08.749638+0000","last_fullsized":"2023-10-30T13:35:08.749638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:15.022866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489395+0000","last_change":"2023-10-30T13:35:07.780131+0000","last_active":"2023-10-30T13:35:10.489395+0000","last_peered":"2023-10-30T13:35:10.489395+0000","last_clean":"2023-10-30T13:35:10.489395+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:35:10.489395+0000","last_undegraded":"2023-10-30T13:35:10.489395+0000","last_fullsized":"2023-10-30T13:35:10.489395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:04.046338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994792+0000","last_change":"2023-10-30T13:35:07.791464+0000","last_active":"2023-10-30T13:35:10.994792+0000","last_peered":"2023-10-30T13:35:10.994792+0000","last_clean":"2023-10-30T13:35:10.994792+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:35:10.994792+0000","last_undegraded":"2023-10-30T13:35:10.994792+0000","last_fullsized":"2023-10-30T13:35:10.994792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993329+0000","last_change":"2023-10-30T13:35:07.783079+0000","last_active":"2023-10-30T13:35:10.993329+0000","last_peered":"2023-10-30T13:35:10.993329+0000","last_clean":"2023-10-30T13:35:10.993329+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:35:10.993329+0000","last_undegraded":"2023-10-30T13:35:10.993329+0000","last_fullsized":"2023-10-30T13:35:10.993329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:43:32.360152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298481+0000","last_change":"2023-10-30T13:35:07.794666+0000","last_active":"2023-10-30T13:35:09.298481+0000","last_peered":"2023-10-30T13:35:09.298481+0000","last_clean":"2023-10-30T13:35:09.298481+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:35:09.298481+0000","last_undegraded":"2023-10-30T13:35:09.298481+0000","last_fullsized":"2023-10-30T13:35:09.298481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:52.189022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747346+0000","last_change":"2023-10-30T13:35:07.796601+0000","last_active":"2023-10-30T13:35:08.747346+0000","last_peered":"2023-10-30T13:35:08.747346+0000","last_clean":"2023-10-30T13:35:08.747346+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:35:08.747346+0000","last_undegraded":"2023-10-30T13:35:08.747346+0000","last_fullsized":"2023-10-30T13:35:08.747346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:07.572028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531135+0000","last_change":"2023-10-30T13:35:07.767965+0000","last_active":"2023-10-30T13:35:09.531135+0000","last_peered":"2023-10-30T13:35:09.531135+0000","last_clean":"2023-10-30T13:35:09.531135+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:35:09.531135+0000","last_undegraded":"2023-10-30T13:35:09.531135+0000","last_fullsized":"2023-10-30T13:35:09.531135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:15.544113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748017+0000","last_change":"2023-10-30T13:35:07.772029+0000","last_active":"2023-10-30T13:35:08.748017+0000","last_peered":"2023-10-30T13:35:08.748017+0000","last_clean":"2023-10-30T13:35:08.748017+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:35:08.748017+0000","last_undegraded":"2023-10-30T13:35:08.748017+0000","last_fullsized":"2023-10-30T13:35:08.748017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:13.305147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531716+0000","last_change":"2023-10-30T13:35:07.758274+0000","last_active":"2023-10-30T13:35:09.531716+0000","last_peered":"2023-10-30T13:35:09.531716+0000","last_clean":"2023-10-30T13:35:09.531716+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:35:09.531716+0000","last_undegraded":"2023-10-30T13:35:09.531716+0000","last_fullsized":"2023-10-30T13:35:09.531716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:15.191933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994099+0000","last_change":"2023-10-30T13:35:07.803220+0000","last_active":"2023-10-30T13:35:10.994099+0000","last_peered":"2023-10-30T13:35:10.994099+0000","last_clean":"2023-10-30T13:35:10.994099+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:35:10.994099+0000","last_undegraded":"2023-10-30T13:35:10.994099+0000","last_fullsized":"2023-10-30T13:35:10.994099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:45.165258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743680+0000","last_change":"2023-10-30T13:35:07.753288+0000","last_active":"2023-10-30T13:35:08.743680+0000","last_peered":"2023-10-30T13:35:08.743680+0000","last_clean":"2023-10-30T13:35:08.743680+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:35:08.743680+0000","last_undegraded":"2023-10-30T13:35:08.743680+0000","last_fullsized":"2023-10-30T13:35:08.743680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:43.815379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531511+0000","last_change":"2023-10-30T13:35:07.767931+0000","last_active":"2023-10-30T13:35:09.531511+0000","last_peered":"2023-10-30T13:35:09.531511+0000","last_clean":"2023-10-30T13:35:09.531511+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:35:09.531511+0000","last_undegraded":"2023-10-30T13:35:09.531511+0000","last_fullsized":"2023-10-30T13:35:09.531511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:43.830918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298592+0000","last_change":"2023-10-30T13:35:07.753695+0000","last_active":"2023-10-30T13:35:09.298592+0000","last_peered":"2023-10-30T13:35:09.298592+0000","last_clean":"2023-10-30T13:35:09.298592+0000","last_became_active":"2023-10-30T13:35:07.753358+0000","last_became_peered":"2023-10-30T13:35:07.753358+0000","last_unstale":"2023-10-30T13:35:09.298592+0000","last_undegraded":"2023-10-30T13:35:09.298592+0000","last_fullsized":"2023-10-30T13:35:09.298592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:12.840727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747580+0000","last_change":"2023-10-30T13:35:07.790236+0000","last_active":"2023-10-30T13:35:08.747580+0000","last_peered":"2023-10-30T13:35:08.747580+0000","last_clean":"2023-10-30T13:35:08.747580+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:35:08.747580+0000","last_undegraded":"2023-10-30T13:35:08.747580+0000","last_fullsized":"2023-10-30T13:35:08.747580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:22.029281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489368+0000","last_change":"2023-10-30T13:35:07.790720+0000","last_active":"2023-10-30T13:35:10.489368+0000","last_peered":"2023-10-30T13:35:10.489368+0000","last_clean":"2023-10-30T13:35:10.489368+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:35:10.489368+0000","last_undegraded":"2023-10-30T13:35:10.489368+0000","last_fullsized":"2023-10-30T13:35:10.489368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:16.691133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992429+0000","last_change":"2023-10-30T13:35:07.780576+0000","last_active":"2023-10-30T13:35:10.992429+0000","last_peered":"2023-10-30T13:35:10.992429+0000","last_clean":"2023-10-30T13:35:10.992429+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:35:10.992429+0000","last_undegraded":"2023-10-30T13:35:10.992429+0000","last_fullsized":"2023-10-30T13:35:10.992429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:20.704354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749626+0000","last_change":"2023-10-30T13:35:07.755919+0000","last_active":"2023-10-30T13:35:08.749626+0000","last_peered":"2023-10-30T13:35:08.749626+0000","last_clean":"2023-10-30T13:35:08.749626+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:35:08.749626+0000","last_undegraded":"2023-10-30T13:35:08.749626+0000","last_fullsized":"2023-10-30T13:35:08.749626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:01.978170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.360079+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:35:19.360079+0000","last_peered":"2023-10-30T13:35:19.360079+0000","last_clean":"2023-10-30T13:35:19.360079+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:35:19.360079+0000","last_undegraded":"2023-10-30T13:35:19.360079+0000","last_fullsized":"2023-10-30T13:35:19.360079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":880,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298837+0000","last_change":"2023-10-30T13:35:07.762428+0000","last_active":"2023-10-30T13:35:09.298837+0000","last_peered":"2023-10-30T13:35:09.298837+0000","last_clean":"2023-10-30T13:35:09.298837+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:35:09.298837+0000","last_undegraded":"2023-10-30T13:35:09.298837+0000","last_fullsized":"2023-10-30T13:35:09.298837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:14.191396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804153+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:35:24.804153+0000","last_peered":"2023-10-30T13:35:24.804153+0000","last_clean":"2023-10-30T13:35:24.804153+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:35:24.804153+0000","last_undegraded":"2023-10-30T13:35:24.804153+0000","last_fullsized":"2023-10-30T13:35:24.804153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.803715+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:35:24.803715+0000","last_peered":"2023-10-30T13:35:24.803715+0000","last_clean":"2023-10-30T13:35:24.803715+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:35:24.803715+0000","last_undegraded":"2023-10-30T13:35:24.803715+0000","last_fullsized":"2023-10-30T13:35:24.803715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531211+0000","last_change":"2023-10-30T13:35:07.751131+0000","last_active":"2023-10-30T13:35:09.531211+0000","last_peered":"2023-10-30T13:35:09.531211+0000","last_clean":"2023-10-30T13:35:09.531211+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:35:09.531211+0000","last_undegraded":"2023-10-30T13:35:09.531211+0000","last_fullsized":"2023-10-30T13:35:09.531211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:55:46.387299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748359+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:35:08.748359+0000","last_peered":"2023-10-30T13:35:08.748359+0000","last_clean":"2023-10-30T13:35:08.748359+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:35:08.748359+0000","last_undegraded":"2023-10-30T13:35:08.748359+0000","last_fullsized":"2023-10-30T13:35:08.748359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743438+0000","last_change":"2023-10-30T13:35:07.750392+0000","last_active":"2023-10-30T13:35:08.743438+0000","last_peered":"2023-10-30T13:35:08.743438+0000","last_clean":"2023-10-30T13:35:08.743438+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:35:08.743438+0000","last_undegraded":"2023-10-30T13:35:08.743438+0000","last_fullsized":"2023-10-30T13:35:08.743438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:33.554754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.359410+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:35:19.359410+0000","last_peered":"2023-10-30T13:35:19.359410+0000","last_clean":"2023-10-30T13:35:19.359410+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:35:19.359410+0000","last_undegraded":"2023-10-30T13:35:19.359410+0000","last_fullsized":"2023-10-30T13:35:19.359410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749206+0000","last_change":"2023-10-30T13:35:07.797513+0000","last_active":"2023-10-30T13:35:08.749206+0000","last_peered":"2023-10-30T13:35:08.749206+0000","last_clean":"2023-10-30T13:35:08.749206+0000","last_became_active":"2023-10-30T13:35:07.796748+0000","last_became_peered":"2023-10-30T13:35:07.796748+0000","last_unstale":"2023-10-30T13:35:08.749206+0000","last_undegraded":"2023-10-30T13:35:08.749206+0000","last_fullsized":"2023-10-30T13:35:08.749206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:41.838836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.351941+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:35:19.351941+0000","last_peered":"2023-10-30T13:35:19.351941+0000","last_clean":"2023-10-30T13:35:19.351941+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:35:19.351941+0000","last_undegraded":"2023-10-30T13:35:19.351941+0000","last_fullsized":"2023-10-30T13:35:19.351941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298743+0000","last_change":"2023-10-30T13:35:07.789983+0000","last_active":"2023-10-30T13:35:09.298743+0000","last_peered":"2023-10-30T13:35:09.298743+0000","last_clean":"2023-10-30T13:35:09.298743+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:35:09.298743+0000","last_undegraded":"2023-10-30T13:35:09.298743+0000","last_fullsized":"2023-10-30T13:35:09.298743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:03.304499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.290855+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:35:13.290855+0000","last_peered":"2023-10-30T13:35:13.290855+0000","last_clean":"2023-10-30T13:35:13.290855+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:35:13.290855+0000","last_undegraded":"2023-10-30T13:35:13.290855+0000","last_fullsized":"2023-10-30T13:35:13.290855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750279+0000","last_change":"2023-10-30T13:35:07.794930+0000","last_active":"2023-10-30T13:35:08.750279+0000","last_peered":"2023-10-30T13:35:08.750279+0000","last_clean":"2023-10-30T13:35:08.750279+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:35:08.750279+0000","last_undegraded":"2023-10-30T13:35:08.750279+0000","last_fullsized":"2023-10-30T13:35:08.750279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:19.259009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:32.855435+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:35:32.855435+0000","last_peered":"2023-10-30T13:35:32.855435+0000","last_clean":"2023-10-30T13:35:32.855435+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:35:32.855435+0000","last_undegraded":"2023-10-30T13:35:32.855435+0000","last_fullsized":"2023-10-30T13:35:32.855435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1870,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993842+0000","last_change":"2023-10-30T13:35:07.791923+0000","last_active":"2023-10-30T13:35:10.993842+0000","last_peered":"2023-10-30T13:35:10.993842+0000","last_clean":"2023-10-30T13:35:10.993842+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:35:10.993842+0000","last_undegraded":"2023-10-30T13:35:10.993842+0000","last_fullsized":"2023-10-30T13:35:10.993842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:33.080768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.801219+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:35:24.801219+0000","last_peered":"2023-10-30T13:35:24.801219+0000","last_clean":"2023-10-30T13:35:24.801219+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:35:24.801219+0000","last_undegraded":"2023-10-30T13:35:24.801219+0000","last_fullsized":"2023-10-30T13:35:24.801219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298364+0000","last_change":"2023-10-30T13:35:07.794790+0000","last_active":"2023-10-30T13:35:09.298364+0000","last_peered":"2023-10-30T13:35:09.298364+0000","last_clean":"2023-10-30T13:35:09.298364+0000","last_became_active":"2023-10-30T13:35:07.794477+0000","last_became_peered":"2023-10-30T13:35:07.794477+0000","last_unstale":"2023-10-30T13:35:09.298364+0000","last_undegraded":"2023-10-30T13:35:09.298364+0000","last_fullsized":"2023-10-30T13:35:09.298364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:45.565940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.354274+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:35:19.354274+0000","last_peered":"2023-10-30T13:35:19.354274+0000","last_clean":"2023-10-30T13:35:19.354274+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:35:19.354274+0000","last_undegraded":"2023-10-30T13:35:19.354274+0000","last_fullsized":"2023-10-30T13:35:19.354274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993612+0000","last_change":"2023-10-30T13:35:07.797247+0000","last_active":"2023-10-30T13:35:10.993612+0000","last_peered":"2023-10-30T13:35:10.993612+0000","last_clean":"2023-10-30T13:35:10.993612+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:35:10.993612+0000","last_undegraded":"2023-10-30T13:35:10.993612+0000","last_fullsized":"2023-10-30T13:35:10.993612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:57:02.556658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.803960+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:35:24.803960+0000","last_peered":"2023-10-30T13:35:24.803960+0000","last_clean":"2023-10-30T13:35:24.803960+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:35:24.803960+0000","last_undegraded":"2023-10-30T13:35:24.803960+0000","last_fullsized":"2023-10-30T13:35:24.803960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993402+0000","last_change":"2023-10-30T13:35:07.773683+0000","last_active":"2023-10-30T13:35:10.993402+0000","last_peered":"2023-10-30T13:35:10.993402+0000","last_clean":"2023-10-30T13:35:10.993402+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:35:10.993402+0000","last_undegraded":"2023-10-30T13:35:10.993402+0000","last_fullsized":"2023-10-30T13:35:10.993402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:30:03.553514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804386+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:35:24.804386+0000","last_peered":"2023-10-30T13:35:24.804386+0000","last_clean":"2023-10-30T13:35:24.804386+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:35:24.804386+0000","last_undegraded":"2023-10-30T13:35:24.804386+0000","last_fullsized":"2023-10-30T13:35:24.804386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993760+0000","last_change":"2023-10-30T13:35:07.791266+0000","last_active":"2023-10-30T13:35:10.993760+0000","last_peered":"2023-10-30T13:35:10.993760+0000","last_clean":"2023-10-30T13:35:10.993760+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:35:10.993760+0000","last_undegraded":"2023-10-30T13:35:10.993760+0000","last_fullsized":"2023-10-30T13:35:10.993760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:20.086559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291409+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:35:13.291409+0000","last_peered":"2023-10-30T13:35:13.291409+0000","last_clean":"2023-10-30T13:35:13.291409+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:35:13.291409+0000","last_undegraded":"2023-10-30T13:35:13.291409+0000","last_fullsized":"2023-10-30T13:35:13.291409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994310+0000","last_change":"2023-10-30T13:35:07.794747+0000","last_active":"2023-10-30T13:35:10.994310+0000","last_peered":"2023-10-30T13:35:10.994310+0000","last_clean":"2023-10-30T13:35:10.994310+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:35:10.994310+0000","last_undegraded":"2023-10-30T13:35:10.994310+0000","last_fullsized":"2023-10-30T13:35:10.994310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:40.026054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.287725+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:35:13.287725+0000","last_peered":"2023-10-30T13:35:13.287725+0000","last_clean":"2023-10-30T13:35:13.287725+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:35:13.287725+0000","last_undegraded":"2023-10-30T13:35:13.287725+0000","last_fullsized":"2023-10-30T13:35:13.287725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994800+0000","last_change":"2023-10-30T13:35:07.793255+0000","last_active":"2023-10-30T13:35:10.994800+0000","last_peered":"2023-10-30T13:35:10.994800+0000","last_clean":"2023-10-30T13:35:10.994800+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:35:10.994800+0000","last_undegraded":"2023-10-30T13:35:10.994800+0000","last_fullsized":"2023-10-30T13:35:10.994800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:56.491002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236788+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:35:09.236788+0000","last_peered":"2023-10-30T13:35:09.236788+0000","last_clean":"2023-10-30T13:35:09.236788+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:35:09.236788+0000","last_undegraded":"2023-10-30T13:35:09.236788+0000","last_fullsized":"2023-10-30T13:35:09.236788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748659+0000","last_change":"2023-10-30T13:35:07.791920+0000","last_active":"2023-10-30T13:35:08.748659+0000","last_peered":"2023-10-30T13:35:08.748659+0000","last_clean":"2023-10-30T13:35:08.748659+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:35:08.748659+0000","last_undegraded":"2023-10-30T13:35:08.748659+0000","last_fullsized":"2023-10-30T13:35:08.748659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:04.379349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.800093+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:35:24.800093+0000","last_peered":"2023-10-30T13:35:24.800093+0000","last_clean":"2023-10-30T13:35:24.800093+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:35:24.800093+0000","last_undegraded":"2023-10-30T13:35:24.800093+0000","last_fullsized":"2023-10-30T13:35:24.800093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994214+0000","last_change":"2023-10-30T13:35:07.803211+0000","last_active":"2023-10-30T13:35:10.994214+0000","last_peered":"2023-10-30T13:35:10.994214+0000","last_clean":"2023-10-30T13:35:10.994214+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:35:10.994214+0000","last_undegraded":"2023-10-30T13:35:10.994214+0000","last_fullsized":"2023-10-30T13:35:10.994214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:55.371993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768067+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:35:07.768067+0000","last_peered":"2023-10-30T13:35:07.768067+0000","last_clean":"2023-10-30T13:35:07.768067+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:35:07.768067+0000","last_undegraded":"2023-10-30T13:35:07.768067+0000","last_fullsized":"2023-10-30T13:35:07.768067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.353148+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:35:19.353148+0000","last_peered":"2023-10-30T13:35:19.353148+0000","last_clean":"2023-10-30T13:35:19.353148+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:35:19.353148+0000","last_undegraded":"2023-10-30T13:35:19.353148+0000","last_fullsized":"2023-10-30T13:35:19.353148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531304+0000","last_change":"2023-10-30T13:35:07.792082+0000","last_active":"2023-10-30T13:35:09.531304+0000","last_peered":"2023-10-30T13:35:09.531304+0000","last_clean":"2023-10-30T13:35:09.531304+0000","last_became_active":"2023-10-30T13:35:07.791831+0000","last_became_peered":"2023-10-30T13:35:07.791831+0000","last_unstale":"2023-10-30T13:35:09.531304+0000","last_undegraded":"2023-10-30T13:35:09.531304+0000","last_fullsized":"2023-10-30T13:35:09.531304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:47.863235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799794+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:35:24.799794+0000","last_peered":"2023-10-30T13:35:24.799794+0000","last_clean":"2023-10-30T13:35:24.799794+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:35:24.799794+0000","last_undegraded":"2023-10-30T13:35:24.799794+0000","last_fullsized":"2023-10-30T13:35:24.799794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298228+0000","last_change":"2023-10-30T13:35:07.760132+0000","last_active":"2023-10-30T13:35:09.298228+0000","last_peered":"2023-10-30T13:35:09.298228+0000","last_clean":"2023-10-30T13:35:09.298228+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:35:09.298228+0000","last_undegraded":"2023-10-30T13:35:09.298228+0000","last_fullsized":"2023-10-30T13:35:09.298228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:14.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799584+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:35:24.799584+0000","last_peered":"2023-10-30T13:35:24.799584+0000","last_clean":"2023-10-30T13:35:24.799584+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:35:24.799584+0000","last_undegraded":"2023-10-30T13:35:24.799584+0000","last_fullsized":"2023-10-30T13:35:24.799584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.750958+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:53.697186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747960+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:35:08.747960+0000","last_peered":"2023-10-30T13:35:08.747960+0000","last_clean":"2023-10-30T13:35:08.747960+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:35:08.747960+0000","last_undegraded":"2023-10-30T13:35:08.747960+0000","last_fullsized":"2023-10-30T13:35:08.747960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750179+0000","last_change":"2023-10-30T13:35:07.768640+0000","last_active":"2023-10-30T13:35:08.750179+0000","last_peered":"2023-10-30T13:35:08.750179+0000","last_clean":"2023-10-30T13:35:08.750179+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:35:08.750179+0000","last_undegraded":"2023-10-30T13:35:08.750179+0000","last_fullsized":"2023-10-30T13:35:08.750179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:26.852057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743612+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:35:08.743612+0000","last_peered":"2023-10-30T13:35:08.743612+0000","last_clean":"2023-10-30T13:35:08.743612+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:35:08.743612+0000","last_undegraded":"2023-10-30T13:35:08.743612+0000","last_fullsized":"2023-10-30T13:35:08.743612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749093+0000","last_change":"2023-10-30T13:35:07.805535+0000","last_active":"2023-10-30T13:35:08.749093+0000","last_peered":"2023-10-30T13:35:08.749093+0000","last_clean":"2023-10-30T13:35:08.749093+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:35:08.749093+0000","last_undegraded":"2023-10-30T13:35:08.749093+0000","last_fullsized":"2023-10-30T13:35:08.749093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:06.549889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994784+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:35:10.994784+0000","last_peered":"2023-10-30T13:35:10.994784+0000","last_clean":"2023-10-30T13:35:10.994784+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:35:10.994784+0000","last_undegraded":"2023-10-30T13:35:10.994784+0000","last_fullsized":"2023-10-30T13:35:10.994784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743474+0000","last_change":"2023-10-30T13:35:07.754171+0000","last_active":"2023-10-30T13:35:08.743474+0000","last_peered":"2023-10-30T13:35:08.743474+0000","last_clean":"2023-10-30T13:35:08.743474+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:35:08.743474+0000","last_undegraded":"2023-10-30T13:35:08.743474+0000","last_fullsized":"2023-10-30T13:35:08.743474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:50.849695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.801594+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:35:24.801594+0000","last_peered":"2023-10-30T13:35:24.801594+0000","last_clean":"2023-10-30T13:35:24.801594+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:35:24.801594+0000","last_undegraded":"2023-10-30T13:35:24.801594+0000","last_fullsized":"2023-10-30T13:35:24.801594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531754+0000","last_change":"2023-10-30T13:35:07.794804+0000","last_active":"2023-10-30T13:35:09.531754+0000","last_peered":"2023-10-30T13:35:09.531754+0000","last_clean":"2023-10-30T13:35:09.531754+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:35:09.531754+0000","last_undegraded":"2023-10-30T13:35:09.531754+0000","last_fullsized":"2023-10-30T13:35:09.531754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:00.913592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.355614+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:35:19.355614+0000","last_peered":"2023-10-30T13:35:19.355614+0000","last_clean":"2023-10-30T13:35:19.355614+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:35:19.355614+0000","last_undegraded":"2023-10-30T13:35:19.355614+0000","last_fullsized":"2023-10-30T13:35:19.355614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298809+0000","last_change":"2023-10-30T13:35:07.752077+0000","last_active":"2023-10-30T13:35:09.298809+0000","last_peered":"2023-10-30T13:35:09.298809+0000","last_clean":"2023-10-30T13:35:09.298809+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:35:09.298809+0000","last_undegraded":"2023-10-30T13:35:09.298809+0000","last_fullsized":"2023-10-30T13:35:09.298809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:07.661975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.350893+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:35:19.350893+0000","last_peered":"2023-10-30T13:35:19.350893+0000","last_clean":"2023-10-30T13:35:19.350893+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:35:19.350893+0000","last_undegraded":"2023-10-30T13:35:19.350893+0000","last_fullsized":"2023-10-30T13:35:19.350893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748158+0000","last_change":"2023-10-30T13:35:07.759389+0000","last_active":"2023-10-30T13:35:08.748158+0000","last_peered":"2023-10-30T13:35:08.748158+0000","last_clean":"2023-10-30T13:35:08.748158+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:35:08.748158+0000","last_undegraded":"2023-10-30T13:35:08.748158+0000","last_fullsized":"2023-10-30T13:35:08.748158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:15.536615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.351639+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:35:19.351639+0000","last_peered":"2023-10-30T13:35:19.351639+0000","last_clean":"2023-10-30T13:35:19.351639+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:35:19.351639+0000","last_undegraded":"2023-10-30T13:35:19.351639+0000","last_fullsized":"2023-10-30T13:35:19.351639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531288+0000","last_change":"2023-10-30T13:35:07.753673+0000","last_active":"2023-10-30T13:35:09.531288+0000","last_peered":"2023-10-30T13:35:09.531288+0000","last_clean":"2023-10-30T13:35:09.531288+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:35:09.531288+0000","last_undegraded":"2023-10-30T13:35:09.531288+0000","last_fullsized":"2023-10-30T13:35:09.531288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:27:18.430183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236727+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:35:09.236727+0000","last_peered":"2023-10-30T13:35:09.236727+0000","last_clean":"2023-10-30T13:35:09.236727+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:35:09.236727+0000","last_undegraded":"2023-10-30T13:35:09.236727+0000","last_fullsized":"2023-10-30T13:35:09.236727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531465+0000","last_change":"2023-10-30T13:35:07.755302+0000","last_active":"2023-10-30T13:35:09.531465+0000","last_peered":"2023-10-30T13:35:09.531465+0000","last_clean":"2023-10-30T13:35:09.531465+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:35:09.531465+0000","last_undegraded":"2023-10-30T13:35:09.531465+0000","last_fullsized":"2023-10-30T13:35:09.531465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:31.543587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.352360+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:35:19.352360+0000","last_peered":"2023-10-30T13:35:19.352360+0000","last_clean":"2023-10-30T13:35:19.352360+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:35:19.352360+0000","last_undegraded":"2023-10-30T13:35:19.352360+0000","last_fullsized":"2023-10-30T13:35:19.352360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992766+0000","last_change":"2023-10-30T13:35:07.780297+0000","last_active":"2023-10-30T13:35:10.992766+0000","last_peered":"2023-10-30T13:35:10.992766+0000","last_clean":"2023-10-30T13:35:10.992766+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:35:10.992766+0000","last_undegraded":"2023-10-30T13:35:10.992766+0000","last_fullsized":"2023-10-30T13:35:10.992766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:48.971885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748946+0000","last_change":"2023-10-30T13:35:07.806203+0000","last_active":"2023-10-30T13:35:08.748946+0000","last_peered":"2023-10-30T13:35:08.748946+0000","last_clean":"2023-10-30T13:35:08.748946+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:35:08.748946+0000","last_undegraded":"2023-10-30T13:35:08.748946+0000","last_fullsized":"2023-10-30T13:35:08.748946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:34.175218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294221+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:35:13.294221+0000","last_peered":"2023-10-30T13:35:13.294221+0000","last_clean":"2023-10-30T13:35:13.294221+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:35:13.294221+0000","last_undegraded":"2023-10-30T13:35:13.294221+0000","last_fullsized":"2023-10-30T13:35:13.294221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748564+0000","last_change":"2023-10-30T13:35:07.791319+0000","last_active":"2023-10-30T13:35:08.748564+0000","last_peered":"2023-10-30T13:35:08.748564+0000","last_clean":"2023-10-30T13:35:08.748564+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:35:08.748564+0000","last_undegraded":"2023-10-30T13:35:08.748564+0000","last_fullsized":"2023-10-30T13:35:08.748564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:42.560856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799715+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:35:24.799715+0000","last_peered":"2023-10-30T13:35:24.799715+0000","last_clean":"2023-10-30T13:35:24.799715+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:35:24.799715+0000","last_undegraded":"2023-10-30T13:35:24.799715+0000","last_fullsized":"2023-10-30T13:35:24.799715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298499+0000","last_change":"2023-10-30T13:35:07.790084+0000","last_active":"2023-10-30T13:35:09.298499+0000","last_peered":"2023-10-30T13:35:09.298499+0000","last_clean":"2023-10-30T13:35:09.298499+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:35:09.298499+0000","last_undegraded":"2023-10-30T13:35:09.298499+0000","last_fullsized":"2023-10-30T13:35:09.298499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:45.193072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291592+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:35:13.291592+0000","last_peered":"2023-10-30T13:35:13.291592+0000","last_clean":"2023-10-30T13:35:13.291592+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:35:13.291592+0000","last_undegraded":"2023-10-30T13:35:13.291592+0000","last_fullsized":"2023-10-30T13:35:13.291592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531695+0000","last_change":"2023-10-30T13:35:07.782341+0000","last_active":"2023-10-30T13:35:09.531695+0000","last_peered":"2023-10-30T13:35:09.531695+0000","last_clean":"2023-10-30T13:35:09.531695+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:35:09.531695+0000","last_undegraded":"2023-10-30T13:35:09.531695+0000","last_fullsized":"2023-10-30T13:35:09.531695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:19:06.392683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804103+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:35:24.804103+0000","last_peered":"2023-10-30T13:35:24.804103+0000","last_clean":"2023-10-30T13:35:24.804103+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:35:24.804103+0000","last_undegraded":"2023-10-30T13:35:24.804103+0000","last_fullsized":"2023-10-30T13:35:24.804103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5738,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":664793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509452,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":75691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509452,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":73669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":73795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]}]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":74229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]}]},{"osd":4,"up_from":14,"seq":60129542155,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":73565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":663879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:35:37.142 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-30T13:35:37.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:35:37.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4270a6b640 1 Processor -- start 2023-10-30T13:35:37.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4270a6b640 1 -- start start 2023-10-30T13:35:37.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4270a6b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f426c15cf90 0x7f426c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4270a6b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c159f20 con 0x7f426c153c80 2023-10-30T13:35:37.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4270a6b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c0c47e0 con 0x7f426c0c4a10 2023-10-30T13:35:37.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4270a6b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426c15dec0 con 0x7f426c15cf90 2023-10-30T13:35:37.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f426a59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f426c153c80 0x7f426c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43424/0 (socket says 172.21.15.31:43424) 2023-10-30T13:35:37.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f426a59c640 1 -- 172.21.15.31:0/2507532616 learned_addr learned my addr 172.21.15.31:0/2507532616 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:37.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f426ad9d640 1 --2- 172.21.15.31:0/2507532616 >> v2:172.21.15.31:3300/0 conn(0x7f426c15cf90 0x7f426c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4269d9b640 1 -- 172.21.15.31:0/2507532616 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131926228 0 0) 0x7f426c159f20 con 0x7f426c153c80 2023-10-30T13:35:37.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f4269d9b640 1 -- 172.21.15.31:0/2507532616 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4258003530 con 0x7f426c153c80 2023-10-30T13:35:37.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f426ad9d640 1 -- 172.21.15.31:0/2507532616 >> v1:172.21.15.33:6789/0 conn(0x7f426c0c4a10 legacy=0x7f426c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f426ad9d640 1 -- 172.21.15.31:0/2507532616 >> v1:172.21.15.31:6789/0 conn(0x7f426c153c80 legacy=0x7f426c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.316+0000 7f426ad9d640 1 -- 172.21.15.31:0/2507532616 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c15d620 con 0x7f426c15cf90 2023-10-30T13:35:37.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f426ad9d640 1 --2- 172.21.15.31:0/2507532616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0c42a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4264009d90 tx=0x7f426401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d745f50508467a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4269d9b640 1 -- 172.21.15.31:0/2507532616 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640af600 con 0x7f426c15cf90 2023-10-30T13:35:37.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4269d9b640 1 -- 172.21.15.31:0/2507532616 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42640ab050 con 0x7f426c15cf90 2023-10-30T13:35:37.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4269d9b640 1 -- 172.21.15.31:0/2507532616 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640af980 con 0x7f426c15cf90 2023-10-30T13:35:37.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4270a6b640 1 -- 172.21.15.31:0/2507532616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 msgr2=0x7f426c0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4270a6b640 1 --2- 172.21.15.31:0/2507532616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0c42a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4264009d90 tx=0x7f426401f130 comp rx=0 tx=0).stop 2023-10-30T13:35:37.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4270a6b640 1 -- 172.21.15.31:0/2507532616 shutdown_connections 2023-10-30T13:35:37.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4270a6b640 1 --2- 172.21.15.31:0/2507532616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0c42a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4270a6b640 1 -- 172.21.15.31:0/2507532616 >> 172.21.15.31:0/2507532616 conn(0x7f426c0715c0 msgr2=0x7f426c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:37.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4270a6b640 1 -- 172.21.15.31:0/2507532616 shutdown_connections 2023-10-30T13:35:37.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.320+0000 7f4270a6b640 1 -- 172.21.15.31:0/2507532616 wait complete. 2023-10-30T13:35:37.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 Processor -- start 2023-10-30T13:35:37.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 -- start start 2023-10-30T13:35:37.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f426c0c4a10 0x7f426c1eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f426c153c80 0x7f426c1f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c159a70 con 0x7f426c15cf90 2023-10-30T13:35:37.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1ef460 con 0x7f426c0c4a10 2023-10-30T13:35:37.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c159260 con 0x7f426c153c80 2023-10-30T13:35:37.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51374/0 (socket says 172.21.15.31:51374) 2023-10-30T13:35:37.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 -- 172.21.15.31:0/1537649592 learned_addr learned my addr 172.21.15.31:0/1537649592 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:37.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426a59c640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f426c0c4a10 0x7f426c1eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4263fff640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f426c153c80 0x7f426c1f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 -- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f426c153c80 msgr2=0x7f426c1f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f426c153c80 0x7f426c1f1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 -- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f426c0c4a10 msgr2=0x7f426c1eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f426c0c4a10 0x7f426c1eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 -- 172.21.15.31:0/1537649592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4264009a70 con 0x7f426c15cf90 2023-10-30T13:35:37.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426a59c640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f426c0c4a10 0x7f426c1eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:37.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f426ad9d640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0ca140 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4264009d90 tx=0x7f426401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4262ffd640 1 -- 172.21.15.31:0/1537649592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264009450 con 0x7f426c15cf90 2023-10-30T13:35:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4262ffd640 1 -- 172.21.15.31:0/1537649592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42640ba480 con 0x7f426c15cf90 2023-10-30T13:35:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c1f1f50 con 0x7f426c15cf90 2023-10-30T13:35:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4262ffd640 1 -- 172.21.15.31:0/1537649592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640ba830 con 0x7f426c15cf90 2023-10-30T13:35:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.324+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c159790 con 0x7f426c15cf90 2023-10-30T13:35:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.328+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c04e170 con 0x7f426c15cf90 2023-10-30T13:35:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.328+0000 7f4262ffd640 1 -- 172.21.15.31:0/1537649592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42640b6040 con 0x7f426c15cf90 2023-10-30T13:35:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.328+0000 7f4262ffd640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f424c0664a0 0x7f424c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.328+0000 7f4262ffd640 1 -- 172.21.15.31:0/1537649592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4264122880 con 0x7f426c15cf90 2023-10-30T13:35:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.328+0000 7f426a59c640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f424c0664a0 0x7f424c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.328+0000 7f426a59c640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f424c0664a0 0x7f424c068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f425c0011f0 tx=0x7f425c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.332+0000 7f4262ffd640 1 -- 172.21.15.31:0/1537649592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42640efbd0 con 0x7f426c15cf90 2023-10-30T13:35:37.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.488+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f426c15dbe0 con 0x7f424c0664a0 2023-10-30T13:35:37.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.508+0000 7f4262ffd640 1 -- 172.21.15.31:0/1537649592 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264699 (secure 0 0 0) 0x7f425c001480 con 0x7f424c0664a0 2023-10-30T13:35:37.514 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:37.515 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:35:37.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f424c0664a0 msgr2=0x7f424c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f424c0664a0 0x7f424c068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f425c0011f0 tx=0x7f425c009040 comp rx=0 tx=0).stop 2023-10-30T13:35:37.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 msgr2=0x7f426c0ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0ca140 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4264009d90 tx=0x7f426401bed0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 shutdown_connections 2023-10-30T13:35:37.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f424c0664a0 0x7f424c068960 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f426c15cf90 0x7f426c0ca140 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f426c153c80 0x7f426c1f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 --2- 172.21.15.31:0/1537649592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f426c0c4a10 0x7f426c1eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 >> 172.21.15.31:0/1537649592 conn(0x7f426c0715c0 msgr2=0x7f426c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:37.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 shutdown_connections 2023-10-30T13:35:37.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.512+0000 7f4270a6b640 1 -- 172.21.15.31:0/1537649592 wait complete. 2023-10-30T13:35:37.534 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-30T13:35:37.167123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596106,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":287,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219080,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755640,"statfs":{"total":773094113280,"available":772869775360,"internally_reserved":0,"allocated":3022848,"data_stored":1775020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1922,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005330"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994380+0000","last_change":"2023-10-30T13:35:07.792742+0000","last_active":"2023-10-30T13:35:10.994380+0000","last_peered":"2023-10-30T13:35:10.994380+0000","last_clean":"2023-10-30T13:35:10.994380+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:35:10.994380+0000","last_undegraded":"2023-10-30T13:35:10.994380+0000","last_fullsized":"2023-10-30T13:35:10.994380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:02.805042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.744019+0000","last_change":"2023-10-30T13:35:07.755187+0000","last_active":"2023-10-30T13:35:08.744019+0000","last_peered":"2023-10-30T13:35:08.744019+0000","last_clean":"2023-10-30T13:35:08.744019+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:35:08.744019+0000","last_undegraded":"2023-10-30T13:35:08.744019+0000","last_fullsized":"2023-10-30T13:35:08.744019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:22.764808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298537+0000","last_change":"2023-10-30T13:35:07.754357+0000","last_active":"2023-10-30T13:35:09.298537+0000","last_peered":"2023-10-30T13:35:09.298537+0000","last_clean":"2023-10-30T13:35:09.298537+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:35:09.298537+0000","last_undegraded":"2023-10-30T13:35:09.298537+0000","last_fullsized":"2023-10-30T13:35:09.298537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:23.201416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748146+0000","last_change":"2023-10-30T13:35:07.792466+0000","last_active":"2023-10-30T13:35:08.748146+0000","last_peered":"2023-10-30T13:35:08.748146+0000","last_clean":"2023-10-30T13:35:08.748146+0000","last_became_active":"2023-10-30T13:35:07.789749+0000","last_became_peered":"2023-10-30T13:35:07.789749+0000","last_unstale":"2023-10-30T13:35:08.748146+0000","last_undegraded":"2023-10-30T13:35:08.748146+0000","last_fullsized":"2023-10-30T13:35:08.748146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:46.527032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993635+0000","last_change":"2023-10-30T13:35:07.793403+0000","last_active":"2023-10-30T13:35:10.993635+0000","last_peered":"2023-10-30T13:35:10.993635+0000","last_clean":"2023-10-30T13:35:10.993635+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:35:10.993635+0000","last_undegraded":"2023-10-30T13:35:10.993635+0000","last_fullsized":"2023-10-30T13:35:10.993635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:27.304652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747210+0000","last_change":"2023-10-30T13:35:07.792285+0000","last_active":"2023-10-30T13:35:08.747210+0000","last_peered":"2023-10-30T13:35:08.747210+0000","last_clean":"2023-10-30T13:35:08.747210+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:35:08.747210+0000","last_undegraded":"2023-10-30T13:35:08.747210+0000","last_fullsized":"2023-10-30T13:35:08.747210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:31.537183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993398+0000","last_change":"2023-10-30T13:35:07.776386+0000","last_active":"2023-10-30T13:35:10.993398+0000","last_peered":"2023-10-30T13:35:10.993398+0000","last_clean":"2023-10-30T13:35:10.993398+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:35:10.993398+0000","last_undegraded":"2023-10-30T13:35:10.993398+0000","last_fullsized":"2023-10-30T13:35:10.993398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:27.492389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531107+0000","last_change":"2023-10-30T13:35:07.790487+0000","last_active":"2023-10-30T13:35:09.531107+0000","last_peered":"2023-10-30T13:35:09.531107+0000","last_clean":"2023-10-30T13:35:09.531107+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:35:09.531107+0000","last_undegraded":"2023-10-30T13:35:09.531107+0000","last_fullsized":"2023-10-30T13:35:09.531107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:27.384592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994435+0000","last_change":"2023-10-30T13:35:07.804038+0000","last_active":"2023-10-30T13:35:10.994435+0000","last_peered":"2023-10-30T13:35:10.994435+0000","last_clean":"2023-10-30T13:35:10.994435+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:35:10.994435+0000","last_undegraded":"2023-10-30T13:35:10.994435+0000","last_fullsized":"2023-10-30T13:35:10.994435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:09.137404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993722+0000","last_change":"2023-10-30T13:35:07.800657+0000","last_active":"2023-10-30T13:35:10.993722+0000","last_peered":"2023-10-30T13:35:10.993722+0000","last_clean":"2023-10-30T13:35:10.993722+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:35:10.993722+0000","last_undegraded":"2023-10-30T13:35:10.993722+0000","last_fullsized":"2023-10-30T13:35:10.993722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:12.080911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993171+0000","last_change":"2023-10-30T13:35:07.802245+0000","last_active":"2023-10-30T13:35:10.993171+0000","last_peered":"2023-10-30T13:35:10.993171+0000","last_clean":"2023-10-30T13:35:10.993171+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:35:10.993171+0000","last_undegraded":"2023-10-30T13:35:10.993171+0000","last_fullsized":"2023-10-30T13:35:10.993171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:58.462586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750176+0000","last_change":"2023-10-30T13:35:07.783940+0000","last_active":"2023-10-30T13:35:08.750176+0000","last_peered":"2023-10-30T13:35:08.750176+0000","last_clean":"2023-10-30T13:35:08.750176+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:35:08.750176+0000","last_undegraded":"2023-10-30T13:35:08.750176+0000","last_fullsized":"2023-10-30T13:35:08.750176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:07.873653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531433+0000","last_change":"2023-10-30T13:35:07.780349+0000","last_active":"2023-10-30T13:35:09.531433+0000","last_peered":"2023-10-30T13:35:09.531433+0000","last_clean":"2023-10-30T13:35:09.531433+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:35:09.531433+0000","last_undegraded":"2023-10-30T13:35:09.531433+0000","last_fullsized":"2023-10-30T13:35:09.531433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:15.619776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298339+0000","last_change":"2023-10-30T13:35:07.750959+0000","last_active":"2023-10-30T13:35:09.298339+0000","last_peered":"2023-10-30T13:35:09.298339+0000","last_clean":"2023-10-30T13:35:09.298339+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:35:09.298339+0000","last_undegraded":"2023-10-30T13:35:09.298339+0000","last_fullsized":"2023-10-30T13:35:09.298339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:56.853163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.768466+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:27.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748523+0000","last_change":"2023-10-30T13:35:07.789918+0000","last_active":"2023-10-30T13:35:08.748523+0000","last_peered":"2023-10-30T13:35:08.748523+0000","last_clean":"2023-10-30T13:35:08.748523+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:35:08.748523+0000","last_undegraded":"2023-10-30T13:35:08.748523+0000","last_fullsized":"2023-10-30T13:35:08.748523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:53.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749638+0000","last_change":"2023-10-30T13:35:07.796429+0000","last_active":"2023-10-30T13:35:08.749638+0000","last_peered":"2023-10-30T13:35:08.749638+0000","last_clean":"2023-10-30T13:35:08.749638+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:35:08.749638+0000","last_undegraded":"2023-10-30T13:35:08.749638+0000","last_fullsized":"2023-10-30T13:35:08.749638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:15.022866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489395+0000","last_change":"2023-10-30T13:35:07.780131+0000","last_active":"2023-10-30T13:35:10.489395+0000","last_peered":"2023-10-30T13:35:10.489395+0000","last_clean":"2023-10-30T13:35:10.489395+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:35:10.489395+0000","last_undegraded":"2023-10-30T13:35:10.489395+0000","last_fullsized":"2023-10-30T13:35:10.489395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:04.046338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994792+0000","last_change":"2023-10-30T13:35:07.791464+0000","last_active":"2023-10-30T13:35:10.994792+0000","last_peered":"2023-10-30T13:35:10.994792+0000","last_clean":"2023-10-30T13:35:10.994792+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:35:10.994792+0000","last_undegraded":"2023-10-30T13:35:10.994792+0000","last_fullsized":"2023-10-30T13:35:10.994792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993329+0000","last_change":"2023-10-30T13:35:07.783079+0000","last_active":"2023-10-30T13:35:10.993329+0000","last_peered":"2023-10-30T13:35:10.993329+0000","last_clean":"2023-10-30T13:35:10.993329+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:35:10.993329+0000","last_undegraded":"2023-10-30T13:35:10.993329+0000","last_fullsized":"2023-10-30T13:35:10.993329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:43:32.360152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298481+0000","last_change":"2023-10-30T13:35:07.794666+0000","last_active":"2023-10-30T13:35:09.298481+0000","last_peered":"2023-10-30T13:35:09.298481+0000","last_clean":"2023-10-30T13:35:09.298481+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:35:09.298481+0000","last_undegraded":"2023-10-30T13:35:09.298481+0000","last_fullsized":"2023-10-30T13:35:09.298481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:52.189022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747346+0000","last_change":"2023-10-30T13:35:07.796601+0000","last_active":"2023-10-30T13:35:08.747346+0000","last_peered":"2023-10-30T13:35:08.747346+0000","last_clean":"2023-10-30T13:35:08.747346+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:35:08.747346+0000","last_undegraded":"2023-10-30T13:35:08.747346+0000","last_fullsized":"2023-10-30T13:35:08.747346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:07.572028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531135+0000","last_change":"2023-10-30T13:35:07.767965+0000","last_active":"2023-10-30T13:35:09.531135+0000","last_peered":"2023-10-30T13:35:09.531135+0000","last_clean":"2023-10-30T13:35:09.531135+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:35:09.531135+0000","last_undegraded":"2023-10-30T13:35:09.531135+0000","last_fullsized":"2023-10-30T13:35:09.531135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:15.544113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748017+0000","last_change":"2023-10-30T13:35:07.772029+0000","last_active":"2023-10-30T13:35:08.748017+0000","last_peered":"2023-10-30T13:35:08.748017+0000","last_clean":"2023-10-30T13:35:08.748017+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:35:08.748017+0000","last_undegraded":"2023-10-30T13:35:08.748017+0000","last_fullsized":"2023-10-30T13:35:08.748017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:13.305147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531716+0000","last_change":"2023-10-30T13:35:07.758274+0000","last_active":"2023-10-30T13:35:09.531716+0000","last_peered":"2023-10-30T13:35:09.531716+0000","last_clean":"2023-10-30T13:35:09.531716+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:35:09.531716+0000","last_undegraded":"2023-10-30T13:35:09.531716+0000","last_fullsized":"2023-10-30T13:35:09.531716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:15.191933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994099+0000","last_change":"2023-10-30T13:35:07.803220+0000","last_active":"2023-10-30T13:35:10.994099+0000","last_peered":"2023-10-30T13:35:10.994099+0000","last_clean":"2023-10-30T13:35:10.994099+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:35:10.994099+0000","last_undegraded":"2023-10-30T13:35:10.994099+0000","last_fullsized":"2023-10-30T13:35:10.994099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:45.165258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743680+0000","last_change":"2023-10-30T13:35:07.753288+0000","last_active":"2023-10-30T13:35:08.743680+0000","last_peered":"2023-10-30T13:35:08.743680+0000","last_clean":"2023-10-30T13:35:08.743680+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:35:08.743680+0000","last_undegraded":"2023-10-30T13:35:08.743680+0000","last_fullsized":"2023-10-30T13:35:08.743680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:43.815379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531511+0000","last_change":"2023-10-30T13:35:07.767931+0000","last_active":"2023-10-30T13:35:09.531511+0000","last_peered":"2023-10-30T13:35:09.531511+0000","last_clean":"2023-10-30T13:35:09.531511+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:35:09.531511+0000","last_undegraded":"2023-10-30T13:35:09.531511+0000","last_fullsized":"2023-10-30T13:35:09.531511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:43.830918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298592+0000","last_change":"2023-10-30T13:35:07.753695+0000","last_active":"2023-10-30T13:35:09.298592+0000","last_peered":"2023-10-30T13:35:09.298592+0000","last_clean":"2023-10-30T13:35:09.298592+0000","last_became_active":"2023-10-30T13:35:07.753358+0000","last_became_peered":"2023-10-30T13:35:07.753358+0000","last_unstale":"2023-10-30T13:35:09.298592+0000","last_undegraded":"2023-10-30T13:35:09.298592+0000","last_fullsized":"2023-10-30T13:35:09.298592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:12.840727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747580+0000","last_change":"2023-10-30T13:35:07.790236+0000","last_active":"2023-10-30T13:35:08.747580+0000","last_peered":"2023-10-30T13:35:08.747580+0000","last_clean":"2023-10-30T13:35:08.747580+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:35:08.747580+0000","last_undegraded":"2023-10-30T13:35:08.747580+0000","last_fullsized":"2023-10-30T13:35:08.747580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:22.029281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.489368+0000","last_change":"2023-10-30T13:35:07.790720+0000","last_active":"2023-10-30T13:35:10.489368+0000","last_peered":"2023-10-30T13:35:10.489368+0000","last_clean":"2023-10-30T13:35:10.489368+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:35:10.489368+0000","last_undegraded":"2023-10-30T13:35:10.489368+0000","last_fullsized":"2023-10-30T13:35:10.489368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:16.691133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992429+0000","last_change":"2023-10-30T13:35:07.780576+0000","last_active":"2023-10-30T13:35:10.992429+0000","last_peered":"2023-10-30T13:35:10.992429+0000","last_clean":"2023-10-30T13:35:10.992429+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:35:10.992429+0000","last_undegraded":"2023-10-30T13:35:10.992429+0000","last_fullsized":"2023-10-30T13:35:10.992429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:20.704354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749626+0000","last_change":"2023-10-30T13:35:07.755919+0000","last_active":"2023-10-30T13:35:08.749626+0000","last_peered":"2023-10-30T13:35:08.749626+0000","last_clean":"2023-10-30T13:35:08.749626+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:35:08.749626+0000","last_undegraded":"2023-10-30T13:35:08.749626+0000","last_fullsized":"2023-10-30T13:35:08.749626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:01.978170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.360079+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:35:19.360079+0000","last_peered":"2023-10-30T13:35:19.360079+0000","last_clean":"2023-10-30T13:35:19.360079+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:35:19.360079+0000","last_undegraded":"2023-10-30T13:35:19.360079+0000","last_fullsized":"2023-10-30T13:35:19.360079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":880,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298837+0000","last_change":"2023-10-30T13:35:07.762428+0000","last_active":"2023-10-30T13:35:09.298837+0000","last_peered":"2023-10-30T13:35:09.298837+0000","last_clean":"2023-10-30T13:35:09.298837+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:35:09.298837+0000","last_undegraded":"2023-10-30T13:35:09.298837+0000","last_fullsized":"2023-10-30T13:35:09.298837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:14.191396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804153+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:35:24.804153+0000","last_peered":"2023-10-30T13:35:24.804153+0000","last_clean":"2023-10-30T13:35:24.804153+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:35:24.804153+0000","last_undegraded":"2023-10-30T13:35:24.804153+0000","last_fullsized":"2023-10-30T13:35:24.804153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.803715+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:35:24.803715+0000","last_peered":"2023-10-30T13:35:24.803715+0000","last_clean":"2023-10-30T13:35:24.803715+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:35:24.803715+0000","last_undegraded":"2023-10-30T13:35:24.803715+0000","last_fullsized":"2023-10-30T13:35:24.803715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531211+0000","last_change":"2023-10-30T13:35:07.751131+0000","last_active":"2023-10-30T13:35:09.531211+0000","last_peered":"2023-10-30T13:35:09.531211+0000","last_clean":"2023-10-30T13:35:09.531211+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:35:09.531211+0000","last_undegraded":"2023-10-30T13:35:09.531211+0000","last_fullsized":"2023-10-30T13:35:09.531211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:55:46.387299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748359+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:35:08.748359+0000","last_peered":"2023-10-30T13:35:08.748359+0000","last_clean":"2023-10-30T13:35:08.748359+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:35:08.748359+0000","last_undegraded":"2023-10-30T13:35:08.748359+0000","last_fullsized":"2023-10-30T13:35:08.748359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743438+0000","last_change":"2023-10-30T13:35:07.750392+0000","last_active":"2023-10-30T13:35:08.743438+0000","last_peered":"2023-10-30T13:35:08.743438+0000","last_clean":"2023-10-30T13:35:08.743438+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:35:08.743438+0000","last_undegraded":"2023-10-30T13:35:08.743438+0000","last_fullsized":"2023-10-30T13:35:08.743438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:33.554754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.359410+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:35:19.359410+0000","last_peered":"2023-10-30T13:35:19.359410+0000","last_clean":"2023-10-30T13:35:19.359410+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:35:19.359410+0000","last_undegraded":"2023-10-30T13:35:19.359410+0000","last_fullsized":"2023-10-30T13:35:19.359410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749206+0000","last_change":"2023-10-30T13:35:07.797513+0000","last_active":"2023-10-30T13:35:08.749206+0000","last_peered":"2023-10-30T13:35:08.749206+0000","last_clean":"2023-10-30T13:35:08.749206+0000","last_became_active":"2023-10-30T13:35:07.796748+0000","last_became_peered":"2023-10-30T13:35:07.796748+0000","last_unstale":"2023-10-30T13:35:08.749206+0000","last_undegraded":"2023-10-30T13:35:08.749206+0000","last_fullsized":"2023-10-30T13:35:08.749206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:41.838836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.351941+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:35:19.351941+0000","last_peered":"2023-10-30T13:35:19.351941+0000","last_clean":"2023-10-30T13:35:19.351941+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:35:19.351941+0000","last_undegraded":"2023-10-30T13:35:19.351941+0000","last_fullsized":"2023-10-30T13:35:19.351941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298743+0000","last_change":"2023-10-30T13:35:07.789983+0000","last_active":"2023-10-30T13:35:09.298743+0000","last_peered":"2023-10-30T13:35:09.298743+0000","last_clean":"2023-10-30T13:35:09.298743+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:35:09.298743+0000","last_undegraded":"2023-10-30T13:35:09.298743+0000","last_fullsized":"2023-10-30T13:35:09.298743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:03.304499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.290855+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:35:13.290855+0000","last_peered":"2023-10-30T13:35:13.290855+0000","last_clean":"2023-10-30T13:35:13.290855+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:35:13.290855+0000","last_undegraded":"2023-10-30T13:35:13.290855+0000","last_fullsized":"2023-10-30T13:35:13.290855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750279+0000","last_change":"2023-10-30T13:35:07.794930+0000","last_active":"2023-10-30T13:35:08.750279+0000","last_peered":"2023-10-30T13:35:08.750279+0000","last_clean":"2023-10-30T13:35:08.750279+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:35:08.750279+0000","last_undegraded":"2023-10-30T13:35:08.750279+0000","last_fullsized":"2023-10-30T13:35:08.750279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:19.259009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:32.855435+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:35:32.855435+0000","last_peered":"2023-10-30T13:35:32.855435+0000","last_clean":"2023-10-30T13:35:32.855435+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:35:32.855435+0000","last_undegraded":"2023-10-30T13:35:32.855435+0000","last_fullsized":"2023-10-30T13:35:32.855435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1870,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993842+0000","last_change":"2023-10-30T13:35:07.791923+0000","last_active":"2023-10-30T13:35:10.993842+0000","last_peered":"2023-10-30T13:35:10.993842+0000","last_clean":"2023-10-30T13:35:10.993842+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:35:10.993842+0000","last_undegraded":"2023-10-30T13:35:10.993842+0000","last_fullsized":"2023-10-30T13:35:10.993842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:33.080768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.801219+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:35:24.801219+0000","last_peered":"2023-10-30T13:35:24.801219+0000","last_clean":"2023-10-30T13:35:24.801219+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:35:24.801219+0000","last_undegraded":"2023-10-30T13:35:24.801219+0000","last_fullsized":"2023-10-30T13:35:24.801219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298364+0000","last_change":"2023-10-30T13:35:07.794790+0000","last_active":"2023-10-30T13:35:09.298364+0000","last_peered":"2023-10-30T13:35:09.298364+0000","last_clean":"2023-10-30T13:35:09.298364+0000","last_became_active":"2023-10-30T13:35:07.794477+0000","last_became_peered":"2023-10-30T13:35:07.794477+0000","last_unstale":"2023-10-30T13:35:09.298364+0000","last_undegraded":"2023-10-30T13:35:09.298364+0000","last_fullsized":"2023-10-30T13:35:09.298364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:45.565940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.354274+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:35:19.354274+0000","last_peered":"2023-10-30T13:35:19.354274+0000","last_clean":"2023-10-30T13:35:19.354274+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:35:19.354274+0000","last_undegraded":"2023-10-30T13:35:19.354274+0000","last_fullsized":"2023-10-30T13:35:19.354274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993612+0000","last_change":"2023-10-30T13:35:07.797247+0000","last_active":"2023-10-30T13:35:10.993612+0000","last_peered":"2023-10-30T13:35:10.993612+0000","last_clean":"2023-10-30T13:35:10.993612+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:35:10.993612+0000","last_undegraded":"2023-10-30T13:35:10.993612+0000","last_fullsized":"2023-10-30T13:35:10.993612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:57:02.556658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.803960+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:35:24.803960+0000","last_peered":"2023-10-30T13:35:24.803960+0000","last_clean":"2023-10-30T13:35:24.803960+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:35:24.803960+0000","last_undegraded":"2023-10-30T13:35:24.803960+0000","last_fullsized":"2023-10-30T13:35:24.803960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993402+0000","last_change":"2023-10-30T13:35:07.773683+0000","last_active":"2023-10-30T13:35:10.993402+0000","last_peered":"2023-10-30T13:35:10.993402+0000","last_clean":"2023-10-30T13:35:10.993402+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:35:10.993402+0000","last_undegraded":"2023-10-30T13:35:10.993402+0000","last_fullsized":"2023-10-30T13:35:10.993402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:30:03.553514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804386+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:35:24.804386+0000","last_peered":"2023-10-30T13:35:24.804386+0000","last_clean":"2023-10-30T13:35:24.804386+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:35:24.804386+0000","last_undegraded":"2023-10-30T13:35:24.804386+0000","last_fullsized":"2023-10-30T13:35:24.804386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.993760+0000","last_change":"2023-10-30T13:35:07.791266+0000","last_active":"2023-10-30T13:35:10.993760+0000","last_peered":"2023-10-30T13:35:10.993760+0000","last_clean":"2023-10-30T13:35:10.993760+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:35:10.993760+0000","last_undegraded":"2023-10-30T13:35:10.993760+0000","last_fullsized":"2023-10-30T13:35:10.993760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:20.086559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291409+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:35:13.291409+0000","last_peered":"2023-10-30T13:35:13.291409+0000","last_clean":"2023-10-30T13:35:13.291409+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:35:13.291409+0000","last_undegraded":"2023-10-30T13:35:13.291409+0000","last_fullsized":"2023-10-30T13:35:13.291409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994310+0000","last_change":"2023-10-30T13:35:07.794747+0000","last_active":"2023-10-30T13:35:10.994310+0000","last_peered":"2023-10-30T13:35:10.994310+0000","last_clean":"2023-10-30T13:35:10.994310+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:35:10.994310+0000","last_undegraded":"2023-10-30T13:35:10.994310+0000","last_fullsized":"2023-10-30T13:35:10.994310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:40.026054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.287725+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:35:13.287725+0000","last_peered":"2023-10-30T13:35:13.287725+0000","last_clean":"2023-10-30T13:35:13.287725+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:35:13.287725+0000","last_undegraded":"2023-10-30T13:35:13.287725+0000","last_fullsized":"2023-10-30T13:35:13.287725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994800+0000","last_change":"2023-10-30T13:35:07.793255+0000","last_active":"2023-10-30T13:35:10.994800+0000","last_peered":"2023-10-30T13:35:10.994800+0000","last_clean":"2023-10-30T13:35:10.994800+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:35:10.994800+0000","last_undegraded":"2023-10-30T13:35:10.994800+0000","last_fullsized":"2023-10-30T13:35:10.994800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:56.491002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236788+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:35:09.236788+0000","last_peered":"2023-10-30T13:35:09.236788+0000","last_clean":"2023-10-30T13:35:09.236788+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:35:09.236788+0000","last_undegraded":"2023-10-30T13:35:09.236788+0000","last_fullsized":"2023-10-30T13:35:09.236788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748659+0000","last_change":"2023-10-30T13:35:07.791920+0000","last_active":"2023-10-30T13:35:08.748659+0000","last_peered":"2023-10-30T13:35:08.748659+0000","last_clean":"2023-10-30T13:35:08.748659+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:35:08.748659+0000","last_undegraded":"2023-10-30T13:35:08.748659+0000","last_fullsized":"2023-10-30T13:35:08.748659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:04.379349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.800093+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:35:24.800093+0000","last_peered":"2023-10-30T13:35:24.800093+0000","last_clean":"2023-10-30T13:35:24.800093+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:35:24.800093+0000","last_undegraded":"2023-10-30T13:35:24.800093+0000","last_fullsized":"2023-10-30T13:35:24.800093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994214+0000","last_change":"2023-10-30T13:35:07.803211+0000","last_active":"2023-10-30T13:35:10.994214+0000","last_peered":"2023-10-30T13:35:10.994214+0000","last_clean":"2023-10-30T13:35:10.994214+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:35:10.994214+0000","last_undegraded":"2023-10-30T13:35:10.994214+0000","last_fullsized":"2023-10-30T13:35:10.994214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:55.371993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:35:07.768067+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:35:07.768067+0000","last_peered":"2023-10-30T13:35:07.768067+0000","last_clean":"2023-10-30T13:35:07.768067+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:35:07.768067+0000","last_undegraded":"2023-10-30T13:35:07.768067+0000","last_fullsized":"2023-10-30T13:35:07.768067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.353148+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:35:19.353148+0000","last_peered":"2023-10-30T13:35:19.353148+0000","last_clean":"2023-10-30T13:35:19.353148+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:35:19.353148+0000","last_undegraded":"2023-10-30T13:35:19.353148+0000","last_fullsized":"2023-10-30T13:35:19.353148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531304+0000","last_change":"2023-10-30T13:35:07.792082+0000","last_active":"2023-10-30T13:35:09.531304+0000","last_peered":"2023-10-30T13:35:09.531304+0000","last_clean":"2023-10-30T13:35:09.531304+0000","last_became_active":"2023-10-30T13:35:07.791831+0000","last_became_peered":"2023-10-30T13:35:07.791831+0000","last_unstale":"2023-10-30T13:35:09.531304+0000","last_undegraded":"2023-10-30T13:35:09.531304+0000","last_fullsized":"2023-10-30T13:35:09.531304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:47.863235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799794+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:35:24.799794+0000","last_peered":"2023-10-30T13:35:24.799794+0000","last_clean":"2023-10-30T13:35:24.799794+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:35:24.799794+0000","last_undegraded":"2023-10-30T13:35:24.799794+0000","last_fullsized":"2023-10-30T13:35:24.799794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298228+0000","last_change":"2023-10-30T13:35:07.760132+0000","last_active":"2023-10-30T13:35:09.298228+0000","last_peered":"2023-10-30T13:35:09.298228+0000","last_clean":"2023-10-30T13:35:09.298228+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:35:09.298228+0000","last_undegraded":"2023-10-30T13:35:09.298228+0000","last_fullsized":"2023-10-30T13:35:09.298228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:14.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799584+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:35:24.799584+0000","last_peered":"2023-10-30T13:35:24.799584+0000","last_clean":"2023-10-30T13:35:24.799584+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:35:24.799584+0000","last_undegraded":"2023-10-30T13:35:24.799584+0000","last_fullsized":"2023-10-30T13:35:24.799584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743486+0000","last_change":"2023-10-30T13:35:07.750958+0000","last_active":"2023-10-30T13:35:08.743486+0000","last_peered":"2023-10-30T13:35:08.743486+0000","last_clean":"2023-10-30T13:35:08.743486+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:35:08.743486+0000","last_undegraded":"2023-10-30T13:35:08.743486+0000","last_fullsized":"2023-10-30T13:35:08.743486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:53.697186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.747960+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:35:08.747960+0000","last_peered":"2023-10-30T13:35:08.747960+0000","last_clean":"2023-10-30T13:35:08.747960+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:35:08.747960+0000","last_undegraded":"2023-10-30T13:35:08.747960+0000","last_fullsized":"2023-10-30T13:35:08.747960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.750179+0000","last_change":"2023-10-30T13:35:07.768640+0000","last_active":"2023-10-30T13:35:08.750179+0000","last_peered":"2023-10-30T13:35:08.750179+0000","last_clean":"2023-10-30T13:35:08.750179+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:35:08.750179+0000","last_undegraded":"2023-10-30T13:35:08.750179+0000","last_fullsized":"2023-10-30T13:35:08.750179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:26.852057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743612+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:35:08.743612+0000","last_peered":"2023-10-30T13:35:08.743612+0000","last_clean":"2023-10-30T13:35:08.743612+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:35:08.743612+0000","last_undegraded":"2023-10-30T13:35:08.743612+0000","last_fullsized":"2023-10-30T13:35:08.743612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.749093+0000","last_change":"2023-10-30T13:35:07.805535+0000","last_active":"2023-10-30T13:35:08.749093+0000","last_peered":"2023-10-30T13:35:08.749093+0000","last_clean":"2023-10-30T13:35:08.749093+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:35:08.749093+0000","last_undegraded":"2023-10-30T13:35:08.749093+0000","last_fullsized":"2023-10-30T13:35:08.749093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:06.549889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.994784+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:35:10.994784+0000","last_peered":"2023-10-30T13:35:10.994784+0000","last_clean":"2023-10-30T13:35:10.994784+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:35:10.994784+0000","last_undegraded":"2023-10-30T13:35:10.994784+0000","last_fullsized":"2023-10-30T13:35:10.994784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.743474+0000","last_change":"2023-10-30T13:35:07.754171+0000","last_active":"2023-10-30T13:35:08.743474+0000","last_peered":"2023-10-30T13:35:08.743474+0000","last_clean":"2023-10-30T13:35:08.743474+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:35:08.743474+0000","last_undegraded":"2023-10-30T13:35:08.743474+0000","last_fullsized":"2023-10-30T13:35:08.743474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:50.849695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.801594+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:35:24.801594+0000","last_peered":"2023-10-30T13:35:24.801594+0000","last_clean":"2023-10-30T13:35:24.801594+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:35:24.801594+0000","last_undegraded":"2023-10-30T13:35:24.801594+0000","last_fullsized":"2023-10-30T13:35:24.801594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531754+0000","last_change":"2023-10-30T13:35:07.794804+0000","last_active":"2023-10-30T13:35:09.531754+0000","last_peered":"2023-10-30T13:35:09.531754+0000","last_clean":"2023-10-30T13:35:09.531754+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:35:09.531754+0000","last_undegraded":"2023-10-30T13:35:09.531754+0000","last_fullsized":"2023-10-30T13:35:09.531754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:00.913592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.355614+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:35:19.355614+0000","last_peered":"2023-10-30T13:35:19.355614+0000","last_clean":"2023-10-30T13:35:19.355614+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:35:19.355614+0000","last_undegraded":"2023-10-30T13:35:19.355614+0000","last_fullsized":"2023-10-30T13:35:19.355614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298809+0000","last_change":"2023-10-30T13:35:07.752077+0000","last_active":"2023-10-30T13:35:09.298809+0000","last_peered":"2023-10-30T13:35:09.298809+0000","last_clean":"2023-10-30T13:35:09.298809+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:35:09.298809+0000","last_undegraded":"2023-10-30T13:35:09.298809+0000","last_fullsized":"2023-10-30T13:35:09.298809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:07.661975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.350893+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:35:19.350893+0000","last_peered":"2023-10-30T13:35:19.350893+0000","last_clean":"2023-10-30T13:35:19.350893+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:35:19.350893+0000","last_undegraded":"2023-10-30T13:35:19.350893+0000","last_fullsized":"2023-10-30T13:35:19.350893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748158+0000","last_change":"2023-10-30T13:35:07.759389+0000","last_active":"2023-10-30T13:35:08.748158+0000","last_peered":"2023-10-30T13:35:08.748158+0000","last_clean":"2023-10-30T13:35:08.748158+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:35:08.748158+0000","last_undegraded":"2023-10-30T13:35:08.748158+0000","last_fullsized":"2023-10-30T13:35:08.748158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:15.536615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.351639+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:35:19.351639+0000","last_peered":"2023-10-30T13:35:19.351639+0000","last_clean":"2023-10-30T13:35:19.351639+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:35:19.351639+0000","last_undegraded":"2023-10-30T13:35:19.351639+0000","last_fullsized":"2023-10-30T13:35:19.351639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531288+0000","last_change":"2023-10-30T13:35:07.753673+0000","last_active":"2023-10-30T13:35:09.531288+0000","last_peered":"2023-10-30T13:35:09.531288+0000","last_clean":"2023-10-30T13:35:09.531288+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:35:09.531288+0000","last_undegraded":"2023-10-30T13:35:09.531288+0000","last_fullsized":"2023-10-30T13:35:09.531288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:27:18.430183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.236727+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:35:09.236727+0000","last_peered":"2023-10-30T13:35:09.236727+0000","last_clean":"2023-10-30T13:35:09.236727+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:35:09.236727+0000","last_undegraded":"2023-10-30T13:35:09.236727+0000","last_fullsized":"2023-10-30T13:35:09.236727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531465+0000","last_change":"2023-10-30T13:35:07.755302+0000","last_active":"2023-10-30T13:35:09.531465+0000","last_peered":"2023-10-30T13:35:09.531465+0000","last_clean":"2023-10-30T13:35:09.531465+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:35:09.531465+0000","last_undegraded":"2023-10-30T13:35:09.531465+0000","last_fullsized":"2023-10-30T13:35:09.531465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:31.543587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:19.352360+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:35:19.352360+0000","last_peered":"2023-10-30T13:35:19.352360+0000","last_clean":"2023-10-30T13:35:19.352360+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:35:19.352360+0000","last_undegraded":"2023-10-30T13:35:19.352360+0000","last_fullsized":"2023-10-30T13:35:19.352360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:10.992766+0000","last_change":"2023-10-30T13:35:07.780297+0000","last_active":"2023-10-30T13:35:10.992766+0000","last_peered":"2023-10-30T13:35:10.992766+0000","last_clean":"2023-10-30T13:35:10.992766+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:35:10.992766+0000","last_undegraded":"2023-10-30T13:35:10.992766+0000","last_fullsized":"2023-10-30T13:35:10.992766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:48.971885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748946+0000","last_change":"2023-10-30T13:35:07.806203+0000","last_active":"2023-10-30T13:35:08.748946+0000","last_peered":"2023-10-30T13:35:08.748946+0000","last_clean":"2023-10-30T13:35:08.748946+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:35:08.748946+0000","last_undegraded":"2023-10-30T13:35:08.748946+0000","last_fullsized":"2023-10-30T13:35:08.748946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:34.175218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.294221+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:35:13.294221+0000","last_peered":"2023-10-30T13:35:13.294221+0000","last_clean":"2023-10-30T13:35:13.294221+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:35:13.294221+0000","last_undegraded":"2023-10-30T13:35:13.294221+0000","last_fullsized":"2023-10-30T13:35:13.294221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:08.748564+0000","last_change":"2023-10-30T13:35:07.791319+0000","last_active":"2023-10-30T13:35:08.748564+0000","last_peered":"2023-10-30T13:35:08.748564+0000","last_clean":"2023-10-30T13:35:08.748564+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:35:08.748564+0000","last_undegraded":"2023-10-30T13:35:08.748564+0000","last_fullsized":"2023-10-30T13:35:08.748564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:42.560856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.799715+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:35:24.799715+0000","last_peered":"2023-10-30T13:35:24.799715+0000","last_clean":"2023-10-30T13:35:24.799715+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:35:24.799715+0000","last_undegraded":"2023-10-30T13:35:24.799715+0000","last_fullsized":"2023-10-30T13:35:24.799715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.298499+0000","last_change":"2023-10-30T13:35:07.790084+0000","last_active":"2023-10-30T13:35:09.298499+0000","last_peered":"2023-10-30T13:35:09.298499+0000","last_clean":"2023-10-30T13:35:09.298499+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:35:09.298499+0000","last_undegraded":"2023-10-30T13:35:09.298499+0000","last_fullsized":"2023-10-30T13:35:09.298499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:45.193072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:13.291592+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:35:13.291592+0000","last_peered":"2023-10-30T13:35:13.291592+0000","last_clean":"2023-10-30T13:35:13.291592+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:35:13.291592+0000","last_undegraded":"2023-10-30T13:35:13.291592+0000","last_fullsized":"2023-10-30T13:35:13.291592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:09.531695+0000","last_change":"2023-10-30T13:35:07.782341+0000","last_active":"2023-10-30T13:35:09.531695+0000","last_peered":"2023-10-30T13:35:09.531695+0000","last_clean":"2023-10-30T13:35:09.531695+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:35:09.531695+0000","last_undegraded":"2023-10-30T13:35:09.531695+0000","last_fullsized":"2023-10-30T13:35:09.531695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:19:06.392683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:35:24.804103+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:35:24.804103+0000","last_peered":"2023-10-30T13:35:24.804103+0000","last_clean":"2023-10-30T13:35:24.804103+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:35:24.804103+0000","last_undegraded":"2023-10-30T13:35:24.804103+0000","last_fullsized":"2023-10-30T13:35:24.804103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5738,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":664793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509452,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":75691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509452,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":73669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":73795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]}]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":74229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]}]},{"osd":4,"up_from":14,"seq":60129542155,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":73565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":663879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:35:37.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:37.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f8a9c640 1 Processor -- start 2023-10-30T13:35:37.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f8a9c640 1 -- start start 2023-10-30T13:35:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f8a9c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3f4153aa0 0x7fb3f415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f8a9c640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40c7b00 con 0x7fb3f406b3b0 2023-10-30T13:35:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f8a9c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40c2050 con 0x7fb3f40701e0 2023-10-30T13:35:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f8a9c640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f4070a50 con 0x7fb3f4153aa0 2023-10-30T13:35:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f259c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3f4153aa0 0x7fb3f415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f2d9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb3f406b3b0 0x7fb3f406b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43434/0 (socket says 172.21.15.31:43434) 2023-10-30T13:35:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f2d9d640 1 -- 172.21.15.31:0/1861955526 learned_addr learned my addr 172.21.15.31:0/1861955526 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f259c640 1 -- 172.21.15.31:0/1861955526 >> v1:172.21.15.33:6789/0 conn(0x7fb3f40701e0 legacy=0x7fb3f40ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f259c640 1 -- 172.21.15.31:0/1861955526 >> v1:172.21.15.31:6789/0 conn(0x7fb3f406b3b0 legacy=0x7fb3f406b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.716+0000 7fb3f259c640 1 -- 172.21.15.31:0/1861955526 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f40c2330 con 0x7fb3f4153aa0 2023-10-30T13:35:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f259c640 1 --2- 172.21.15.31:0/1861955526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f4153aa0 0x7fb3f415d840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40097c0 tx=0x7fb3e401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514d4dbb8f20c4b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f1d9b640 1 -- 172.21.15.31:0/1861955526 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40a8970 con 0x7fb3f4153aa0 2023-10-30T13:35:37.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f1d9b640 1 -- 172.21.15.31:0/1861955526 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e40a8ad0 con 0x7fb3f4153aa0 2023-10-30T13:35:37.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f1d9b640 1 -- 172.21.15.31:0/1861955526 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40a8e60 con 0x7fb3f4153aa0 2023-10-30T13:35:37.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1861955526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f4153aa0 msgr2=0x7fb3f415d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1861955526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f4153aa0 0x7fb3f415d840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40097c0 tx=0x7fb3e401ec70 comp rx=0 tx=0).stop 2023-10-30T13:35:37.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1861955526 shutdown_connections 2023-10-30T13:35:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1861955526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f4153aa0 0x7fb3f415d840 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1861955526 >> 172.21.15.31:0/1861955526 conn(0x7fb3f4071fb0 msgr2=0x7fb3f40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1861955526 shutdown_connections 2023-10-30T13:35:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.720+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1861955526 wait complete. 2023-10-30T13:35:37.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 Processor -- start 2023-10-30T13:35:37.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 -- start start 2023-10-30T13:35:37.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f406b3b0 0x7fb3f40ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f40701e0 0x7fb3f40cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3f4153aa0 0x7fb3f4156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f414e890 con 0x7fb3f406b3b0 2023-10-30T13:35:37.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4156280 con 0x7fb3f40701e0 2023-10-30T13:35:37.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f41552f0 con 0x7fb3f4153aa0 2023-10-30T13:35:37.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f406b3b0 0x7fb3f40ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f406b3b0 0x7fb3f40ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51392/0 (socket says 172.21.15.31:51392) 2023-10-30T13:35:37.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 -- 172.21.15.31:0/1293688471 learned_addr learned my addr 172.21.15.31:0/1293688471 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:37.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3ebfff640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f40701e0 0x7fb3f40cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f2d9d640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3f4153aa0 0x7fb3f4156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 -- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3f4153aa0 msgr2=0x7fb3f4156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3f4153aa0 0x7fb3f4156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 -- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f40701e0 msgr2=0x7fb3f40cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f40701e0 0x7fb3f40cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 -- 172.21.15.31:0/1293688471 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4009420 con 0x7fb3f406b3b0 2023-10-30T13:35:37.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f259c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f406b3b0 0x7fb3f40ccd60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40b3040 tx=0x7fb3e401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3eaffd640 1 -- 172.21.15.31:0/1293688471 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40b9630 con 0x7fb3f406b3b0 2023-10-30T13:35:37.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3eaffd640 1 -- 172.21.15.31:0/1293688471 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e40b9790 con 0x7fb3f406b3b0 2023-10-30T13:35:37.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f40cc710 con 0x7fb3f406b3b0 2023-10-30T13:35:37.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3eaffd640 1 -- 172.21.15.31:0/1293688471 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40b9b40 con 0x7fb3f406b3b0 2023-10-30T13:35:37.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.724+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f4155950 con 0x7fb3f406b3b0 2023-10-30T13:35:37.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.728+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f40672b0 con 0x7fb3f406b3b0 2023-10-30T13:35:37.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.728+0000 7fb3eaffd640 1 -- 172.21.15.31:0/1293688471 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3e40b7040 con 0x7fb3f406b3b0 2023-10-30T13:35:37.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.728+0000 7fb3eaffd640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3d4066520 0x7fb3d40689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.728+0000 7fb3eaffd640 1 -- 172.21.15.31:0/1293688471 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3e4121e30 con 0x7fb3f406b3b0 2023-10-30T13:35:37.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.728+0000 7fb3ebfff640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3d4066520 0x7fb3d40689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.728+0000 7fb3ebfff640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3d4066520 0x7fb3d40689e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb3f415a920 tx=0x7fb3dc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.732+0000 7fb3eaffd640 1 -- 172.21.15.31:0/1293688471 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3e40ef320 con 0x7fb3f406b3b0 2023-10-30T13:35:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3f404e170 con 0x7fb3f406b3b0 2023-10-30T13:35:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3eaffd640 1 -- 172.21.15.31:0/1293688471 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fb3e40b5030 con 0x7fb3f406b3b0 2023-10-30T13:35:37.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:37.896 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:37.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3d4066520 msgr2=0x7fb3d40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3d4066520 0x7fb3d40689e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb3f415a920 tx=0x7fb3dc009040 comp rx=0 tx=0).stop 2023-10-30T13:35:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f406b3b0 msgr2=0x7fb3f40ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f406b3b0 0x7fb3f40ccd60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40b3040 tx=0x7fb3e401ba10 comp rx=0 tx=0).stop 2023-10-30T13:35:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 shutdown_connections 2023-10-30T13:35:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3d4066520 0x7fb3d40689e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb3f4153aa0 0x7fb3f4156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f40701e0 0x7fb3f40cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 --2- 172.21.15.31:0/1293688471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f406b3b0 0x7fb3f40ccd60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 >> 172.21.15.31:0/1293688471 conn(0x7fb3f4071fb0 msgr2=0x7fb3f4152150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 shutdown_connections 2023-10-30T13:35:37.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:37.892+0000 7fb3f8a9c640 1 -- 172.21.15.31:0/1293688471 wait complete. 2023-10-30T13:35:37.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:35:05.304460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:35:06.652927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:35:37.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:35:38.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.060+0000 7f481b868640 1 Processor -- start 2023-10-30T13:35:38.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.060+0000 7f481b868640 1 -- start start 2023-10-30T13:35:38.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.060+0000 7f481b868640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f4814153ba0 0x7f48140cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.060+0000 7f481b868640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4814153f80 con 0x7f48140c4810 2023-10-30T13:35:38.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.060+0000 7f481b868640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4814159df0 con 0x7f48140c4410 2023-10-30T13:35:38.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.060+0000 7f481b868640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48141597c0 con 0x7f4814153ba0 2023-10-30T13:35:38.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819e05640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f48140c4810 0x7f48140cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43438/0 (socket says 172.21.15.31:43438) 2023-10-30T13:35:38.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819e05640 1 -- 172.21.15.31:0/82225031 learned_addr learned my addr 172.21.15.31:0/82225031 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:38.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819604640 1 --2- 172.21.15.31:0/82225031 >> v2:172.21.15.33:3300/0 conn(0x7f4814153ba0 0x7f48140cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977388617 0 0) 0x7f4814153f80 con 0x7f48140c4810 2023-10-30T13:35:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800003530 con 0x7f48140c4810 2023-10-30T13:35:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441521589 0 0) 0x7f4814159df0 con 0x7f48140c4410 2023-10-30T13:35:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4814153f80 con 0x7f48140c4410 2023-10-30T13:35:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255135527 0 0) 0x7f4800003530 con 0x7f48140c4810 2023-10-30T13:35:38.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 >> v2:172.21.15.33:3300/0 conn(0x7f4814153ba0 msgr2=0x7f48140cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:38.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 --2- 172.21.15.31:0/82225031 >> v2:172.21.15.33:3300/0 conn(0x7f4814153ba0 0x7f48140cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 >> v1:172.21.15.33:6789/0 conn(0x7f48140c4410 legacy=0x7f481415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481415d580 con 0x7f48140c4810 2023-10-30T13:35:38.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f480c001980 con 0x7f48140c4810 2023-10-30T13:35:38.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 >> v1:172.21.15.31:6789/0 conn(0x7f48140c4810 legacy=0x7f48140cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4800003b10 0x7f4800003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48000047d0 0x7f4800004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4800005190 0x7f480000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48141597c0 con 0x7f48000047d0 2023-10-30T13:35:38.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480000c0c0 con 0x7f4800005190 2023-10-30T13:35:38.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480000c280 con 0x7f4800003b10 2023-10-30T13:35:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4818e03640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48000047d0 0x7f4800004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819e05640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4800005190 0x7f480000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819604640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4800003b10 0x7f4800003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4818e03640 1 -- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4800003b10 msgr2=0x7f4800003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4818e03640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4800003b10 0x7f4800003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4818e03640 1 -- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4800005190 msgr2=0x7f480000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4818e03640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4800005190 0x7f480000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4818e03640 1 -- 172.21.15.31:0/82225031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481415d580 con 0x7f48000047d0 2023-10-30T13:35:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819e05640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4800005190 0x7f480000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819e05640 1 -- 172.21.15.31:0/82225031 reap_dead start 2023-10-30T13:35:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.064+0000 7f4819604640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4800003b10 0x7f4800003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:38.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f4818e03640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48000047d0 0x7f4800004c50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f48040018d0 tx=0x7f480401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9889859a7753595 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48040052d0 con 0x7f48000047d0 2023-10-30T13:35:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48040a7050 con 0x7f48000047d0 2023-10-30T13:35:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f480bfff640 1 -- 172.21.15.31:0/82225031 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804003600 con 0x7f48000047d0 2023-10-30T13:35:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 -- 172.21.15.31:0/82225031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48000047d0 msgr2=0x7f4800004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48000047d0 0x7f4800004c50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f48040018d0 tx=0x7f480401c510 comp rx=0 tx=0).stop 2023-10-30T13:35:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 -- 172.21.15.31:0/82225031 shutdown_connections 2023-10-30T13:35:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 --2- 172.21.15.31:0/82225031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48000047d0 0x7f4800004c50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 -- 172.21.15.31:0/82225031 >> 172.21.15.31:0/82225031 conn(0x7f4814071790 msgr2=0x7f4814071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:38.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 -- 172.21.15.31:0/82225031 shutdown_connections 2023-10-30T13:35:38.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 -- 172.21.15.31:0/82225031 wait complete. 2023-10-30T13:35:38.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 Processor -- start 2023-10-30T13:35:38.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.068+0000 7f481b868640 1 -- start start 2023-10-30T13:35:38.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48000047d0 0x7f481414d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48140cdf80 0x7f481414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4814152890 0x7f481414e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4814158a00 con 0x7f48140cdf80 2023-10-30T13:35:38.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4814153290 con 0x7f48000047d0 2023-10-30T13:35:38.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4814157ec0 con 0x7f4814152890 2023-10-30T13:35:38.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48140cdf80 0x7f481414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48140cdf80 0x7f481414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51404/0 (socket says 172.21.15.31:51404) 2023-10-30T13:35:38.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 -- 172.21.15.31:0/3745966390 learned_addr learned my addr 172.21.15.31:0/3745966390 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:38.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4819604640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48000047d0 0x7f481414d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4819e05640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4814152890 0x7f481414e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 -- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4814152890 msgr2=0x7f481414e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4814152890 0x7f481414e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 -- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48000047d0 msgr2=0x7f481414d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48000047d0 0x7f481414d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 -- 172.21.15.31:0/3745966390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4814158160 con 0x7f48140cdf80 2023-10-30T13:35:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4818e03640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48140cdf80 0x7f481414efe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4804000c10 tx=0x7f4800005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4809ffb640 1 -- 172.21.15.31:0/3745966390 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48040ba070 con 0x7f48140cdf80 2023-10-30T13:35:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4809ffb640 1 -- 172.21.15.31:0/3745966390 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4804007020 con 0x7f48140cdf80 2023-10-30T13:35:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f4809ffb640 1 -- 172.21.15.31:0/3745966390 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48040073d0 con 0x7f48140cdf80 2023-10-30T13:35:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4814158440 con 0x7f48140cdf80 2023-10-30T13:35:38.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481414c890 con 0x7f48140cdf80 2023-10-30T13:35:38.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.072+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481404e170 con 0x7f48140cdf80 2023-10-30T13:35:38.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.076+0000 7f4809ffb640 1 -- 172.21.15.31:0/3745966390 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4804007530 con 0x7f48140cdf80 2023-10-30T13:35:38.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.076+0000 7f4809ffb640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47f80664a0 0x7f47f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.076+0000 7f4809ffb640 1 -- 172.21.15.31:0/3745966390 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f480411d950 con 0x7f48140cdf80 2023-10-30T13:35:38.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.076+0000 7f4819604640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47f80664a0 0x7f47f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.076+0000 7f4819604640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47f80664a0 0x7f47f8068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f481404fb10 tx=0x7f4810008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.080+0000 7f4809ffb640 1 -- 172.21.15.31:0/3745966390 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48040ead30 con 0x7f48140cdf80 2023-10-30T13:35:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.236+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48140ceed0 con 0x7f48140cdf80 2023-10-30T13:35:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.236+0000 7f4809ffb640 1 -- 172.21.15.31:0/3745966390 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f48040ee350 con 0x7f48140cdf80 2023-10-30T13:35:38.241 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:35:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.236+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47f80664a0 msgr2=0x7f47f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.236+0000 7f481b868640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47f80664a0 0x7f47f8068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f481404fb10 tx=0x7f4810008040 comp rx=0 tx=0).stop 2023-10-30T13:35:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.236+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48140cdf80 msgr2=0x7f481414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.236+0000 7f481b868640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48140cdf80 0x7f481414efe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4804000c10 tx=0x7f4800005800 comp rx=0 tx=0).stop 2023-10-30T13:35:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 shutdown_connections 2023-10-30T13:35:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47f80664a0 0x7f47f8068960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4814152890 0x7f481414e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48140cdf80 0x7f481414efe0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 --2- 172.21.15.31:0/3745966390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48000047d0 0x7f481414d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 >> 172.21.15.31:0/3745966390 conn(0x7f4814071790 msgr2=0x7f48140ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 shutdown_connections 2023-10-30T13:35:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.240+0000 7f481b868640 1 -- 172.21.15.31:0/3745966390 wait complete. 2023-10-30T13:35:38.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:35:38.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f859b7c5640 1 Processor -- start 2023-10-30T13:35:38.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f859b7c5640 1 -- start start 2023-10-30T13:35:38.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f859b7c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 0x7f8594072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f859b7c5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85941581f0 con 0x7f85940725c0 2023-10-30T13:35:38.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f859b7c5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8594151e00 con 0x7f8594151a20 2023-10-30T13:35:38.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f859b7c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85941576d0 con 0x7f85940c1020 2023-10-30T13:35:38.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f85940725c0 0x7f85940c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43442/0 (socket says 172.21.15.31:43442) 2023-10-30T13:35:38.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 -- 172.21.15.31:0/2065586105 learned_addr learned my addr 172.21.15.31:0/2065586105 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:38.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8598d60640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 0x7f8594072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555985036 0 0) 0x7f85941581f0 con 0x7f85940725c0 2023-10-30T13:35:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8580003530 con 0x7f85940725c0 2023-10-30T13:35:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4916914 0 0) 0x7f8580003530 con 0x7f85940725c0 2023-10-30T13:35:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 msgr2=0x7f8594072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 0x7f8594072080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 >> v1:172.21.15.33:6789/0 conn(0x7f8594151a20 legacy=0x7f85940cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85940c19d0 con 0x7f85940725c0 2023-10-30T13:35:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f858c001a00 con 0x7f85940725c0 2023-10-30T13:35:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 >> v1:172.21.15.31:6789/0 conn(0x7f85940725c0 legacy=0x7f85940c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 0x7f85800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8580003ef0 0x7f85800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f8580005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858000b9c0 con 0x7f8580004d10 2023-10-30T13:35:38.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858000bb30 con 0x7f8580003ef0 2023-10-30T13:35:38.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858000bcf0 con 0x7f85940c1020 2023-10-30T13:35:38.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f8580005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8598d60640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8580003ef0 0x7f85800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599561640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 0x7f85800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 -- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 msgr2=0x7f85800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:38.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85940c1020 0x7f85800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 -- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8580003ef0 msgr2=0x7f85800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:38.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8580003ef0 0x7f85800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8599d62640 1 -- 172.21.15.31:0/2065586105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8580003530 con 0x7f8580004d10 2023-10-30T13:35:38.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.404+0000 7f8598d60640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8580003ef0 0x7f85800047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:38.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f8599d62640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f8580005190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f858c00a830 tx=0x7f858c01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3ff2600386f438 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c0ad630 con 0x7f8580004d10 2023-10-30T13:35:38.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858c0a8070 con 0x7f8580004d10 2023-10-30T13:35:38.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f858bfff640 1 -- 172.21.15.31:0/2065586105 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c0ada30 con 0x7f8580004d10 2023-10-30T13:35:38.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f859b7c5640 1 -- 172.21.15.31:0/2065586105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 msgr2=0x7f8580005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f859b7c5640 1 --2- 172.21.15.31:0/2065586105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f8580005190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f858c00a830 tx=0x7f858c01d340 comp rx=0 tx=0).stop 2023-10-30T13:35:38.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f8599d62640 1 -- 172.21.15.31:0/2065586105 reap_dead start 2023-10-30T13:35:38.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f859b7c5640 1 -- 172.21.15.31:0/2065586105 shutdown_connections 2023-10-30T13:35:38.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f859b7c5640 1 -- 172.21.15.31:0/2065586105 >> 172.21.15.31:0/2065586105 conn(0x7f859406da40 msgr2=0x7f859406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:38.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f859b7c5640 1 -- 172.21.15.31:0/2065586105 shutdown_connections 2023-10-30T13:35:38.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.408+0000 7f859b7c5640 1 -- 172.21.15.31:0/2065586105 wait complete. 2023-10-30T13:35:38.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 Processor -- start 2023-10-30T13:35:38.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 -- start start 2023-10-30T13:35:38.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f85941f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f859414cc80 0x7f8594070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8594071520 0x7f85940c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85941ef870 con 0x7f8580004d10 2023-10-30T13:35:38.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85941f30b0 con 0x7f8594071520 2023-10-30T13:35:38.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85941eec60 con 0x7f859414cc80 2023-10-30T13:35:38.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f85941f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f85941f2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51422/0 (socket says 172.21.15.31:51422) 2023-10-30T13:35:38.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 -- 172.21.15.31:0/1572555957 learned_addr learned my addr 172.21.15.31:0/1572555957 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:38.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8598d60640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f859414cc80 0x7f8594070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599d62640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8594071520 0x7f85940c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 -- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f859414cc80 msgr2=0x7f8594070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f859414cc80 0x7f8594070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 -- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8594071520 msgr2=0x7f85940c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8594071520 0x7f85940c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 -- 172.21.15.31:0/1572555957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85941eef40 con 0x7f8580004d10 2023-10-30T13:35:38.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8598d60640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f859414cc80 0x7f8594070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:38.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8599561640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f85941f2b70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f859000e5d0 tx=0x7f859000ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8589ffb640 1 -- 172.21.15.31:0/1572555957 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85900053a0 con 0x7f8580004d10 2023-10-30T13:35:38.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8589ffb640 1 -- 172.21.15.31:0/1572555957 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8590003600 con 0x7f8580004d10 2023-10-30T13:35:38.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f8589ffb640 1 -- 172.21.15.31:0/1572555957 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85900039b0 con 0x7f8580004d10 2023-10-30T13:35:38.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85941ef220 con 0x7f8580004d10 2023-10-30T13:35:38.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.412+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85940719b0 con 0x7f8580004d10 2023-10-30T13:35:38.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.416+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859404e170 con 0x7f8580004d10 2023-10-30T13:35:38.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.416+0000 7f8589ffb640 1 -- 172.21.15.31:0/1572555957 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8590003b10 con 0x7f8580004d10 2023-10-30T13:35:38.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.416+0000 7f8589ffb640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85780664a0 0x7f8578068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.416+0000 7f8589ffb640 1 -- 172.21.15.31:0/1572555957 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8590083d80 con 0x7f8580004d10 2023-10-30T13:35:38.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.416+0000 7f8598d60640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85780664a0 0x7f8578068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.416+0000 7f8598d60640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85780664a0 0x7f8578068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f85941e9410 tx=0x7f8584008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.420+0000 7f8589ffb640 1 -- 172.21.15.31:0/1572555957 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85900511f0 con 0x7f8580004d10 2023-10-30T13:35:38.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.576+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85940c0040 con 0x7f8580004d10 2023-10-30T13:35:38.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.576+0000 7f8589ffb640 1 -- 172.21.15.31:0/1572555957 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f859001a310 con 0x7f8580004d10 2023-10-30T13:35:38.581 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-10-30T13:35:38.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85780664a0 msgr2=0x7f8578068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85780664a0 0x7f8578068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f85941e9410 tx=0x7f8584008040 comp rx=0 tx=0).stop 2023-10-30T13:35:38.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 msgr2=0x7f85941f2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f85941f2b70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f859000e5d0 tx=0x7f859000ea90 comp rx=0 tx=0).stop 2023-10-30T13:35:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 shutdown_connections 2023-10-30T13:35:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85780664a0 0x7f8578068960 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8594071520 0x7f85940c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f859414cc80 0x7f8594070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 --2- 172.21.15.31:0/1572555957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8580004d10 0x7f85941f2b70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 >> 172.21.15.31:0/1572555957 conn(0x7f859406da40 msgr2=0x7f8594156620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 shutdown_connections 2023-10-30T13:35:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.580+0000 7f859b7c5640 1 -- 172.21.15.31:0/1572555957 wait complete. 2023-10-30T13:35:38.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:35:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.744+0000 7f2c22d3e640 1 Processor -- start 2023-10-30T13:35:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.744+0000 7f2c22d3e640 1 -- start start 2023-10-30T13:35:38.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.744+0000 7f2c22d3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2c1c0c74a0 0x7f2c1c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.744+0000 7f2c22d3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c1581f0 con 0x7f2c1c151a20 2023-10-30T13:35:38.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.744+0000 7f2c22d3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c151e00 con 0x7f2c1c0c1020 2023-10-30T13:35:38.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.744+0000 7f2c22d3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c1576d0 con 0x7f2c1c0c74a0 2023-10-30T13:35:38.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c212db640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2c1c0c74a0 0x7f2c1c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c20ada640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2c1c151a20 0x7f2c1c0cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43446/0 (socket says 172.21.15.31:43446) 2023-10-30T13:35:38.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c20ada640 1 -- 172.21.15.31:0/4138479432 learned_addr learned my addr 172.21.15.31:0/4138479432 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:38.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c212db640 1 -- 172.21.15.31:0/4138479432 >> v1:172.21.15.33:6789/0 conn(0x7f2c1c0c1020 legacy=0x7f2c1c072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c212db640 1 -- 172.21.15.31:0/4138479432 >> v1:172.21.15.31:6789/0 conn(0x7f2c1c151a20 legacy=0x7f2c1c0cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c212db640 1 -- 172.21.15.31:0/4138479432 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0c19d0 con 0x7f2c1c0c74a0 2023-10-30T13:35:38.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c1b7fe640 1 -- 172.21.15.31:0/4138479432 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743808282 0 0) 0x7f2c1c1581f0 con 0x7f2c1c151a20 2023-10-30T13:35:38.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c212db640 1 --2- 172.21.15.31:0/4138479432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c74a0 0x7f2c1c0c78d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009d90 tx=0x7f2c1401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6a640ab3f3ffef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c1b7fe640 1 -- 172.21.15.31:0/4138479432 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c140af600 con 0x7f2c1c0c74a0 2023-10-30T13:35:38.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c1b7fe640 1 -- 172.21.15.31:0/4138479432 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c140ab050 con 0x7f2c1c0c74a0 2023-10-30T13:35:38.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.748+0000 7f2c1b7fe640 1 -- 172.21.15.31:0/4138479432 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c140af980 con 0x7f2c1c0c74a0 2023-10-30T13:35:38.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- 172.21.15.31:0/4138479432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c74a0 msgr2=0x7f2c1c0c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/4138479432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c74a0 0x7f2c1c0c78d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009d90 tx=0x7f2c1401f130 comp rx=0 tx=0).stop 2023-10-30T13:35:38.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- 172.21.15.31:0/4138479432 shutdown_connections 2023-10-30T13:35:38.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/4138479432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c74a0 0x7f2c1c0c78d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- 172.21.15.31:0/4138479432 >> 172.21.15.31:0/4138479432 conn(0x7f2c1c06da40 msgr2=0x7f2c1c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:38.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- 172.21.15.31:0/4138479432 shutdown_connections 2023-10-30T13:35:38.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- 172.21.15.31:0/4138479432 wait complete. 2023-10-30T13:35:38.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 Processor -- start 2023-10-30T13:35:38.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- start start 2023-10-30T13:35:38.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c1020 0x7f2c1c14eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c1c0c74a0 0x7f2c1c14c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c1c151a20 0x7f2c1c1f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c48f0 con 0x7f2c1c0c1020 2023-10-30T13:35:38.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c4120 con 0x7f2c1c151a20 2023-10-30T13:35:38.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c22d3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c4350 con 0x7f2c1c0c74a0 2023-10-30T13:35:38.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c1020 0x7f2c1c14eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c1020 0x7f2c1c14eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51432/0 (socket says 172.21.15.31:51432) 2023-10-30T13:35:38.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 -- 172.21.15.31:0/2892793913 learned_addr learned my addr 172.21.15.31:0/2892793913 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:38.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c1bfff640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c1c0c74a0 0x7f2c1c14c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c212db640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c1c151a20 0x7f2c1c1f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 -- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c1c0c74a0 msgr2=0x7f2c1c14c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c1c0c74a0 0x7f2c1c14c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 -- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c1c151a20 msgr2=0x7f2c1c1f00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c1c151a20 0x7f2c1c1f00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.752+0000 7f2c20ada640 1 -- 172.21.15.31:0/2892793913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c14009a70 con 0x7f2c1c0c1020 2023-10-30T13:35:38.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c20ada640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c1020 0x7f2c1c14eca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c4a0 tx=0x7f2c1000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c197fa640 1 -- 172.21.15.31:0/2892793913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100074e0 con 0x7f2c1c0c1020 2023-10-30T13:35:38.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c197fa640 1 -- 172.21.15.31:0/2892793913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c10007640 con 0x7f2c1c0c1020 2023-10-30T13:35:38.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c197fa640 1 -- 172.21.15.31:0/2892793913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10004a30 con 0x7f2c1c0c1020 2023-10-30T13:35:38.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c0c4610 con 0x7f2c1c0c1020 2023-10-30T13:35:38.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c14d840 con 0x7f2c1c0c1020 2023-10-30T13:35:38.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c14d3c0 con 0x7f2c1c0c1020 2023-10-30T13:35:38.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c197fa640 1 -- 172.21.15.31:0/2892793913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c10004b90 con 0x7f2c1c0c1020 2023-10-30T13:35:38.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c197fa640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c000664a0 0x7f2c00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:38.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c197fa640 1 -- 172.21.15.31:0/2892793913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c10083180 con 0x7f2c1c0c1020 2023-10-30T13:35:38.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.756+0000 7f2c1bfff640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c000664a0 0x7f2c00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:38.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.760+0000 7f2c1bfff640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c000664a0 0x7f2c00068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0c4a50 tx=0x7f2c08009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.760+0000 7f2c197fa640 1 -- 172.21.15.31:0/2892793913 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c1c14d3c0 con 0x7f2c1c0c1020 2023-10-30T13:35:38.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.916+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2c1c158660 con 0x7f2c1c0c1020 2023-10-30T13:35:38.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.916+0000 7f2c197fa640 1 -- 172.21.15.31:0/2892793913 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2c1c158660 con 0x7f2c1c0c1020 2023-10-30T13:35:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-10-30T13:35:38.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c000664a0 msgr2=0x7f2c00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c000664a0 0x7f2c00068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0c4a50 tx=0x7f2c08009040 comp rx=0 tx=0).stop 2023-10-30T13:35:38.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c1020 msgr2=0x7f2c1c14eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:38.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c1020 0x7f2c1c14eca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c4a0 tx=0x7f2c1000c960 comp rx=0 tx=0).stop 2023-10-30T13:35:38.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 shutdown_connections 2023-10-30T13:35:38.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c000664a0 0x7f2c00068960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c1c151a20 0x7f2c1c1f00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c1c0c74a0 0x7f2c1c14c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 --2- 172.21.15.31:0/2892793913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1c0c1020 0x7f2c1c14eca0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:38.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 >> 172.21.15.31:0/2892793913 conn(0x7f2c1c06da40 msgr2=0x7f2c1c0c5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:38.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 shutdown_connections 2023-10-30T13:35:38.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:38.920+0000 7f2c22d3e640 1 -- 172.21.15.31:0/2892793913 wait complete. 2023-10-30T13:35:38.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:39.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d4046640 1 Processor -- start 2023-10-30T13:35:39.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d4046640 1 -- start start 2023-10-30T13:35:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d4046640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f10cc151840 0x7f10cc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d4046640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f10cc151c20 0x7f10cc072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d4046640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0c5030 con 0x7f10cc0cb8b0 2023-10-30T13:35:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d4046640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10cc157830 con 0x7f10cc151840 2023-10-30T13:35:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d4046640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10cc157b10 con 0x7f10cc151c20 2023-10-30T13:35:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d25e3640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f10cc0cb8b0 0x7f10cc06b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43450/0 (socket says 172.21.15.31:43450) 2023-10-30T13:35:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d1de2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f10cc151840 0x7f10cc0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d25e3640 1 -- 172.21.15.31:0/2593608828 learned_addr learned my addr 172.21.15.31:0/2593608828 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:39.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d15e1640 1 --2- 172.21.15.31:0/2593608828 >> v2:172.21.15.33:3300/0 conn(0x7f10cc151c20 0x7f10cc072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d1de2640 1 -- 172.21.15.31:0/2593608828 >> v2:172.21.15.33:3300/0 conn(0x7f10cc151c20 msgr2=0x7f10cc072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:39.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d1de2640 1 --2- 172.21.15.31:0/2593608828 >> v2:172.21.15.33:3300/0 conn(0x7f10cc151c20 0x7f10cc072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d1de2640 1 -- 172.21.15.31:0/2593608828 >> v1:172.21.15.31:6789/0 conn(0x7f10cc0cb8b0 legacy=0x7f10cc06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d1de2640 1 -- 172.21.15.31:0/2593608828 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc157df0 con 0x7f10cc151840 2023-10-30T13:35:39.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d0de0640 1 -- 172.21.15.31:0/2593608828 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324349407 0 0) 0x7f10cc0c5030 con 0x7f10cc0cb8b0 2023-10-30T13:35:39.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.084+0000 7f10d15e1640 1 --2- 172.21.15.31:0/2593608828 >> v2:172.21.15.33:3300/0 conn(0x7f10cc151c20 0x7f10cc072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:39.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d1de2640 1 --2- 172.21.15.31:0/2593608828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc0c49a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10cc06d4d0 tx=0x7f10c801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873cd7f9ba1d70e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d0de0640 1 -- 172.21.15.31:0/2593608828 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80a8950 con 0x7f10cc151840 2023-10-30T13:35:39.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d0de0640 1 -- 172.21.15.31:0/2593608828 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c80a8ab0 con 0x7f10cc151840 2023-10-30T13:35:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d0de0640 1 -- 172.21.15.31:0/2593608828 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80a8e40 con 0x7f10cc151840 2023-10-30T13:35:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 -- 172.21.15.31:0/2593608828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 msgr2=0x7f10cc0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 --2- 172.21.15.31:0/2593608828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc0c49a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10cc06d4d0 tx=0x7f10c801ec50 comp rx=0 tx=0).stop 2023-10-30T13:35:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 -- 172.21.15.31:0/2593608828 shutdown_connections 2023-10-30T13:35:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 --2- 172.21.15.31:0/2593608828 >> v2:172.21.15.33:3300/0 conn(0x7f10cc151c20 0x7f10cc072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 --2- 172.21.15.31:0/2593608828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc0c49a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 -- 172.21.15.31:0/2593608828 >> 172.21.15.31:0/2593608828 conn(0x7f10cc0cac70 msgr2=0x7f10cc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 -- 172.21.15.31:0/2593608828 shutdown_connections 2023-10-30T13:35:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.088+0000 7f10d4046640 1 -- 172.21.15.31:0/2593608828 wait complete. 2023-10-30T13:35:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 Processor -- start 2023-10-30T13:35:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 -- start start 2023-10-30T13:35:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10cc0cb8b0 0x7f10cc1448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc1ec650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f10cc151c20 0x7f10cc1ecb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0c4ae0 con 0x7f10cc151840 2023-10-30T13:35:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc1ed0d0 con 0x7f10cc0cb8b0 2023-10-30T13:35:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc14be60 con 0x7f10cc151c20 2023-10-30T13:35:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc1ec650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc1ec650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51448/0 (socket says 172.21.15.31:51448) 2023-10-30T13:35:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 -- 172.21.15.31:0/1851144975 learned_addr learned my addr 172.21.15.31:0/1851144975 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d1de2640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10cc0cb8b0 0x7f10cc1448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d25e3640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f10cc151c20 0x7f10cc1ecb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 -- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f10cc151c20 msgr2=0x7f10cc1ecb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f10cc151c20 0x7f10cc1ecb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 -- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10cc0cb8b0 msgr2=0x7f10cc1448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10cc0cb8b0 0x7f10cc1448c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 -- 172.21.15.31:0/1851144975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8009420 con 0x7f10cc151840 2023-10-30T13:35:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d15e1640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc1ec650 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00d800 tx=0x7f10bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10c27fc640 1 -- 172.21.15.31:0/1851144975 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0096c0 con 0x7f10cc151840 2023-10-30T13:35:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10c27fc640 1 -- 172.21.15.31:0/1851144975 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc009820 con 0x7f10cc151840 2023-10-30T13:35:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10c27fc640 1 -- 172.21.15.31:0/1851144975 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc009bd0 con 0x7f10cc151840 2023-10-30T13:35:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc1f0700 con 0x7f10cc151840 2023-10-30T13:35:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc14baf0 con 0x7f10cc151840 2023-10-30T13:35:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.092+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc14c010 con 0x7f10cc151840 2023-10-30T13:35:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.096+0000 7f10c27fc640 1 -- 172.21.15.31:0/1851144975 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10bc009d30 con 0x7f10cc151840 2023-10-30T13:35:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.096+0000 7f10c27fc640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f10b00664a0 0x7f10b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.096+0000 7f10c27fc640 1 -- 172.21.15.31:0/1851144975 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10bc0845d0 con 0x7f10cc151840 2023-10-30T13:35:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.096+0000 7f10d1de2640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f10b00664a0 0x7f10b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.096+0000 7f10d1de2640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f10b00664a0 0x7f10b0068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f10c80b2000 tx=0x7f10c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.100+0000 7f10c27fc640 1 -- 172.21.15.31:0/1851144975 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10bc051a40 con 0x7f10cc151840 2023-10-30T13:35:39.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.288+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10cc04e170 con 0x7f10cc151840 2023-10-30T13:35:39.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10c27fc640 1 -- 172.21.15.31:0/1851144975 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7f10bc055060 con 0x7f10cc151840 2023-10-30T13:35:39.295 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:39.295 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:39.295 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:35:39.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f10b00664a0 msgr2=0x7f10b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f10b00664a0 0x7f10b0068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f10c80b2000 tx=0x7f10c8008040 comp rx=0 tx=0).stop 2023-10-30T13:35:39.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 msgr2=0x7f10cc1ec650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc1ec650 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00d800 tx=0x7f10bc00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 shutdown_connections 2023-10-30T13:35:39.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f10b00664a0 0x7f10b0068960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f10cc151c20 0x7f10cc1ecb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc151840 0x7f10cc1ec650 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 --2- 172.21.15.31:0/1851144975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10cc0cb8b0 0x7f10cc1448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 >> 172.21.15.31:0/1851144975 conn(0x7f10cc0cac70 msgr2=0x7f10cc0cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:39.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 shutdown_connections 2023-10-30T13:35:39.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.292+0000 7f10d4046640 1 -- 172.21.15.31:0/1851144975 wait complete. 2023-10-30T13:35:39.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:35:39.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb7572640 1 Processor -- start 2023-10-30T13:35:39.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb7572640 1 -- start start 2023-10-30T13:35:39.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb7572640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0fb0151840 0x7f0fb00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb7572640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0fb00cb8b0 0x7f0fb006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb7572640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb00c5030 con 0x7f0fb0151c20 2023-10-30T13:35:39.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb7572640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb0157830 con 0x7f0fb0151840 2023-10-30T13:35:39.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb7572640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb0157b10 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb530e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0fb0151840 0x7f0fb00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb4b0d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f0fb0151c20 0x7f0fb0072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43464/0 (socket says 172.21.15.31:43464) 2023-10-30T13:35:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb4b0d640 1 -- 172.21.15.31:0/3884633805 learned_addr learned my addr 172.21.15.31:0/3884633805 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:35:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb5b0f640 1 --2- 172.21.15.31:0/3884633805 >> v2:172.21.15.33:3300/0 conn(0x7f0fb00cb8b0 0x7f0fb006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb530e640 1 -- 172.21.15.31:0/3884633805 >> v2:172.21.15.33:3300/0 conn(0x7f0fb00cb8b0 msgr2=0x7f0fb006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb530e640 1 --2- 172.21.15.31:0/3884633805 >> v2:172.21.15.33:3300/0 conn(0x7f0fb00cb8b0 0x7f0fb006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb530e640 1 -- 172.21.15.31:0/3884633805 >> v1:172.21.15.31:6789/0 conn(0x7f0fb0151c20 legacy=0x7f0fb0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fa7fff640 1 -- 172.21.15.31:0/3884633805 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938188225 0 0) 0x7f0fb00c5030 con 0x7f0fb0151c20 2023-10-30T13:35:39.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.456+0000 7f0fb530e640 1 -- 172.21.15.31:0/3884633805 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0157df0 con 0x7f0fb0151840 2023-10-30T13:35:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb530e640 1 --2- 172.21.15.31:0/3884633805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb0151840 0x7f0fb00c49a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00c6cb0 tx=0x7f0fac01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7bb2f4b25734d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fa7fff640 1 -- 172.21.15.31:0/3884633805 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0a8950 con 0x7f0fb0151840 2023-10-30T13:35:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fa7fff640 1 -- 172.21.15.31:0/3884633805 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac0a8ab0 con 0x7f0fb0151840 2023-10-30T13:35:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fa7fff640 1 -- 172.21.15.31:0/3884633805 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0a8e40 con 0x7f0fb0151840 2023-10-30T13:35:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 -- 172.21.15.31:0/3884633805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb0151840 msgr2=0x7f0fb00c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 --2- 172.21.15.31:0/3884633805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb0151840 0x7f0fb00c49a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00c6cb0 tx=0x7f0fac01ec50 comp rx=0 tx=0).stop 2023-10-30T13:35:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 -- 172.21.15.31:0/3884633805 shutdown_connections 2023-10-30T13:35:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 --2- 172.21.15.31:0/3884633805 >> v2:172.21.15.33:3300/0 conn(0x7f0fb00cb8b0 0x7f0fb006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 --2- 172.21.15.31:0/3884633805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb0151840 0x7f0fb00c49a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 -- 172.21.15.31:0/3884633805 >> 172.21.15.31:0/3884633805 conn(0x7f0fb00cac70 msgr2=0x7f0fb00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 -- 172.21.15.31:0/3884633805 shutdown_connections 2023-10-30T13:35:39.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.460+0000 7f0fb7572640 1 -- 172.21.15.31:0/3884633805 wait complete. 2023-10-30T13:35:39.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 Processor -- start 2023-10-30T13:35:39.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 -- start start 2023-10-30T13:35:39.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb00cb8b0 0x7f0fb01e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fb0151840 0x7f0fb0156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fb0151c20 0x7f0fb014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0150000 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0156c40 con 0x7f0fb0151c20 2023-10-30T13:35:39.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0070460 con 0x7f0fb0151840 2023-10-30T13:35:39.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb00cb8b0 0x7f0fb01e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb00cb8b0 0x7f0fb01e33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51464/0 (socket says 172.21.15.31:51464) 2023-10-30T13:35:39.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 -- 172.21.15.31:0/1223908723 learned_addr learned my addr 172.21.15.31:0/1223908723 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:35:39.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb4b0d640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fb0151840 0x7f0fb0156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb5b0f640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fb0151c20 0x7f0fb014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 -- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fb0151840 msgr2=0x7f0fb0156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fb0151840 0x7f0fb0156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 -- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fb0151c20 msgr2=0x7f0fb014f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fb0151c20 0x7f0fb014f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 -- 172.21.15.31:0/1223908723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac009420 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb530e640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb00cb8b0 0x7f0fb01e33a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009880 tx=0x7f0fac01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fa5ffb640 1 -- 172.21.15.31:0/1223908723 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0b1df0 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fa5ffb640 1 -- 172.21.15.31:0/1223908723 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac0b9410 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fa5ffb640 1 -- 172.21.15.31:0/1223908723 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0b97a0 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb01f21c0 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.464+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb01f2780 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.468+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb004e170 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.468+0000 7f0fa5ffb640 1 -- 172.21.15.31:0/1223908723 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fac0b7040 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.468+0000 7f0fa5ffb640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f940664a0 0x7f0f94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.468+0000 7f0fa5ffb640 1 -- 172.21.15.31:0/1223908723 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fac121900 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.468+0000 7f0fb4b0d640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f940664a0 0x7f0f94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.468+0000 7f0fb4b0d640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f940664a0 0x7f0f94068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01e3d00 tx=0x7f0fa0007c10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.472+0000 7f0fa5ffb640 1 -- 172.21.15.31:0/1223908723 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fac0eed80 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.636+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fb00c6730 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.636+0000 7f0fa5ffb640 1 -- 172.21.15.31:0/1223908723 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f0fac0f23a0 con 0x7f0fb00cb8b0 2023-10-30T13:35:39.641 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:35:39.641 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:35:08.735927+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:35:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f940664a0 msgr2=0x7f0f94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f940664a0 0x7f0f94068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01e3d00 tx=0x7f0fa0007c10 comp rx=0 tx=0).stop 2023-10-30T13:35:39.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb00cb8b0 msgr2=0x7f0fb01e33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb00cb8b0 0x7f0fb01e33a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009880 tx=0x7f0fac01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 shutdown_connections 2023-10-30T13:35:39.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f940664a0 0x7f0f94068960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fb0151c20 0x7f0fb014f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fb0151840 0x7f0fb0156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 --2- 172.21.15.31:0/1223908723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb00cb8b0 0x7f0fb01e33a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 >> 172.21.15.31:0/1223908723 conn(0x7f0fb00cac70 msgr2=0x7f0fb0150560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 shutdown_connections 2023-10-30T13:35:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:35:39.640+0000 7f0fb7572640 1 -- 172.21.15.31:0/1223908723 wait complete. 2023-10-30T13:35:39.663 DEBUG:tasks.mds_thrash:Yielding 2023-10-30T13:35:39.663 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T13:35:39.675 INFO:tasks.workunit:Pulling workunits from ref 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:35:39.675 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T13:35:39.676 INFO:tasks.workunit:timeout=6h 2023-10-30T13:35:39.676 INFO:tasks.workunit:cleanup=True 2023-10-30T13:35:39.676 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:39.677 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-30T13:35:39.677 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2023-10-30T13:35:39.687 INFO:teuthology.orchestra.run.smithi031.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:39.687 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T13:35:39.687 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-10-30T13:35:39.688 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:35:39.688 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-30T13:35:39.688 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-30 13:35:13.275040619 +0000 2023-10-30T13:35:39.688 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-30 13:35:32.848855401 +0000 2023-10-30T13:35:39.688 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-30 13:35:13.275040619 +0000 2023-10-30T13:35:39.688 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-30T13:35:39.689 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-30T13:35:39.760 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:35:39.816 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T13:35:54.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:54.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:10.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:10.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.824+0000 7f33bc9d3640 1 Processor -- start 2023-10-30T13:36:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.824+0000 7f33bc9d3640 1 -- start start 2023-10-30T13:36:10.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.824+0000 7f33bc9d3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f33b8153c80 0x7f33b815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.824+0000 7f33bc9d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b80c4a10 0x7f33b80c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.824+0000 7f33bc9d3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b8159f20 con 0x7f33b815a3b0 2023-10-30T13:36:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.824+0000 7f33bc9d3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b815dec0 con 0x7f33b8153c80 2023-10-30T13:36:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.824+0000 7f33bc9d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b81599a0 con 0x7f33b80c4a10 2023-10-30T13:36:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f33b8153c80 0x7f33b815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f33b8153c80 0x7f33b815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39816/0 (socket says 172.21.15.31:39816) 2023-10-30T13:36:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 -- 172.21.15.31:0/4015397348 learned_addr learned my addr 172.21.15.31:0/4015397348 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b5d9b640 1 --2- 172.21.15.31:0/4015397348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b80c4a10 0x7f33b80c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 -- 172.21.15.31:0/4015397348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b80c4a10 msgr2=0x7f33b80c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:10.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 --2- 172.21.15.31:0/4015397348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b80c4a10 0x7f33b80c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:10.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 -- 172.21.15.31:0/4015397348 >> v1:172.21.15.33:6789/0 conn(0x7f33b815a3b0 legacy=0x7f33b80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:10.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 -- 172.21.15.31:0/4015397348 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80c47e0 con 0x7f33b8153c80 2023-10-30T13:36:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b659c640 1 --2- 172.21.15.31:0/4015397348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b8153c80 0x7f33b815cf90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009870 tx=0x7f33a801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e3530dff3993d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b559a640 1 -- 172.21.15.31:0/4015397348 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a80a8a30 con 0x7f33b8153c80 2023-10-30T13:36:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b559a640 1 -- 172.21.15.31:0/4015397348 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a80a8b90 con 0x7f33b8153c80 2023-10-30T13:36:10.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.828+0000 7f33b559a640 1 -- 172.21.15.31:0/4015397348 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a80b0440 con 0x7f33b8153c80 2023-10-30T13:36:10.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- 172.21.15.31:0/4015397348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b8153c80 msgr2=0x7f33b815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/4015397348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b8153c80 0x7f33b815cf90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009870 tx=0x7f33a801ed30 comp rx=0 tx=0).stop 2023-10-30T13:36:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- 172.21.15.31:0/4015397348 shutdown_connections 2023-10-30T13:36:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/4015397348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b80c4a10 0x7f33b80c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/4015397348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b8153c80 0x7f33b815cf90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- 172.21.15.31:0/4015397348 >> 172.21.15.31:0/4015397348 conn(0x7f33b80715c0 msgr2=0x7f33b80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:10.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- 172.21.15.31:0/4015397348 shutdown_connections 2023-10-30T13:36:10.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- 172.21.15.31:0/4015397348 wait complete. 2023-10-30T13:36:10.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 Processor -- start 2023-10-30T13:36:10.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- start start 2023-10-30T13:36:10.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33b80c4a10 0x7f33b806c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:10.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b8153c80 0x7f33b81ef310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:10.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b815a3b0 0x7f33b80cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:10.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80ce6c0 con 0x7f33b815a3b0 2023-10-30T13:36:10.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80ce960 con 0x7f33b80c4a10 2023-10-30T13:36:10.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33bc9d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80cec40 con 0x7f33b8153c80 2023-10-30T13:36:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.832+0000 7f33b6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b815a3b0 0x7f33b80cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b815a3b0 0x7f33b80cf6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39822/0 (socket says 172.21.15.31:39822) 2023-10-30T13:36:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 -- 172.21.15.31:0/3408882705 learned_addr learned my addr 172.21.15.31:0/3408882705 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b5d9b640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b8153c80 0x7f33b81ef310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:10.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b659c640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33b80c4a10 0x7f33b806c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:10.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 -- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b8153c80 msgr2=0x7f33b81ef310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:10.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b8153c80 0x7f33b81ef310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:10.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 -- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33b80c4a10 msgr2=0x7f33b806c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:10.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33b80c4a10 0x7f33b806c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:10.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 -- 172.21.15.31:0/3408882705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a80094d0 con 0x7f33b815a3b0 2023-10-30T13:36:10.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33b6d9d640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b815a3b0 0x7f33b80cf6a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f33b000c730 tx=0x7f33b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:10.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33a6ffd640 1 -- 172.21.15.31:0/3408882705 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b0003b80 con 0x7f33b815a3b0 2023-10-30T13:36:10.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33a6ffd640 1 -- 172.21.15.31:0/3408882705 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b0003ce0 con 0x7f33b815a3b0 2023-10-30T13:36:10.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33a6ffd640 1 -- 172.21.15.31:0/3408882705 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b000aa30 con 0x7f33b815a3b0 2023-10-30T13:36:10.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b80cef20 con 0x7f33b815a3b0 2023-10-30T13:36:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b81eedf0 con 0x7f33b815a3b0 2023-10-30T13:36:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b806ceb0 con 0x7f33b815a3b0 2023-10-30T13:36:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.836+0000 7f33a6ffd640 1 -- 172.21.15.31:0/3408882705 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33b000ab90 con 0x7f33b815a3b0 2023-10-30T13:36:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.840+0000 7f33a6ffd640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33980664a0 0x7f3398068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.840+0000 7f33a6ffd640 1 -- 172.21.15.31:0/3408882705 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f33b0083cb0 con 0x7f33b815a3b0 2023-10-30T13:36:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.840+0000 7f33b659c640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33980664a0 0x7f3398068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.840+0000 7f33b659c640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33980664a0 0x7f3398068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33a80b2000 tx=0x7f33a8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:10.844+0000 7f33a6ffd640 1 -- 172.21.15.31:0/3408882705 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33b0051020 con 0x7f33b815a3b0 2023-10-30T13:36:11.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.040+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33b804e170 con 0x7f33b815a3b0 2023-10-30T13:36:11.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.040+0000 7f33a6ffd640 1 -- 172.21.15.31:0/3408882705 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7f33b0054640 con 0x7f33b815a3b0 2023-10-30T13:36:11.046 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:11.046 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33980664a0 msgr2=0x7f3398068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33980664a0 0x7f3398068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33a80b2000 tx=0x7f33a8009040 comp rx=0 tx=0).stop 2023-10-30T13:36:11.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b815a3b0 msgr2=0x7f33b80cf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b815a3b0 0x7f33b80cf6a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f33b000c730 tx=0x7f33b000cbf0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 shutdown_connections 2023-10-30T13:36:11.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33980664a0 0x7f3398068960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33b815a3b0 0x7f33b80cf6a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33b8153c80 0x7f33b81ef310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 --2- 172.21.15.31:0/3408882705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33b80c4a10 0x7f33b806c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 >> 172.21.15.31:0/3408882705 conn(0x7f33b80715c0 msgr2=0x7f33b80ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:11.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 shutdown_connections 2023-10-30T13:36:11.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.044+0000 7f33bc9d3640 1 -- 172.21.15.31:0/3408882705 wait complete. 2023-10-30T13:36:11.068 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=2) 2023-10-30T13:36:11.069 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:36:11.069 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:36:11.084 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:36:11.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:11.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.236+0000 7fc44a394640 1 Processor -- start 2023-10-30T13:36:11.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.236+0000 7fc44a394640 1 -- start start 2023-10-30T13:36:11.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc44a394640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc44415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:11.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc44a394640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc4440c42a0 0x7fc4440ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:11.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc44a394640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc444159f20 con 0x7fc4440c4a10 2023-10-30T13:36:11.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc44a394640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44415dec0 con 0x7fc4440c42a0 2023-10-30T13:36:11.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc44a394640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4441599a0 con 0x7fc444153c80 2023-10-30T13:36:11.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc443fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc44415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:11.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc448931640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc4440c42a0 0x7fc4440ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:11.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc443fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc44415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:47886/0 (socket says 172.21.15.31:47886) 2023-10-30T13:36:11.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc443fff640 1 -- 172.21.15.31:0/205519200 learned_addr learned my addr 172.21.15.31:0/205519200 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:11.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc443fff640 1 -- 172.21.15.31:0/205519200 >> v2:172.21.15.33:3300/0 conn(0x7fc4440c42a0 msgr2=0x7fc4440ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:11.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc443fff640 1 --2- 172.21.15.31:0/205519200 >> v2:172.21.15.33:3300/0 conn(0x7fc4440c42a0 0x7fc4440ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc443fff640 1 -- 172.21.15.31:0/205519200 >> v1:172.21.15.33:6789/0 conn(0x7fc4440c4a10 legacy=0x7fc44415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.240+0000 7fc443fff640 1 -- 172.21.15.31:0/205519200 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44415d620 con 0x7fc444153c80 2023-10-30T13:36:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc443fff640 1 --2- 172.21.15.31:0/205519200 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc44415cf90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc434009870 tx=0x7fc43401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9940a2925fe25e97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc442ffd640 1 -- 172.21.15.31:0/205519200 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4340a8a30 con 0x7fc444153c80 2023-10-30T13:36:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc442ffd640 1 -- 172.21.15.31:0/205519200 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4340a8b90 con 0x7fc444153c80 2023-10-30T13:36:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc442ffd640 1 -- 172.21.15.31:0/205519200 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4340b0440 con 0x7fc444153c80 2023-10-30T13:36:11.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 -- 172.21.15.31:0/205519200 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 msgr2=0x7fc44415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 --2- 172.21.15.31:0/205519200 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc44415cf90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc434009870 tx=0x7fc43401ed30 comp rx=0 tx=0).stop 2023-10-30T13:36:11.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 -- 172.21.15.31:0/205519200 shutdown_connections 2023-10-30T13:36:11.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 --2- 172.21.15.31:0/205519200 >> v2:172.21.15.33:3300/0 conn(0x7fc4440c42a0 0x7fc4440ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 --2- 172.21.15.31:0/205519200 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc44415cf90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 -- 172.21.15.31:0/205519200 >> 172.21.15.31:0/205519200 conn(0x7fc4440715c0 msgr2=0x7fc4440719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:11.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 -- 172.21.15.31:0/205519200 shutdown_connections 2023-10-30T13:36:11.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.244+0000 7fc44a394640 1 -- 172.21.15.31:0/205519200 wait complete. 2023-10-30T13:36:11.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 Processor -- start 2023-10-30T13:36:11.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 -- start start 2023-10-30T13:36:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4440c42a0 0x7fc44415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4440c4a10 0x7fc44415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc4440c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc444150ff0 con 0x7fc4440c4a10 2023-10-30T13:36:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4441ef1d0 con 0x7fc4440c42a0 2023-10-30T13:36:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc44a394640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44415bfd0 con 0x7fc444153c80 2023-10-30T13:36:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.248+0000 7fc4437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4440c4a10 0x7fc44415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4440c4a10 0x7fc44415ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39824/0 (socket says 172.21.15.31:39824) 2023-10-30T13:36:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 -- 172.21.15.31:0/1669050605 learned_addr learned my addr 172.21.15.31:0/1669050605 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc448931640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc4440c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc443fff640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4440c42a0 0x7fc44415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 -- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 msgr2=0x7fc4440c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc4440c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 -- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4440c42a0 msgr2=0x7fc44415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4440c42a0 0x7fc44415ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 -- 172.21.15.31:0/1669050605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4340094d0 con 0x7fc4440c4a10 2023-10-30T13:36:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc4437fe640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4440c4a10 0x7fc44415ab60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c4d0 tx=0x7fc43800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc440ff9640 1 -- 172.21.15.31:0/1669050605 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380052d0 con 0x7fc4440c4a10 2023-10-30T13:36:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc440ff9640 1 -- 172.21.15.31:0/1669050605 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc438005430 con 0x7fc4440c4a10 2023-10-30T13:36:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4441ee900 con 0x7fc4440c4a10 2023-10-30T13:36:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc440ff9640 1 -- 172.21.15.31:0/1669050605 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438012830 con 0x7fc4440c4a10 2023-10-30T13:36:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4441eeb80 con 0x7fc4440c4a10 2023-10-30T13:36:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4441ef340 con 0x7fc4440c4a10 2023-10-30T13:36:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.252+0000 7fc440ff9640 1 -- 172.21.15.31:0/1669050605 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc438012990 con 0x7fc4440c4a10 2023-10-30T13:36:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.256+0000 7fc440ff9640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4300664a0 0x7fc430068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.256+0000 7fc440ff9640 1 -- 172.21.15.31:0/1669050605 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc438083900 con 0x7fc4440c4a10 2023-10-30T13:36:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.256+0000 7fc443fff640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4300664a0 0x7fc430068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.256+0000 7fc443fff640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4300664a0 0x7fc430068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc4340b2000 tx=0x7fc4340aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.260+0000 7fc440ff9640 1 -- 172.21.15.31:0/1669050605 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc438050d90 con 0x7fc4440c4a10 2023-10-30T13:36:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.452+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc44404e170 con 0x7fc4440c4a10 2023-10-30T13:36:11.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc440ff9640 1 -- 172.21.15.31:0/1669050605 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7fc4380543b0 con 0x7fc4440c4a10 2023-10-30T13:36:11.459 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:11.459 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4300664a0 msgr2=0x7fc430068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4300664a0 0x7fc430068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc4340b2000 tx=0x7fc4340aa070 comp rx=0 tx=0).stop 2023-10-30T13:36:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4440c4a10 msgr2=0x7fc44415ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4440c4a10 0x7fc44415ab60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c4d0 tx=0x7fc43800c990 comp rx=0 tx=0).stop 2023-10-30T13:36:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 shutdown_connections 2023-10-30T13:36:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4300664a0 0x7fc430068960 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc444153c80 0x7fc4440c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4440c4a10 0x7fc44415ab60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 --2- 172.21.15.31:0/1669050605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4440c42a0 0x7fc44415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 >> 172.21.15.31:0/1669050605 conn(0x7fc4440715c0 msgr2=0x7fc4440c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 shutdown_connections 2023-10-30T13:36:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:11.456+0000 7fc44a394640 1 -- 172.21.15.31:0/1669050605 wait complete. 2023-10-30T13:36:11.481 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:36:13.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa339a09640 1 Processor -- start 2023-10-30T13:36:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa339a09640 1 -- start start 2023-10-30T13:36:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa339a09640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa334071cb0 0x7fa3340c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa339a09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 0x7fa334153790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa339a09640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa334153cd0 con 0x7fa3340718d0 2023-10-30T13:36:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa339a09640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3340cca10 con 0x7fa334071cb0 2023-10-30T13:36:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa339a09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33415d0a0 con 0x7fa334153300 2023-10-30T13:36:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa332ffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fa3340718d0 0x7fa33415a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41354/0 (socket says 172.21.15.31:41354) 2023-10-30T13:36:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa332ffd640 1 -- 172.21.15.31:0/3787697955 learned_addr learned my addr 172.21.15.31:0/3787697955 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa3337fe640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 0x7fa334153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa3327fc640 1 --2- 172.21.15.31:0/3787697955 >> v2:172.21.15.33:3300/0 conn(0x7fa334071cb0 0x7fa3340c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525457504 0 0) 0x7fa324001120 con 0x7fa3340718d0 2023-10-30T13:36:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa31c003530 con 0x7fa3340718d0 2023-10-30T13:36:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2771351747 0 0) 0x7fa31c003530 con 0x7fa3340718d0 2023-10-30T13:36:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 msgr2=0x7fa334153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 0x7fa334153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 >> v2:172.21.15.33:3300/0 conn(0x7fa334071cb0 msgr2=0x7fa3340c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 --2- 172.21.15.31:0/3787697955 >> v2:172.21.15.33:3300/0 conn(0x7fa334071cb0 0x7fa3340c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33415d380 con 0x7fa3340718d0 2023-10-30T13:36:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fa324001810 con 0x7fa3340718d0 2023-10-30T13:36:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 >> v1:172.21.15.31:6789/0 conn(0x7fa3340718d0 legacy=0x7fa33415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 0x7fa31c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c003f00 0x7fa31c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa31c004d10 0x7fa31c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3340cca10 con 0x7fa31c004d10 2023-10-30T13:36:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33415d0a0 con 0x7fa31c003f00 2023-10-30T13:36:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c00b9c0 con 0x7fa334153300 2023-10-30T13:36:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa3337fe640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa31c004d10 0x7fa31c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa332ffd640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 0x7fa31c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.628+0000 7fa3327fc640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c003f00 0x7fa31c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa3337fe640 1 -- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 msgr2=0x7fa31c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa3337fe640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa334153300 0x7fa31c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa3337fe640 1 -- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c003f00 msgr2=0x7fa31c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa3337fe640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c003f00 0x7fa31c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa3337fe640 1 -- 172.21.15.31:0/3787697955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c003530 con 0x7fa31c004d10 2023-10-30T13:36:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa3337fe640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa31c004d10 0x7fa31c005190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009fc0 tx=0x7fa32c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad439a3dadb065c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0a5c70 con 0x7fa31c004d10 2023-10-30T13:36:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c00f040 con 0x7fa31c004d10 2023-10-30T13:36:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.632+0000 7fa331ffb640 1 -- 172.21.15.31:0/3787697955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0ad440 con 0x7fa31c004d10 2023-10-30T13:36:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- 172.21.15.31:0/3787697955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa31c004d10 msgr2=0x7fa31c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 --2- 172.21.15.31:0/3787697955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa31c004d10 0x7fa31c005190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009fc0 tx=0x7fa32c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:36:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa3337fe640 1 -- 172.21.15.31:0/3787697955 reap_dead start 2023-10-30T13:36:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- 172.21.15.31:0/3787697955 shutdown_connections 2023-10-30T13:36:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- 172.21.15.31:0/3787697955 >> 172.21.15.31:0/3787697955 conn(0x7fa33406ab00 msgr2=0x7fa334065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- 172.21.15.31:0/3787697955 shutdown_connections 2023-10-30T13:36:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- 172.21.15.31:0/3787697955 wait complete. 2023-10-30T13:36:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 Processor -- start 2023-10-30T13:36:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- start start 2023-10-30T13:36:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c004d10 0x7fa33414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33415a970 0x7fa3340b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3341f1bf0 0x7fa3340cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33415c1d0 con 0x7fa33415a970 2023-10-30T13:36:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33414edb0 con 0x7fa31c004d10 2023-10-30T13:36:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa339a09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33414f780 con 0x7fa3341f1bf0 2023-10-30T13:36:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa3327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33415a970 0x7fa3340b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa3327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33415a970 0x7fa3340b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39844/0 (socket says 172.21.15.31:39844) 2023-10-30T13:36:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa3327fc640 1 -- 172.21.15.31:0/855616544 learned_addr learned my addr 172.21.15.31:0/855616544 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa332ffd640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c004d10 0x7fa33414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.636+0000 7fa3337fe640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3341f1bf0 0x7fa3340cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa3327fc640 1 -- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3341f1bf0 msgr2=0x7fa3340cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa3327fc640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3341f1bf0 0x7fa3340cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa3327fc640 1 -- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c004d10 msgr2=0x7fa33414f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa3327fc640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c004d10 0x7fa33414f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa3327fc640 1 -- 172.21.15.31:0/855616544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3341f24a0 con 0x7fa33415a970 2023-10-30T13:36:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa3327fc640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33415a970 0x7fa3340b27d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa3280119d0 tx=0x7fa328011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa31bfff640 1 -- 172.21.15.31:0/855616544 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32800faf0 con 0x7fa33415a970 2023-10-30T13:36:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa31bfff640 1 -- 172.21.15.31:0/855616544 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32800fc50 con 0x7fa33415a970 2023-10-30T13:36:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa31bfff640 1 -- 172.21.15.31:0/855616544 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328014440 con 0x7fa33415a970 2023-10-30T13:36:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3341f2780 con 0x7fa33415a970 2023-10-30T13:36:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33414e760 con 0x7fa33415a970 2023-10-30T13:36:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33404e170 con 0x7fa33415a970 2023-10-30T13:36:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.640+0000 7fa31bfff640 1 -- 172.21.15.31:0/855616544 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3280145a0 con 0x7fa33415a970 2023-10-30T13:36:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.644+0000 7fa31bfff640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.644+0000 7fa31bfff640 1 -- 172.21.15.31:0/855616544 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fa328084600 con 0x7fa33415a970 2023-10-30T13:36:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.644+0000 7fa332ffd640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.644+0000 7fa332ffd640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3200664a0 0x7fa320068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa324001280 tx=0x7fa324007a80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.648+0000 7fa31bfff640 1 -- 172.21.15.31:0/855616544 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa328051b40 con 0x7fa33415a970 2023-10-30T13:36:13.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.868+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3340c9f80 con 0x7fa33415a970 2023-10-30T13:36:13.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.868+0000 7fa31bfff640 1 -- 172.21.15.31:0/855616544 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7fa328055160 con 0x7fa33415a970 2023-10-30T13:36:13.874 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:13.874 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:13.874 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3200664a0 msgr2=0x7fa320068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3200664a0 0x7fa320068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa324001280 tx=0x7fa324007a80 comp rx=0 tx=0).stop 2023-10-30T13:36:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33415a970 msgr2=0x7fa3340b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33415a970 0x7fa3340b27d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa3280119d0 tx=0x7fa328011dd0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 shutdown_connections 2023-10-30T13:36:13.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3341f1bf0 0x7fa3340cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33415a970 0x7fa3340b27d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 --2- 172.21.15.31:0/855616544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa31c004d10 0x7fa33414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 >> 172.21.15.31:0/855616544 conn(0x7fa33406ab00 msgr2=0x7fa3340c47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 shutdown_connections 2023-10-30T13:36:13.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:13.872+0000 7fa339a09640 1 -- 172.21.15.31:0/855616544 wait complete. 2023-10-30T13:36:13.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:36:15.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb534846640 1 Processor -- start 2023-10-30T13:36:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb534846640 1 -- start start 2023-10-30T13:36:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb534846640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 0x7fb53015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb534846640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb530159f20 con 0x7fb5300c42a0 2023-10-30T13:36:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb534846640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53015d620 con 0x7fb5300c4a10 2023-10-30T13:36:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb534846640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53015dec0 con 0x7fb530153c80 2023-10-30T13:36:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb52ed9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb5300c42a0 0x7fb5300ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56652/0 (socket says 172.21.15.31:56652) 2023-10-30T13:36:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb52ed9d640 1 -- 172.21.15.31:0/3861221751 learned_addr learned my addr 172.21.15.31:0/3861221751 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.040+0000 7fb52e59c640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 0x7fb53015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831025572 0 0) 0x7fb530159f20 con 0x7fb5300c42a0 2023-10-30T13:36:16.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb518003530 con 0x7fb5300c42a0 2023-10-30T13:36:16.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 304850247 0 0) 0x7fb518003530 con 0x7fb5300c42a0 2023-10-30T13:36:16.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 msgr2=0x7fb53015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:16.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 0x7fb53015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 >> v1:172.21.15.33:6789/0 conn(0x7fb5300c4a10 legacy=0x7fb53015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53015d900 con 0x7fb5300c42a0 2023-10-30T13:36:16.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fb528001a00 con 0x7fb5300c42a0 2023-10-30T13:36:16.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 >> v1:172.21.15.31:6789/0 conn(0x7fb5300c42a0 legacy=0x7fb5300ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb518003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb518004360 0x7fb518004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 0x7fb518005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53015dec0 con 0x7fb5180039b0 2023-10-30T13:36:16.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51800b990 con 0x7fb518004360 2023-10-30T13:36:16.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51800bb50 con 0x7fb530153c80 2023-10-30T13:36:16.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52e59c640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb518003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52ed9d640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 0x7fb518005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52dd9b640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb518004360 0x7fb518004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52e59c640 1 -- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 msgr2=0x7fb518005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:16.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52e59c640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb530153c80 0x7fb518005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52e59c640 1 -- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb518004360 msgr2=0x7fb518004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:16.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52e59c640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb518004360 0x7fb518004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.044+0000 7fb52e59c640 1 -- 172.21.15.31:0/3861221751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb518003530 con 0x7fb5180039b0 2023-10-30T13:36:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb52e59c640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb518003d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb520009ae0 tx=0x7fb52001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba74b6429f03c4d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200a5640 con 0x7fb5180039b0 2023-10-30T13:36:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5200a57a0 con 0x7fb5180039b0 2023-10-30T13:36:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb52d59a640 1 -- 172.21.15.31:0/3861221751 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200a5b50 con 0x7fb5180039b0 2023-10-30T13:36:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 -- 172.21.15.31:0/3861221751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 msgr2=0x7fb518003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 --2- 172.21.15.31:0/3861221751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb518003d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb520009ae0 tx=0x7fb52001ca50 comp rx=0 tx=0).stop 2023-10-30T13:36:16.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb52ed9d640 1 -- 172.21.15.31:0/3861221751 reap_dead start 2023-10-30T13:36:16.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 -- 172.21.15.31:0/3861221751 shutdown_connections 2023-10-30T13:36:16.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 -- 172.21.15.31:0/3861221751 >> 172.21.15.31:0/3861221751 conn(0x7fb5300715c0 msgr2=0x7fb5300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:16.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 -- 172.21.15.31:0/3861221751 shutdown_connections 2023-10-30T13:36:16.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 -- 172.21.15.31:0/3861221751 wait complete. 2023-10-30T13:36:16.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 Processor -- start 2023-10-30T13:36:16.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.048+0000 7fb534846640 1 -- start start 2023-10-30T13:36:16.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb530157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb5301f2dc0 0x7fb530151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5300c88a0 0x7fb5300c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5300b2540 con 0x7fb5180039b0 2023-10-30T13:36:16.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301efb60 con 0x7fb5300c88a0 2023-10-30T13:36:16.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb530157e10 con 0x7fb5301f2dc0 2023-10-30T13:36:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb530157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb530157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60774/0 (socket says 172.21.15.31:60774) 2023-10-30T13:36:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 -- 172.21.15.31:0/1973895711 learned_addr learned my addr 172.21.15.31:0/1973895711 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52dd9b640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb5301f2dc0 0x7fb530151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52ed9d640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5300c88a0 0x7fb5300c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 -- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb5301f2dc0 msgr2=0x7fb530151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb5301f2dc0 0x7fb530151510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 -- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5300c88a0 msgr2=0x7fb5300c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5300c88a0 0x7fb5300c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 -- 172.21.15.31:0/1973895711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5301eef50 con 0x7fb5180039b0 2023-10-30T13:36:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb52e59c640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb530157810 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5200b0040 tx=0x7fb518005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb51effd640 1 -- 172.21.15.31:0/1973895711 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200be070 con 0x7fb5180039b0 2023-10-30T13:36:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb51effd640 1 -- 172.21.15.31:0/1973895711 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520004020 con 0x7fb5180039b0 2023-10-30T13:36:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5301ef230 con 0x7fb5180039b0 2023-10-30T13:36:16.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb51effd640 1 -- 172.21.15.31:0/1973895711 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200a5640 con 0x7fb5180039b0 2023-10-30T13:36:16.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5301ef7f0 con 0x7fb5180039b0 2023-10-30T13:36:16.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5301ef9b0 con 0x7fb5180039b0 2023-10-30T13:36:16.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.052+0000 7fb51effd640 1 -- 172.21.15.31:0/1973895711 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5200b4040 con 0x7fb5180039b0 2023-10-30T13:36:16.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.056+0000 7fb51effd640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb5100664a0 0x7fb510068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:16.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.056+0000 7fb51effd640 1 -- 172.21.15.31:0/1973895711 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb52011dbb0 con 0x7fb5180039b0 2023-10-30T13:36:16.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.056+0000 7fb52dd9b640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb5100664a0 0x7fb510068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:16.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.056+0000 7fb52dd9b640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb5100664a0 0x7fb510068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb5301585a0 tx=0x7fb52400aa70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.064+0000 7fb51effd640 1 -- 172.21.15.31:0/1973895711 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5200eb040 con 0x7fb5180039b0 2023-10-30T13:36:16.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.252+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb530159b60 con 0x7fb5180039b0 2023-10-30T13:36:16.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.252+0000 7fb51effd640 1 -- 172.21.15.31:0/1973895711 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7fb5200b2070 con 0x7fb5180039b0 2023-10-30T13:36:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:16.258 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:16.259 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:16.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb5100664a0 msgr2=0x7fb510068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb5100664a0 0x7fb510068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb5301585a0 tx=0x7fb52400aa70 comp rx=0 tx=0).stop 2023-10-30T13:36:16.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 msgr2=0x7fb530157810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb530157810 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5200b0040 tx=0x7fb518005800 comp rx=0 tx=0).stop 2023-10-30T13:36:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 shutdown_connections 2023-10-30T13:36:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb5100664a0 0x7fb510068960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5300c88a0 0x7fb5300c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb5301f2dc0 0x7fb530151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 --2- 172.21.15.31:0/1973895711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5180039b0 0x7fb530157810 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 >> 172.21.15.31:0/1973895711 conn(0x7fb5300715c0 msgr2=0x7fb5300733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:16.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 shutdown_connections 2023-10-30T13:36:16.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:16.256+0000 7fb534846640 1 -- 172.21.15.31:0/1973895711 wait complete. 2023-10-30T13:36:16.279 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:36:18.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.452+0000 7f252759e640 1 Processor -- start 2023-10-30T13:36:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.452+0000 7f252759e640 1 -- start start 2023-10-30T13:36:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f252759e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2528153b50 0x7f25281597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f252759e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f25280c62b0 0x7f25280ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f252759e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252815a770 con 0x7f25280cb180 2023-10-30T13:36:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f252759e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2528159cf0 con 0x7f25280c62b0 2023-10-30T13:36:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f252759e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252806aa30 con 0x7f2528153b50 2023-10-30T13:36:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f2526d9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f25280cb180 0x7f252806ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56666/0 (socket says 172.21.15.31:56666) 2023-10-30T13:36:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f2526d9d640 1 -- 172.21.15.31:0/1503571162 learned_addr learned my addr 172.21.15.31:0/1503571162 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f2525d9b640 1 --2- 172.21.15.31:0/1503571162 >> v2:172.21.15.31:3300/0 conn(0x7f25280c62b0 0x7f25280ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f252659c640 1 --2- 172.21.15.31:0/1503571162 >> v2:172.21.15.33:3300/0 conn(0x7f2528153b50 0x7f25281597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:18.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f252559a640 1 -- 172.21.15.31:0/1503571162 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671437881 0 0) 0x7f25200014a0 con 0x7f25280cb180 2023-10-30T13:36:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f2525d9b640 1 -- 172.21.15.31:0/1503571162 >> v2:172.21.15.33:3300/0 conn(0x7f2528153b50 msgr2=0x7f25281597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f2525d9b640 1 --2- 172.21.15.31:0/1503571162 >> v2:172.21.15.33:3300/0 conn(0x7f2528153b50 0x7f25281597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f2525d9b640 1 -- 172.21.15.31:0/1503571162 >> v1:172.21.15.31:6789/0 conn(0x7f25280cb180 legacy=0x7f252806ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.456+0000 7f2525d9b640 1 -- 172.21.15.31:0/1503571162 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25280c9d40 con 0x7f25280c62b0 2023-10-30T13:36:18.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.460+0000 7f2525d9b640 1 --2- 172.21.15.31:0/1503571162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280c62b0 0x7f25280ca1d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2528066890 tx=0x7f251c01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356f50b9687cfc3b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:18.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252559a640 1 -- 172.21.15.31:0/1503571162 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c0a8930 con 0x7f25280c62b0 2023-10-30T13:36:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252559a640 1 -- 172.21.15.31:0/1503571162 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c0a8a90 con 0x7f25280c62b0 2023-10-30T13:36:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252559a640 1 -- 172.21.15.31:0/1503571162 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c0a8e20 con 0x7f25280c62b0 2023-10-30T13:36:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 -- 172.21.15.31:0/1503571162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280c62b0 msgr2=0x7f25280ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 --2- 172.21.15.31:0/1503571162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280c62b0 0x7f25280ca1d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2528066890 tx=0x7f251c01ec30 comp rx=0 tx=0).stop 2023-10-30T13:36:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 -- 172.21.15.31:0/1503571162 shutdown_connections 2023-10-30T13:36:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 --2- 172.21.15.31:0/1503571162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280c62b0 0x7f25280ca1d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 --2- 172.21.15.31:0/1503571162 >> v2:172.21.15.33:3300/0 conn(0x7f2528153b50 0x7f25281597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 -- 172.21.15.31:0/1503571162 >> 172.21.15.31:0/1503571162 conn(0x7f2528065a80 msgr2=0x7f2528065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:18.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 -- 172.21.15.31:0/1503571162 shutdown_connections 2023-10-30T13:36:18.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.464+0000 7f252759e640 1 -- 172.21.15.31:0/1503571162 wait complete. 2023-10-30T13:36:18.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 Processor -- start 2023-10-30T13:36:18.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 -- start start 2023-10-30T13:36:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25280c62b0 0x7f25280c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280cb180 0x7f252814c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2528153b50 0x7f25280c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2528157ff0 con 0x7f25280cb180 2023-10-30T13:36:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2528158b70 con 0x7f25280c62b0 2023-10-30T13:36:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252759e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2528158f90 con 0x7f2528153b50 2023-10-30T13:36:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280cb180 0x7f252814c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2526d9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2528153b50 0x7f25280c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f252659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25280c62b0 0x7f25280c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280cb180 0x7f252814c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60794/0 (socket says 172.21.15.31:60794) 2023-10-30T13:36:18.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 -- 172.21.15.31:0/2231996870 learned_addr learned my addr 172.21.15.31:0/2231996870 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 -- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2528153b50 msgr2=0x7f25280c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2528153b50 0x7f25280c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 -- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25280c62b0 msgr2=0x7f25280c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25280c62b0 0x7f25280c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 -- 172.21.15.31:0/2231996870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c009420 con 0x7f25280cb180 2023-10-30T13:36:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2525d9b640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280cb180 0x7f252814c210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f251c0b3040 tx=0x7f251c01b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2516ffd640 1 -- 172.21.15.31:0/2231996870 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c0b9630 con 0x7f25280cb180 2023-10-30T13:36:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2516ffd640 1 -- 172.21.15.31:0/2231996870 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c0b97b0 con 0x7f25280cb180 2023-10-30T13:36:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.468+0000 7f2516ffd640 1 -- 172.21.15.31:0/2231996870 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c0b9b60 con 0x7f25280cb180 2023-10-30T13:36:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.472+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2528159900 con 0x7f25280cb180 2023-10-30T13:36:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.472+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25281584b0 con 0x7f25280cb180 2023-10-30T13:36:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.472+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2528067540 con 0x7f25280cb180 2023-10-30T13:36:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.472+0000 7f2516ffd640 1 -- 172.21.15.31:0/2231996870 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f251c0b7040 con 0x7f25280cb180 2023-10-30T13:36:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.472+0000 7f2516ffd640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25080664a0 0x7f2508068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.472+0000 7f2516ffd640 1 -- 172.21.15.31:0/2231996870 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f251c121e20 con 0x7f25280cb180 2023-10-30T13:36:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.476+0000 7f252659c640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25080664a0 0x7f2508068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.476+0000 7f252659c640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25080664a0 0x7f2508068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2518009ac0 tx=0x7f2518009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.476+0000 7f2516ffd640 1 -- 172.21.15.31:0/2231996870 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f251c0ef220 con 0x7f25280cb180 2023-10-30T13:36:18.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.716+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25280ca980 con 0x7f25280cb180 2023-10-30T13:36:18.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.716+0000 7f2516ffd640 1 -- 172.21.15.31:0/2231996870 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7f251c0b5020 con 0x7f25280cb180 2023-10-30T13:36:18.720 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:18.721 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:18.721 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:18.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25080664a0 msgr2=0x7f2508068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25080664a0 0x7f2508068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2518009ac0 tx=0x7f2518009190 comp rx=0 tx=0).stop 2023-10-30T13:36:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280cb180 msgr2=0x7f252814c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280cb180 0x7f252814c210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f251c0b3040 tx=0x7f251c01b9d0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 shutdown_connections 2023-10-30T13:36:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25080664a0 0x7f2508068960 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2528153b50 0x7f25280c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25280cb180 0x7f252814c210 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 --2- 172.21.15.31:0/2231996870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25280c62b0 0x7f25280c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 >> 172.21.15.31:0/2231996870 conn(0x7f2528065a80 msgr2=0x7f252814e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 shutdown_connections 2023-10-30T13:36:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:18.720+0000 7f252759e640 1 -- 172.21.15.31:0/2231996870 wait complete. 2023-10-30T13:36:18.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:36:20.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.896+0000 7f97c193e640 1 Processor -- start 2023-10-30T13:36:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.896+0000 7f97c193e640 1 -- start start 2023-10-30T13:36:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97c193e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f97bc157d60 0x7f97bc0c8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97c193e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc0c8940 con 0x7f97bc1515d0 2023-10-30T13:36:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97c193e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc0c8120 con 0x7f97bc1519b0 2023-10-30T13:36:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97c193e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc0c8b90 con 0x7f97bc157d60 2023-10-30T13:36:20.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97baffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f97bc1515d0 0x7f97bc0c52d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56672/0 (socket says 172.21.15.31:56672) 2023-10-30T13:36:20.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97baffd640 1 -- 172.21.15.31:0/4096026768 learned_addr learned my addr 172.21.15.31:0/4096026768 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:20.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97ba7fc640 1 --1- 172.21.15.31:0/4096026768 >> v1:172.21.15.33:6789/0 conn(0x7f97bc1519b0 0x7f97bc157650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:56362/0 (socket says 172.21.15.31:56362) 2023-10-30T13:36:20.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97bb7fe640 1 --2- 172.21.15.31:0/4096026768 >> v2:172.21.15.33:3300/0 conn(0x7f97bc157d60 0x7f97bc0c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:20.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97b9ffb640 1 -- 172.21.15.31:0/4096026768 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951167959 0 0) 0x7f97bc0c8940 con 0x7f97bc1515d0 2023-10-30T13:36:20.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97b9ffb640 1 -- 172.21.15.31:0/4096026768 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a4003530 con 0x7f97bc1515d0 2023-10-30T13:36:20.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97bb7fe640 1 -- 172.21.15.31:0/4096026768 >> v1:172.21.15.33:6789/0 conn(0x7f97bc1519b0 legacy=0x7f97bc157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:20.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97bb7fe640 1 -- 172.21.15.31:0/4096026768 >> v1:172.21.15.31:6789/0 conn(0x7f97bc1515d0 legacy=0x7f97bc0c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:20.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.900+0000 7f97bb7fe640 1 -- 172.21.15.31:0/4096026768 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0c8e70 con 0x7f97bc157d60 2023-10-30T13:36:20.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.904+0000 7f97bb7fe640 1 --2- 172.21.15.31:0/4096026768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 0x7f97bc0c8400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009d90 tx=0x7f97b401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fabf978a863014 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:20.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97b9ffb640 1 -- 172.21.15.31:0/4096026768 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b40af600 con 0x7f97bc157d60 2023-10-30T13:36:20.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97b9ffb640 1 -- 172.21.15.31:0/4096026768 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b40ab050 con 0x7f97bc157d60 2023-10-30T13:36:20.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97b9ffb640 1 -- 172.21.15.31:0/4096026768 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b40af960 con 0x7f97bc157d60 2023-10-30T13:36:20.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 -- 172.21.15.31:0/4096026768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 msgr2=0x7f97bc0c8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 --2- 172.21.15.31:0/4096026768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 0x7f97bc0c8400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009d90 tx=0x7f97b401f130 comp rx=0 tx=0).stop 2023-10-30T13:36:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 -- 172.21.15.31:0/4096026768 shutdown_connections 2023-10-30T13:36:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 --2- 172.21.15.31:0/4096026768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 0x7f97bc0c8400 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 -- 172.21.15.31:0/4096026768 >> 172.21.15.31:0/4096026768 conn(0x7f97bc06e8c0 msgr2=0x7f97bc06eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 -- 172.21.15.31:0/4096026768 shutdown_connections 2023-10-30T13:36:20.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 -- 172.21.15.31:0/4096026768 wait complete. 2023-10-30T13:36:20.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.908+0000 7f97c193e640 1 Processor -- start 2023-10-30T13:36:20.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 -- start start 2023-10-30T13:36:20.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97bc1515d0 0x7f97bc13dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:20.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97bc1519b0 0x7f97bc150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:20.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 0x7f97bc142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:20.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc152ff0 con 0x7f97bc1519b0 2023-10-30T13:36:20.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc1505d0 con 0x7f97bc157d60 2023-10-30T13:36:20.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc13e1f0 con 0x7f97bc1515d0 2023-10-30T13:36:20.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97bc1519b0 0x7f97bc150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:20.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97bc1519b0 0x7f97bc150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60796/0 (socket says 172.21.15.31:60796) 2023-10-30T13:36:20.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 -- 172.21.15.31:0/3145813063 learned_addr learned my addr 172.21.15.31:0/3145813063 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:20.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97baffd640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97bc1515d0 0x7f97bc13dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:20.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97bb7fe640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 0x7f97bc142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:20.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 -- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97bc1515d0 msgr2=0x7f97bc13dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97bc1515d0 0x7f97bc13dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 -- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 msgr2=0x7f97bc142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 0x7f97bc142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 -- 172.21.15.31:0/3145813063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b4009a70 con 0x7f97bc1519b0 2023-10-30T13:36:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97ba7fc640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97bc1519b0 0x7f97bc150800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97b000c500 tx=0x7f97b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97a3fff640 1 -- 172.21.15.31:0/3145813063 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0007500 con 0x7f97bc1519b0 2023-10-30T13:36:20.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97a3fff640 1 -- 172.21.15.31:0/3145813063 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b0007660 con 0x7f97bc1519b0 2023-10-30T13:36:20.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97a3fff640 1 -- 172.21.15.31:0/3145813063 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0004a10 con 0x7f97bc1519b0 2023-10-30T13:36:20.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc14fd30 con 0x7f97bc1519b0 2023-10-30T13:36:20.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.912+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc1502f0 con 0x7f97bc1519b0 2023-10-30T13:36:20.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.916+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc067cb0 con 0x7f97bc1519b0 2023-10-30T13:36:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.916+0000 7f97a3fff640 1 -- 172.21.15.31:0/3145813063 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97b0004b70 con 0x7f97bc1519b0 2023-10-30T13:36:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.916+0000 7f97a3fff640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97a80664d0 0x7f97a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:20.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.916+0000 7f97a3fff640 1 -- 172.21.15.31:0/3145813063 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f97b0083110 con 0x7f97bc1519b0 2023-10-30T13:36:20.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.916+0000 7f97baffd640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97a80664d0 0x7f97a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:20.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.916+0000 7f97baffd640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97a80664d0 0x7f97a8068990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97ac007fd0 tx=0x7f97ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:20.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:20.920+0000 7f97a3fff640 1 -- 172.21.15.31:0/3145813063 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97b00505a0 con 0x7f97bc1519b0 2023-10-30T13:36:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.120+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97bc0ca4b0 con 0x7f97bc1519b0 2023-10-30T13:36:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.120+0000 7f97a3fff640 1 -- 172.21.15.31:0/3145813063 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7f97bc0ca4b0 con 0x7f97bc1519b0 2023-10-30T13:36:21.124 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:21.125 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97a80664d0 msgr2=0x7f97a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97a80664d0 0x7f97a8068990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97ac007fd0 tx=0x7f97ac009040 comp rx=0 tx=0).stop 2023-10-30T13:36:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97bc1519b0 msgr2=0x7f97bc150800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97bc1519b0 0x7f97bc150800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97b000c500 tx=0x7f97b000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:36:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 shutdown_connections 2023-10-30T13:36:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97a80664d0 0x7f97a8068990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97bc157d60 0x7f97bc142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97bc1519b0 0x7f97bc150800 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 --2- 172.21.15.31:0/3145813063 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97bc1515d0 0x7f97bc13dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 >> 172.21.15.31:0/3145813063 conn(0x7f97bc06e8c0 msgr2=0x7f97bc0cabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 shutdown_connections 2023-10-30T13:36:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:21.124+0000 7f97c193e640 1 -- 172.21.15.31:0/3145813063 wait complete. 2023-10-30T13:36:21.147 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:36:23.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:23.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.384+0000 7fbfd8e17640 1 Processor -- start 2023-10-30T13:36:23.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.384+0000 7fbfd8e17640 1 -- start start 2023-10-30T13:36:23.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd8e17640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbfd4153c80 0x7fbfd415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:23.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd8e17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd40c4a10 0x7fbfd40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:23.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd8e17640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd4159f20 con 0x7fbfd415a3b0 2023-10-30T13:36:23.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd8e17640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd415dec0 con 0x7fbfd4153c80 2023-10-30T13:36:23.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd8e17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd41599a0 con 0x7fbfd40c4a10 2023-10-30T13:36:23.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd259c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbfd4153c80 0x7fbfd415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:23.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd259c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbfd4153c80 0x7fbfd415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60800/0 (socket says 172.21.15.31:60800) 2023-10-30T13:36:23.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd259c640 1 -- 172.21.15.31:0/2772490603 learned_addr learned my addr 172.21.15.31:0/2772490603 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:23.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd1d9b640 1 --2- 172.21.15.31:0/2772490603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd40c4a10 0x7fbfd40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd259c640 1 -- 172.21.15.31:0/2772490603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd40c4a10 msgr2=0x7fbfd40c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd259c640 1 --2- 172.21.15.31:0/2772490603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd40c4a10 0x7fbfd40c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd259c640 1 -- 172.21.15.31:0/2772490603 >> v1:172.21.15.33:6789/0 conn(0x7fbfd415a3b0 legacy=0x7fbfd40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.388+0000 7fbfd259c640 1 -- 172.21.15.31:0/2772490603 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40c47e0 con 0x7fbfd4153c80 2023-10-30T13:36:23.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd259c640 1 --2- 172.21.15.31:0/2772490603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd4153c80 0x7fbfd415cf90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009870 tx=0x7fbfc401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbac4ad66a1c3333 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:23.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd159a640 1 -- 172.21.15.31:0/2772490603 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40a8a30 con 0x7fbfd4153c80 2023-10-30T13:36:23.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd159a640 1 -- 172.21.15.31:0/2772490603 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc40a8b90 con 0x7fbfd4153c80 2023-10-30T13:36:23.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd159a640 1 -- 172.21.15.31:0/2772490603 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40b0440 con 0x7fbfd4153c80 2023-10-30T13:36:23.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2772490603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd4153c80 msgr2=0x7fbfd415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:23.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2772490603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd4153c80 0x7fbfd415cf90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009870 tx=0x7fbfc401ed30 comp rx=0 tx=0).stop 2023-10-30T13:36:23.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2772490603 shutdown_connections 2023-10-30T13:36:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2772490603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd40c4a10 0x7fbfd40c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.392+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2772490603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd4153c80 0x7fbfd415cf90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.396+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2772490603 >> 172.21.15.31:0/2772490603 conn(0x7fbfd40715c0 msgr2=0x7fbfd40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.396+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2772490603 shutdown_connections 2023-10-30T13:36:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.396+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2772490603 wait complete. 2023-10-30T13:36:23.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.396+0000 7fbfd8e17640 1 Processor -- start 2023-10-30T13:36:23.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.396+0000 7fbfd8e17640 1 -- start start 2023-10-30T13:36:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd40c4a10 0x7fbfd40b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd4153c80 0x7fbfd4151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbfd415a3b0 0x7fbfd414ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40b4b30 con 0x7fbfd40c4a10 2023-10-30T13:36:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40b38c0 con 0x7fbfd415a3b0 2023-10-30T13:36:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40cfdd0 con 0x7fbfd4153c80 2023-10-30T13:36:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd40c4a10 0x7fbfd40b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd40c4a10 0x7fbfd40b45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60808/0 (socket says 172.21.15.31:60808) 2023-10-30T13:36:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 -- 172.21.15.31:0/2675710353 learned_addr learned my addr 172.21.15.31:0/2675710353 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd1d9b640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd4153c80 0x7fbfd4151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd2d9d640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbfd415a3b0 0x7fbfd414ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:23.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 -- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd4153c80 msgr2=0x7fbfd4151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:23.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd4153c80 0x7fbfd4151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 -- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbfd415a3b0 msgr2=0x7fbfd414ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:23.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbfd415a3b0 0x7fbfd414ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 -- 172.21.15.31:0/2675710353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc40094d0 con 0x7fbfd40c4a10 2023-10-30T13:36:23.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd259c640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd40c4a10 0x7fbfd40b45f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40b3040 tx=0x7fbfc401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:23.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfc2ffd640 1 -- 172.21.15.31:0/2675710353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40b0dd0 con 0x7fbfd40c4a10 2023-10-30T13:36:23.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfc2ffd640 1 -- 172.21.15.31:0/2675710353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc40bb460 con 0x7fbfd40c4a10 2023-10-30T13:36:23.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfc2ffd640 1 -- 172.21.15.31:0/2675710353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40bb810 con 0x7fbfd40c4a10 2023-10-30T13:36:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd40b4160 con 0x7fbfd40c4a10 2023-10-30T13:36:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.400+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd40cfb30 con 0x7fbfd40c4a10 2023-10-30T13:36:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.404+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd404e170 con 0x7fbfd40c4a10 2023-10-30T13:36:23.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.404+0000 7fbfc2ffd640 1 -- 172.21.15.31:0/2675710353 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbfc40b83f0 con 0x7fbfd40c4a10 2023-10-30T13:36:23.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.408+0000 7fbfc2ffd640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbfb4066470 0x7fbfb4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:23.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.408+0000 7fbfc2ffd640 1 -- 172.21.15.31:0/2675710353 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fbfc4121b60 con 0x7fbfd40c4a10 2023-10-30T13:36:23.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.408+0000 7fbfd1d9b640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbfb4066470 0x7fbfb4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.412+0000 7fbfd1d9b640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbfb4066470 0x7fbfb4068930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbfd415c0f0 tx=0x7fbfc8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.412+0000 7fbfc2ffd640 1 -- 172.21.15.31:0/2675710353 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbfc40ef070 con 0x7fbfd40c4a10 2023-10-30T13:36:23.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.800+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfd40cad40 con 0x7fbfd40c4a10 2023-10-30T13:36:23.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.800+0000 7fbfc2ffd640 1 -- 172.21.15.31:0/2675710353 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7fbfc40c0090 con 0x7fbfd40c4a10 2023-10-30T13:36:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:23.806 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:23.806 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:23.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbfb4066470 msgr2=0x7fbfb4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:23.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbfb4066470 0x7fbfb4068930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbfd415c0f0 tx=0x7fbfc8009190 comp rx=0 tx=0).stop 2023-10-30T13:36:23.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd40c4a10 msgr2=0x7fbfd40b45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:23.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd40c4a10 0x7fbfd40b45f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40b3040 tx=0x7fbfc401bad0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 shutdown_connections 2023-10-30T13:36:23.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbfb4066470 0x7fbfb4068930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbfd415a3b0 0x7fbfd414ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbfd4153c80 0x7fbfd4151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 --2- 172.21.15.31:0/2675710353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd40c4a10 0x7fbfd40b45f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 >> 172.21.15.31:0/2675710353 conn(0x7fbfd40715c0 msgr2=0x7fbfd4152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 shutdown_connections 2023-10-30T13:36:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:23.804+0000 7fbfd8e17640 1 -- 172.21.15.31:0/2675710353 wait complete. 2023-10-30T13:36:23.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:36:25.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:25.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:25.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.072+0000 7f8485132640 1 Processor -- start 2023-10-30T13:36:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.072+0000 7f8485132640 1 -- start start 2023-10-30T13:36:26.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f8485132640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84800c8eb0 0x7f848015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:26.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f8485132640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f848015ac70 0x7f848015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:26.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f8485132640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848015dcc0 con 0x7f848006aeb0 2023-10-30T13:36:26.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f8485132640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8480159800 con 0x7f848015ac70 2023-10-30T13:36:26.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f8485132640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8480159d80 con 0x7f84800c8eb0 2023-10-30T13:36:26.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84800c8eb0 0x7f848015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:26.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847f7fe640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f848015ac70 0x7f848015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:26.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84800c8eb0 0x7f848015cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:54506/0 (socket says 172.21.15.31:54506) 2023-10-30T13:36:26.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847ffff640 1 -- 172.21.15.31:0/1909812990 learned_addr learned my addr 172.21.15.31:0/1909812990 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:26.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847ffff640 1 -- 172.21.15.31:0/1909812990 >> v2:172.21.15.33:3300/0 conn(0x7f848015ac70 msgr2=0x7f848015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:26.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847ffff640 1 --2- 172.21.15.31:0/1909812990 >> v2:172.21.15.33:3300/0 conn(0x7f848015ac70 0x7f848015a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847ffff640 1 -- 172.21.15.31:0/1909812990 >> v1:172.21.15.33:6789/0 conn(0x7f848006aeb0 legacy=0x7f8480069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:26.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847ffff640 1 -- 172.21.15.31:0/1909812990 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848015a750 con 0x7f84800c8eb0 2023-10-30T13:36:26.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.076+0000 7f847f7fe640 1 --2- 172.21.15.31:0/1909812990 >> v2:172.21.15.33:3300/0 conn(0x7f848015ac70 0x7f848015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:26.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.080+0000 7f847ffff640 1 --2- 172.21.15.31:0/1909812990 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84800c8eb0 0x7f848015cd70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8470009870 tx=0x7f847001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=132815990ca3de87 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:26.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.080+0000 7f847effd640 1 -- 172.21.15.31:0/1909812990 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700a8a30 con 0x7f84800c8eb0 2023-10-30T13:36:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.080+0000 7f847effd640 1 -- 172.21.15.31:0/1909812990 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84700a8b90 con 0x7f84800c8eb0 2023-10-30T13:36:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.080+0000 7f847effd640 1 -- 172.21.15.31:0/1909812990 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700b0440 con 0x7f84800c8eb0 2023-10-30T13:36:26.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 -- 172.21.15.31:0/1909812990 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84800c8eb0 msgr2=0x7f848015cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:26.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 --2- 172.21.15.31:0/1909812990 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84800c8eb0 0x7f848015cd70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8470009870 tx=0x7f847001ed30 comp rx=0 tx=0).stop 2023-10-30T13:36:26.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 -- 172.21.15.31:0/1909812990 shutdown_connections 2023-10-30T13:36:26.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 --2- 172.21.15.31:0/1909812990 >> v2:172.21.15.33:3300/0 conn(0x7f848015ac70 0x7f848015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 --2- 172.21.15.31:0/1909812990 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84800c8eb0 0x7f848015cd70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 -- 172.21.15.31:0/1909812990 >> 172.21.15.31:0/1909812990 conn(0x7f84800cef00 msgr2=0x7f84800cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:26.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 -- 172.21.15.31:0/1909812990 shutdown_connections 2023-10-30T13:36:26.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 -- 172.21.15.31:0/1909812990 wait complete. 2023-10-30T13:36:26.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 Processor -- start 2023-10-30T13:36:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.084+0000 7f8485132640 1 -- start start 2023-10-30T13:36:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8485132640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f848006aeb0 0x7f84801f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8485132640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84800c8eb0 0x7f84801ef930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8485132640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f848015ac70 0x7f8480157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8485132640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8480154860 con 0x7f848006aeb0 2023-10-30T13:36:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8485132640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84801ef4a0 con 0x7f84800c8eb0 2023-10-30T13:36:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8485132640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84801f33e0 con 0x7f848015ac70 2023-10-30T13:36:26.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f848006aeb0 0x7f84801f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:26.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84800c8eb0 0x7f84801ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:26.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8484931640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f848015ac70 0x7f8480157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:26.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f848006aeb0 0x7f84801f2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35884/0 (socket says 172.21.15.31:35884) 2023-10-30T13:36:26.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 -- 172.21.15.31:0/231021661 learned_addr learned my addr 172.21.15.31:0/231021661 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:26.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 -- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f848015ac70 msgr2=0x7f8480157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:26.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f848015ac70 0x7f8480157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 -- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84800c8eb0 msgr2=0x7f84801ef930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:26.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84800c8eb0 0x7f84801ef930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 -- 172.21.15.31:0/231021661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8474009420 con 0x7f848006aeb0 2023-10-30T13:36:26.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847ffff640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f848006aeb0 0x7f84801f2500 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f84700b3040 tx=0x7f847001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:26.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847cff9640 1 -- 172.21.15.31:0/231021661 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700b0dd0 con 0x7f848006aeb0 2023-10-30T13:36:26.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847cff9640 1 -- 172.21.15.31:0/231021661 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84700bb460 con 0x7f848006aeb0 2023-10-30T13:36:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f847cff9640 1 -- 172.21.15.31:0/231021661 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700bb810 con 0x7f848006aeb0 2023-10-30T13:36:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.088+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84700094d0 con 0x7f848006aeb0 2023-10-30T13:36:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.092+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84801f2e20 con 0x7f848006aeb0 2023-10-30T13:36:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.092+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848004e170 con 0x7f848006aeb0 2023-10-30T13:36:26.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.092+0000 7f847cff9640 1 -- 172.21.15.31:0/231021661 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84700b83f0 con 0x7f848006aeb0 2023-10-30T13:36:26.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.096+0000 7f847cff9640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f846c0664f0 0x7f846c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:26.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.096+0000 7f847cff9640 1 -- 172.21.15.31:0/231021661 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8470121b20 con 0x7f848006aeb0 2023-10-30T13:36:26.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.096+0000 7f847f7fe640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f846c0664f0 0x7f846c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.096+0000 7f847f7fe640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f846c0664f0 0x7f846c0689b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84800c54c0 tx=0x7f8474009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:26.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.104+0000 7f847cff9640 1 -- 172.21.15.31:0/231021661 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84700eef10 con 0x7f848006aeb0 2023-10-30T13:36:26.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.488+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84800c8710 con 0x7f848006aeb0 2023-10-30T13:36:26.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.488+0000 7f847cff9640 1 -- 172.21.15.31:0/231021661 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6517 (secure 0 0 0) 0x7f84700b5020 con 0x7f848006aeb0 2023-10-30T13:36:26.494 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:26.494 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:35:24.811863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.33:6836/2545654386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2545654386},{"type":"v1","addr":"172.21.15.33:6836","nonce":2545654386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:26.495 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-10-30T13:36:26.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f846c0664f0 msgr2=0x7f846c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:26.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f846c0664f0 0x7f846c0689b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84800c54c0 tx=0x7f8474009190 comp rx=0 tx=0).stop 2023-10-30T13:36:26.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f848006aeb0 msgr2=0x7f84801f2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:26.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f848006aeb0 0x7f84801f2500 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f84700b3040 tx=0x7f847001bad0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 shutdown_connections 2023-10-30T13:36:26.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f846c0664f0 0x7f846c0689b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f848015ac70 0x7f8480157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84800c8eb0 0x7f84801ef930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 --2- 172.21.15.31:0/231021661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f848006aeb0 0x7f84801f2500 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 >> 172.21.15.31:0/231021661 conn(0x7f84800cef00 msgr2=0x7f84800c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 shutdown_connections 2023-10-30T13:36:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:26.492+0000 7f8485132640 1 -- 172.21.15.31:0/231021661 wait complete. 2023-10-30T13:36:26.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:36:28.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.760+0000 7f9f85017640 1 Processor -- start 2023-10-30T13:36:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.760+0000 7f9f85017640 1 -- start start 2023-10-30T13:36:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f85017640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f9f800c8eb0 0x7f9f8015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f85017640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9f8006aeb0 0x7f9f8006b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f85017640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8015dcc0 con 0x7f9f8015ac70 2023-10-30T13:36:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f85017640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f80159800 con 0x7f9f8006aeb0 2023-10-30T13:36:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f85017640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f80159d80 con 0x7f9f800c8eb0 2023-10-30T13:36:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f7ffff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f9f800c8eb0 0x7f9f8015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f7ffff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f9f800c8eb0 0x7f9f8015cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:52878/0 (socket says 172.21.15.31:52878) 2023-10-30T13:36:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2452228878 learned_addr learned my addr 172.21.15.31:0/2452228878 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f84816640 1 --2- 172.21.15.31:0/2452228878 >> v2:172.21.15.31:3300/0 conn(0x7f9f8006aeb0 0x7f9f8006b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2452228878 >> v2:172.21.15.31:3300/0 conn(0x7f9f8006aeb0 msgr2=0x7f9f8006b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f7ffff640 1 --2- 172.21.15.31:0/2452228878 >> v2:172.21.15.31:3300/0 conn(0x7f9f8006aeb0 0x7f9f8006b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2452228878 >> v1:172.21.15.31:6789/0 conn(0x7f9f8015ac70 legacy=0x7f9f8015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.764+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2452228878 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8015d420 con 0x7f9f800c8eb0 2023-10-30T13:36:28.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.768+0000 7f9f7ffff640 1 --2- 172.21.15.31:0/2452228878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f800c8eb0 0x7f9f8015cd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f8006aa40 tx=0x7f9f7001ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d02d39cd51ae1cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:28.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.768+0000 7f9f7effd640 1 -- 172.21.15.31:0/2452228878 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700a89f0 con 0x7f9f800c8eb0 2023-10-30T13:36:28.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f7effd640 1 -- 172.21.15.31:0/2452228878 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f700a8b50 con 0x7f9f800c8eb0 2023-10-30T13:36:28.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f7effd640 1 -- 172.21.15.31:0/2452228878 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700b0440 con 0x7f9f800c8eb0 2023-10-30T13:36:28.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 -- 172.21.15.31:0/2452228878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f800c8eb0 msgr2=0x7f9f8015cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:28.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 --2- 172.21.15.31:0/2452228878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f800c8eb0 0x7f9f8015cd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f8006aa40 tx=0x7f9f7001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:36:28.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 -- 172.21.15.31:0/2452228878 shutdown_connections 2023-10-30T13:36:28.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 --2- 172.21.15.31:0/2452228878 >> v2:172.21.15.31:3300/0 conn(0x7f9f8006aeb0 0x7f9f8006b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:28.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 --2- 172.21.15.31:0/2452228878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f800c8eb0 0x7f9f8015cd70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:28.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 -- 172.21.15.31:0/2452228878 >> 172.21.15.31:0/2452228878 conn(0x7f9f800cef00 msgr2=0x7f9f800cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:28.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 -- 172.21.15.31:0/2452228878 shutdown_connections 2023-10-30T13:36:28.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 -- 172.21.15.31:0/2452228878 wait complete. 2023-10-30T13:36:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.772+0000 7f9f85017640 1 Processor -- start 2023-10-30T13:36:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f85017640 1 -- start start 2023-10-30T13:36:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f85017640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f8006aeb0 0x7f9f8015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f85017640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9f800c8eb0 0x7f9f8014f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f85017640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f8015ac70 0x7f9f8014f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f85017640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800c6e10 con 0x7f9f8006aeb0 2023-10-30T13:36:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f85017640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015ccd0 con 0x7f9f8015ac70 2023-10-30T13:36:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f85017640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800cdd40 con 0x7f9f800c8eb0 2023-10-30T13:36:28.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f8006aeb0 0x7f9f8015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:28.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f8006aeb0 0x7f9f8015b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35902/0 (socket says 172.21.15.31:35902) 2023-10-30T13:36:28.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2556233246 learned_addr learned my addr 172.21.15.31:0/2556233246 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:28.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7f7fe640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9f800c8eb0 0x7f9f8014f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:28.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f84816640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f8015ac70 0x7f9f8014f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:28.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9f800c8eb0 msgr2=0x7f9f8014f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:28.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9f800c8eb0 0x7f9f8014f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f8015ac70 msgr2=0x7f9f8014f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f8015ac70 0x7f9f8014f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 -- 172.21.15.31:0/2556233246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700094d0 con 0x7f9f8006aeb0 2023-10-30T13:36:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7ffff640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f8006aeb0 0x7f9f8015b2e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9f7001b900 tx=0x7f9f7001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7cff9640 1 -- 172.21.15.31:0/2556233246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700ba600 con 0x7f9f8006aeb0 2023-10-30T13:36:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.776+0000 7f9f7cff9640 1 -- 172.21.15.31:0/2556233246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f700ba760 con 0x7f9f8006aeb0 2023-10-30T13:36:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.780+0000 7f9f7cff9640 1 -- 172.21.15.31:0/2556233246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700bab10 con 0x7f9f8006aeb0 2023-10-30T13:36:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.780+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f800c7020 con 0x7f9f8006aeb0 2023-10-30T13:36:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.780+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f800ce0f0 con 0x7f9f8006aeb0 2023-10-30T13:36:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.780+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8004e170 con 0x7f9f8006aeb0 2023-10-30T13:36:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.784+0000 7f9f7cff9640 1 -- 172.21.15.31:0/2556233246 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f700bac70 con 0x7f9f8006aeb0 2023-10-30T13:36:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.784+0000 7f9f7cff9640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f6c0664f0 0x7f9f6c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.784+0000 7f9f7cff9640 1 -- 172.21.15.31:0/2556233246 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f9f70122e40 con 0x7f9f8006aeb0 2023-10-30T13:36:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.784+0000 7f9f7f7fe640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f6c0664f0 0x7f9f6c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:28.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.784+0000 7f9f7f7fe640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f6c0664f0 0x7f9f6c0689b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9f801ec500 tx=0x7f9f74009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:28.788+0000 7f9f7cff9640 1 -- 172.21.15.31:0/2556233246 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f700f0260 con 0x7f9f8006aeb0 2023-10-30T13:36:29.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.176+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f801599e0 con 0x7f9f8006aeb0 2023-10-30T13:36:29.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.176+0000 7f9f7cff9640 1 -- 172.21.15.31:0/2556233246 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5728 (secure 0 0 0) 0x7f9f700b5090 con 0x7f9f8006aeb0 2023-10-30T13:36:29.181 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:29.181 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:26.753769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 14 2023-10-30T13:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f6c0664f0 msgr2=0x7f9f6c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f6c0664f0 0x7f9f6c0689b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9f801ec500 tx=0x7f9f74009040 comp rx=0 tx=0).stop 2023-10-30T13:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f8006aeb0 msgr2=0x7f9f8015b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f8006aeb0 0x7f9f8015b2e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9f7001b900 tx=0x7f9f7001ba90 comp rx=0 tx=0).stop 2023-10-30T13:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 shutdown_connections 2023-10-30T13:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f6c0664f0 0x7f9f6c0689b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f8015ac70 0x7f9f8014f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9f800c8eb0 0x7f9f8014f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 --2- 172.21.15.31:0/2556233246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f8006aeb0 0x7f9f8015b2e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 >> 172.21.15.31:0/2556233246 conn(0x7f9f800cef00 msgr2=0x7f9f800c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:29.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 shutdown_connections 2023-10-30T13:36:29.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.180+0000 7f9f85017640 1 -- 172.21.15.31:0/2556233246 wait complete. 2023-10-30T13:36:29.214 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:36:29.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:36:29.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:29.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.448+0000 7fea86aad640 1 Processor -- start 2023-10-30T13:36:29.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.448+0000 7fea86aad640 1 -- start start 2023-10-30T13:36:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.448+0000 7fea86aad640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fea80153b50 0x7fea801597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.448+0000 7fea86aad640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea800ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.448+0000 7fea86aad640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8015a770 con 0x7fea800cb180 2023-10-30T13:36:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.448+0000 7fea86aad640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea80159cf0 con 0x7fea80153b50 2023-10-30T13:36:29.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.448+0000 7fea86aad640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea8006aa30 con 0x7fea800c62b0 2023-10-30T13:36:29.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea85aab640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fea80153b50 0x7fea801597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:29.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea852aa640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea800ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:29.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea85aab640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fea80153b50 0x7fea801597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35918/0 (socket says 172.21.15.31:35918) 2023-10-30T13:36:29.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea85aab640 1 -- 172.21.15.31:0/964953461 learned_addr learned my addr 172.21.15.31:0/964953461 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:29.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea85aab640 1 -- 172.21.15.31:0/964953461 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 msgr2=0x7fea800ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:29.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea85aab640 1 --2- 172.21.15.31:0/964953461 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea800ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea85aab640 1 -- 172.21.15.31:0/964953461 >> v1:172.21.15.33:6789/0 conn(0x7fea800cb180 legacy=0x7fea8006ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.452+0000 7fea85aab640 1 -- 172.21.15.31:0/964953461 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea800c9d40 con 0x7fea80153b50 2023-10-30T13:36:29.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea85aab640 1 --2- 172.21.15.31:0/964953461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea80153b50 0x7fea801597b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009870 tx=0x7fea7c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586ba2f82383a6a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:29.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea84aa9640 1 -- 172.21.15.31:0/964953461 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0ab600 con 0x7fea80153b50 2023-10-30T13:36:29.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea84aa9640 1 -- 172.21.15.31:0/964953461 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c0a7050 con 0x7fea80153b50 2023-10-30T13:36:29.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea84aa9640 1 -- 172.21.15.31:0/964953461 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0ab980 con 0x7fea80153b50 2023-10-30T13:36:29.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 -- 172.21.15.31:0/964953461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea80153b50 msgr2=0x7fea801597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 --2- 172.21.15.31:0/964953461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea80153b50 0x7fea801597b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009870 tx=0x7fea7c01c560 comp rx=0 tx=0).stop 2023-10-30T13:36:29.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 -- 172.21.15.31:0/964953461 shutdown_connections 2023-10-30T13:36:29.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 --2- 172.21.15.31:0/964953461 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea800ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 --2- 172.21.15.31:0/964953461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea80153b50 0x7fea801597b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 -- 172.21.15.31:0/964953461 >> 172.21.15.31:0/964953461 conn(0x7fea80065a80 msgr2=0x7fea80065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:29.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 -- 172.21.15.31:0/964953461 shutdown_connections 2023-10-30T13:36:29.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.456+0000 7fea86aad640 1 -- 172.21.15.31:0/964953461 wait complete. 2023-10-30T13:36:29.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea86aad640 1 Processor -- start 2023-10-30T13:36:29.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea86aad640 1 -- start start 2023-10-30T13:36:29.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea86aad640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea80163590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:29.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea86aad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea800cb180 0x7fea800c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:29.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea86aad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80153b50 0x7fea800c1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:29.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea852aa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea800cb180 0x7fea800c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:29.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea852aa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea800cb180 0x7fea800c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35926/0 (socket says 172.21.15.31:35926) 2023-10-30T13:36:29.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea852aa640 1 -- 172.21.15.31:0/2627753161 learned_addr learned my addr 172.21.15.31:0/2627753161 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:29.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea85aab640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea80163590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:29.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.460+0000 7fea86aad640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea801f5a10 con 0x7fea800cb180 2023-10-30T13:36:29.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea801f5c70 con 0x7fea80153b50 2023-10-30T13:36:29.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea801f5f50 con 0x7fea800c62b0 2023-10-30T13:36:29.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea862ac640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80153b50 0x7fea800c1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:29.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea852aa640 1 -- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 msgr2=0x7fea80163590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea852aa640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea80163590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea852aa640 1 -- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80153b50 msgr2=0x7fea800c1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea852aa640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80153b50 0x7fea800c1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea852aa640 1 -- 172.21.15.31:0/2627753161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c0094d0 con 0x7fea800cb180 2023-10-30T13:36:29.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea852aa640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea800cb180 0x7fea800c1930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea7000e900 tx=0x7fea7000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:29.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea767fc640 1 -- 172.21.15.31:0/2627753161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea70009690 con 0x7fea800cb180 2023-10-30T13:36:29.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea767fc640 1 -- 172.21.15.31:0/2627753161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7000ca00 con 0x7fea800cb180 2023-10-30T13:36:29.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea801f6230 con 0x7fea800cb180 2023-10-30T13:36:29.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea767fc640 1 -- 172.21.15.31:0/2627753161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7000cd90 con 0x7fea800cb180 2023-10-30T13:36:29.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea801f67f0 con 0x7fea800cb180 2023-10-30T13:36:29.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.464+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea80144540 con 0x7fea800cb180 2023-10-30T13:36:29.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.468+0000 7fea767fc640 1 -- 172.21.15.31:0/2627753161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea70013470 con 0x7fea800cb180 2023-10-30T13:36:29.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.468+0000 7fea767fc640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea640664a0 0x7fea64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:29.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.468+0000 7fea767fc640 1 -- 172.21.15.31:0/2627753161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fea70084560 con 0x7fea800cb180 2023-10-30T13:36:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.468+0000 7fea85aab640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea640664a0 0x7fea64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.468+0000 7fea85aab640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea640664a0 0x7fea64068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea8006ad60 tx=0x7fea7c0a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:29.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.476+0000 7fea767fc640 1 -- 172.21.15.31:0/2627753161 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea80144540 con 0x7fea800cb180 2023-10-30T13:36:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.860+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea80144540 con 0x7fea800cb180 2023-10-30T13:36:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.860+0000 7fea767fc640 1 -- 172.21.15.31:0/2627753161 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5728 (secure 0 0 0) 0x7fea80144540 con 0x7fea800cb180 2023-10-30T13:36:29.864 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:29.865 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:26.753769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:29.865 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 14 2023-10-30T13:36:29.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea640664a0 msgr2=0x7fea64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea640664a0 0x7fea64068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea8006ad60 tx=0x7fea7c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:36:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea800cb180 msgr2=0x7fea800c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea800cb180 0x7fea800c1930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea7000e900 tx=0x7fea7000edc0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 shutdown_connections 2023-10-30T13:36:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea640664a0 0x7fea64068960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80153b50 0x7fea800c1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea800cb180 0x7fea800c1930 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 --2- 172.21.15.31:0/2627753161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea800c62b0 0x7fea80163590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 >> 172.21.15.31:0/2627753161 conn(0x7fea80065a80 msgr2=0x7fea80069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 shutdown_connections 2023-10-30T13:36:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:29.864+0000 7fea86aad640 1 -- 172.21.15.31:0/2627753161 wait complete. 2023-10-30T13:36:29.899 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 14, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 14, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4272": { "addr": "172.21.15.31:6837/1956809670", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1956809670, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1956809670, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4276": { "addr": "172.21.15.33:6837/3402459191", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 3402459191, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 3402459191, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4276, "incarnation": 10, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:36:26.753769+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4276, "mds_2": 4272 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:36:31.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:32.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.132+0000 7ff672cad640 1 Processor -- start 2023-10-30T13:36:32.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.132+0000 7ff672cad640 1 -- start start 2023-10-30T13:36:32.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff672cad640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff66c15cd70 0x7ff66c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:32.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff672cad640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c15dcc0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff672cad640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c159d80 con 0x7ff66c15ac70 2023-10-30T13:36:32.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff672cad640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff66c159800 con 0x7ff66c15cd70 2023-10-30T13:36:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff671cab640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff66c0c8eb0 0x7ff66c15a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:38806/0 (socket says 172.21.15.31:38806) 2023-10-30T13:36:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff671cab640 1 -- 172.21.15.31:0/3940352493 learned_addr learned my addr 172.21.15.31:0/3940352493 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff6724ac640 1 --2- 172.21.15.31:0/3940352493 >> v2:172.21.15.33:3300/0 conn(0x7ff66c15cd70 0x7ff66c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff670ca9640 1 -- 172.21.15.31:0/3940352493 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955529957 0 0) 0x7ff668001120 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff670ca9640 1 -- 172.21.15.31:0/3940352493 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff658003530 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff6724ac640 1 -- 172.21.15.31:0/3940352493 >> v1:172.21.15.33:6789/0 conn(0x7ff66c15ac70 legacy=0x7ff66c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff6724ac640 1 -- 172.21.15.31:0/3940352493 >> v1:172.21.15.31:6789/0 conn(0x7ff66c0c8eb0 legacy=0x7ff66c15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.136+0000 7ff6724ac640 1 -- 172.21.15.31:0/3940352493 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c15d1c0 con 0x7ff66c15cd70 2023-10-30T13:36:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.140+0000 7ff6724ac640 1 --2- 172.21.15.31:0/3940352493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15cd70 0x7ff66c069e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff664009d90 tx=0x7ff66401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b012a5de706c3b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:32.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.140+0000 7ff670ca9640 1 -- 172.21.15.31:0/3940352493 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6640af600 con 0x7ff66c15cd70 2023-10-30T13:36:32.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.140+0000 7ff670ca9640 1 -- 172.21.15.31:0/3940352493 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6640ab050 con 0x7ff66c15cd70 2023-10-30T13:36:32.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.140+0000 7ff670ca9640 1 -- 172.21.15.31:0/3940352493 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6640af960 con 0x7ff66c15cd70 2023-10-30T13:36:32.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.140+0000 7ff672cad640 1 -- 172.21.15.31:0/3940352493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15cd70 msgr2=0x7ff66c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:32.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.140+0000 7ff672cad640 1 --2- 172.21.15.31:0/3940352493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15cd70 0x7ff66c069e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff664009d90 tx=0x7ff66401f130 comp rx=0 tx=0).stop 2023-10-30T13:36:32.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.144+0000 7ff672cad640 1 -- 172.21.15.31:0/3940352493 shutdown_connections 2023-10-30T13:36:32.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.144+0000 7ff672cad640 1 --2- 172.21.15.31:0/3940352493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15cd70 0x7ff66c069e90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.144+0000 7ff672cad640 1 -- 172.21.15.31:0/3940352493 >> 172.21.15.31:0/3940352493 conn(0x7ff66c0cef00 msgr2=0x7ff66c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:32.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.144+0000 7ff672cad640 1 -- 172.21.15.31:0/3940352493 shutdown_connections 2023-10-30T13:36:32.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.144+0000 7ff672cad640 1 -- 172.21.15.31:0/3940352493 wait complete. 2023-10-30T13:36:32.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.144+0000 7ff672cad640 1 Processor -- start 2023-10-30T13:36:32.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.144+0000 7ff672cad640 1 -- start start 2023-10-30T13:36:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff66c0c8eb0 0x7ff66c156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15ac70 0x7ff66c155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff66c15cd70 0x7ff66c1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c156920 con 0x7ff66c15cd70 2023-10-30T13:36:32.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c154d90 con 0x7ff66c15ac70 2023-10-30T13:36:32.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c154fe0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6724ac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff66c15cd70 0x7ff66c1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:32.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff671cab640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff66c0c8eb0 0x7ff66c156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:32.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6724ac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff66c15cd70 0x7ff66c1587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35928/0 (socket says 172.21.15.31:35928) 2023-10-30T13:36:32.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6724ac640 1 -- 172.21.15.31:0/3272886994 learned_addr learned my addr 172.21.15.31:0/3272886994 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:32.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6714aa640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15ac70 0x7ff66c155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:32.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff671cab640 1 -- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15ac70 msgr2=0x7ff66c155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:32.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff671cab640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15ac70 0x7ff66c155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff671cab640 1 -- 172.21.15.31:0/3272886994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff66c15cd70 msgr2=0x7ff66c1587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:32.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff671cab640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff66c15cd70 0x7ff66c1587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff671cab640 1 -- 172.21.15.31:0/3272886994 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664009a70 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6724ac640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff66c15cd70 0x7ff66c1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:32.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff671cab640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff66c0c8eb0 0x7ff66c156be0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c700 tx=0x7ff66800cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:32.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6627fc640 1 -- 172.21.15.31:0/3272886994 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680058c0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6627fc640 1 -- 172.21.15.31:0/3272886994 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff668005a20 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff6627fc640 1 -- 172.21.15.31:0/3272886994 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668005df0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66c155320 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.148+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66c157120 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.152+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66c1f08e0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.152+0000 7ff6627fc640 1 -- 172.21.15.31:0/3272886994 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff668007070 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.156+0000 7ff6627fc640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6500664a0 0x7ff650068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:32.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.156+0000 7ff6627fc640 1 -- 172.21.15.31:0/3272886994 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7ff6680834b0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.156+0000 7ff6714aa640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6500664a0 0x7ff650068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:32.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.156+0000 7ff6714aa640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6500664a0 0x7ff650068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff66c1595e0 tx=0x7ff65c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:32.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.164+0000 7ff6627fc640 1 -- 172.21.15.31:0/3272886994 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff668050940 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.548+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff66c15dcc0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff6627fc640 1 -- 172.21.15.31:0/3272886994 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5728 (secure 0 0 0) 0x7ff66c15dcc0 con 0x7ff66c0c8eb0 2023-10-30T13:36:32.555 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:32.555 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:26.753769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:32.556 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 14 2023-10-30T13:36:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6500664a0 msgr2=0x7ff650068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6500664a0 0x7ff650068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff66c1595e0 tx=0x7ff65c009040 comp rx=0 tx=0).stop 2023-10-30T13:36:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff66c0c8eb0 msgr2=0x7ff66c156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff66c0c8eb0 0x7ff66c156be0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c700 tx=0x7ff66800cbc0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 shutdown_connections 2023-10-30T13:36:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6500664a0 0x7ff650068960 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff66c15cd70 0x7ff66c1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff66c15ac70 0x7ff66c155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 --2- 172.21.15.31:0/3272886994 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff66c0c8eb0 0x7ff66c156be0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:32.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 >> 172.21.15.31:0/3272886994 conn(0x7ff66c0cef00 msgr2=0x7ff66c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:32.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 shutdown_connections 2023-10-30T13:36:32.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:32.552+0000 7ff672cad640 1 -- 172.21.15.31:0/3272886994 wait complete. 2023-10-30T13:36:32.588 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:34.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 Processor -- start 2023-10-30T13:36:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 -- start start 2023-10-30T13:36:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fce0806dfc0 0x7fce0806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fce080c5460 0x7fce080706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:34.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0815aa60 0x7fce080c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:34.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce08070be0 con 0x7fce080c5460 2023-10-30T13:36:34.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0806fc30 con 0x7fce0806dfc0 2023-10-30T13:36:34.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.824+0000 7fce0759e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0815b360 con 0x7fce0815aa60 2023-10-30T13:36:34.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce05d9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fce080c5460 0x7fce080706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:34.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fce0806dfc0 0x7fce0806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:34.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce06d9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0815aa60 0x7fce080c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:34.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fce0806dfc0 0x7fce0806d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:41968/0 (socket says 172.21.15.31:41968) 2023-10-30T13:36:34.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 -- 172.21.15.31:0/3819415103 learned_addr learned my addr 172.21.15.31:0/3819415103 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:34.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 -- 172.21.15.31:0/3819415103 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0815aa60 msgr2=0x7fce080c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 --2- 172.21.15.31:0/3819415103 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0815aa60 0x7fce080c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 -- 172.21.15.31:0/3819415103 >> v2:172.21.15.31:3300/0 conn(0x7fce080c5460 msgr2=0x7fce080706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 --2- 172.21.15.31:0/3819415103 >> v2:172.21.15.31:3300/0 conn(0x7fce080c5460 0x7fce080706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce0659c640 1 -- 172.21.15.31:0/3819415103 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0815b620 con 0x7fce0806dfc0 2023-10-30T13:36:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.828+0000 7fce05d9b640 1 --2- 172.21.15.31:0/3819415103 >> v2:172.21.15.31:3300/0 conn(0x7fce080c5460 0x7fce080706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0659c640 1 --2- 172.21.15.31:0/3819415103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce0806dfc0 0x7fce0806d4f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fce08074280 tx=0x7fcdf801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2111776d7f2a335f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0559a640 1 -- 172.21.15.31:0/3819415103 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf80aa5f0 con 0x7fce0806dfc0 2023-10-30T13:36:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0559a640 1 -- 172.21.15.31:0/3819415103 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf80aa750 con 0x7fce0806dfc0 2023-10-30T13:36:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0559a640 1 -- 172.21.15.31:0/3819415103 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf80aaae0 con 0x7fce0806dfc0 2023-10-30T13:36:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0759e640 1 -- 172.21.15.31:0/3819415103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce0806dfc0 msgr2=0x7fce0806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0759e640 1 --2- 172.21.15.31:0/3819415103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce0806dfc0 0x7fce0806d4f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fce08074280 tx=0x7fcdf801c4d0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0759e640 1 -- 172.21.15.31:0/3819415103 shutdown_connections 2023-10-30T13:36:34.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0759e640 1 --2- 172.21.15.31:0/3819415103 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0815aa60 0x7fce080c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0759e640 1 --2- 172.21.15.31:0/3819415103 >> v2:172.21.15.31:3300/0 conn(0x7fce080c5460 0x7fce080706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0759e640 1 --2- 172.21.15.31:0/3819415103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce0806dfc0 0x7fce0806d4f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.832+0000 7fce0759e640 1 -- 172.21.15.31:0/3819415103 >> 172.21.15.31:0/3819415103 conn(0x7fce080caae0 msgr2=0x7fce080caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.836+0000 7fce0759e640 1 -- 172.21.15.31:0/3819415103 shutdown_connections 2023-10-30T13:36:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.836+0000 7fce0759e640 1 -- 172.21.15.31:0/3819415103 wait complete. 2023-10-30T13:36:34.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.836+0000 7fce0759e640 1 Processor -- start 2023-10-30T13:36:34.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.836+0000 7fce0759e640 1 -- start start 2023-10-30T13:36:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0806dfc0 0x7fce080ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce080c5460 0x7fce0814c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce0815aa60 0x7fce0814ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce08153620 con 0x7fce0815aa60 2023-10-30T13:36:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce081529f0 con 0x7fce080c5460 2023-10-30T13:36:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce080be7b0 con 0x7fce0806dfc0 2023-10-30T13:36:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce05d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce080c5460 0x7fce0814c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0659c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0806dfc0 0x7fce080ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce05d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce080c5460 0x7fce0814c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:41982/0 (socket says 172.21.15.31:41982) 2023-10-30T13:36:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0659c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0806dfc0 0x7fce080ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:60576/0 (socket says 172.21.15.31:60576) 2023-10-30T13:36:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce05d9b640 1 -- 172.21.15.31:0/1489253841 learned_addr learned my addr 172.21.15.31:0/1489253841 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce06d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce0815aa60 0x7fce0814ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce06d9d640 1 -- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0806dfc0 msgr2=0x7fce080ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce06d9d640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0806dfc0 0x7fce080ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce06d9d640 1 -- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce080c5460 msgr2=0x7fce0814c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce06d9d640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce080c5460 0x7fce0814c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce06d9d640 1 -- 172.21.15.31:0/1489253841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc009420 con 0x7fce0815aa60 2023-10-30T13:36:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce06d9d640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce0815aa60 0x7fce0814ceb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fce0000eba0 tx=0x7fce0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fcdf6ffd640 1 -- 172.21.15.31:0/1489253841 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000cd20 con 0x7fce0815aa60 2023-10-30T13:36:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fcdf6ffd640 1 -- 172.21.15.31:0/1489253841 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0000f040 con 0x7fce0815aa60 2023-10-30T13:36:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fcdf6ffd640 1 -- 172.21.15.31:0/1489253841 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce000146c0 con 0x7fce0815aa60 2023-10-30T13:36:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf80094d0 con 0x7fce0815aa60 2023-10-30T13:36:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.840+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce081532b0 con 0x7fce0815aa60 2023-10-30T13:36:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.844+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce08153470 con 0x7fce0815aa60 2023-10-30T13:36:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.844+0000 7fcdf6ffd640 1 -- 172.21.15.31:0/1489253841 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce00014820 con 0x7fce0815aa60 2023-10-30T13:36:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.844+0000 7fcdf6ffd640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde80664a0 0x7fcde8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.844+0000 7fcdf6ffd640 1 -- 172.21.15.31:0/1489253841 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fce00084b60 con 0x7fce0815aa60 2023-10-30T13:36:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.844+0000 7fce0659c640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde80664a0 0x7fcde8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.848+0000 7fce0659c640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde80664a0 0x7fcde8068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80ae000 tx=0x7fcdf80b5040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:34.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:34.852+0000 7fcdf6ffd640 1 -- 172.21.15.31:0/1489253841 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fce00051f90 con 0x7fce0815aa60 2023-10-30T13:36:35.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.244+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce080becd0 con 0x7fce0815aa60 2023-10-30T13:36:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.244+0000 7fcdf6ffd640 1 -- 172.21.15.31:0/1489253841 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5730 (secure 0 0 0) 0x7fce080becd0 con 0x7fce0815aa60 2023-10-30T13:36:35.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:35.249 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:31.769646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:resolve","state_seq":24,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:35.249 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2023-10-30T13:36:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde80664a0 msgr2=0x7fcde8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde80664a0 0x7fcde8068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80ae000 tx=0x7fcdf80b5040 comp rx=0 tx=0).stop 2023-10-30T13:36:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce0815aa60 msgr2=0x7fce0814ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce0815aa60 0x7fce0814ceb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fce0000eba0 tx=0x7fce0000c490 comp rx=0 tx=0).stop 2023-10-30T13:36:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 shutdown_connections 2023-10-30T13:36:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde80664a0 0x7fcde8068960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce0815aa60 0x7fce0814ceb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce080c5460 0x7fce0814c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 --2- 172.21.15.31:0/1489253841 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce0806dfc0 0x7fce080ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 >> 172.21.15.31:0/1489253841 conn(0x7fce080caae0 msgr2=0x7fce080733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 shutdown_connections 2023-10-30T13:36:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:35.248+0000 7fce0759e640 1 -- 172.21.15.31:0/1489253841 wait complete. 2023-10-30T13:36:35.282 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:37.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:37.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.520+0000 7f034ebd3640 1 Processor -- start 2023-10-30T13:36:37.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.520+0000 7f034ebd3640 1 -- start start 2023-10-30T13:36:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f034ebd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c4a10 0x7f034815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:37.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f034ebd3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0348159f20 con 0x7f03480c42a0 2023-10-30T13:36:37.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f034ebd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034815d620 con 0x7f0348153c80 2023-10-30T13:36:37.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f034ebd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034815dec0 con 0x7f03480c4a10 2023-10-30T13:36:37.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f034d170640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f03480c42a0 0x7f03480ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:39288/0 (socket says 172.21.15.31:39288) 2023-10-30T13:36:37.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f034d170640 1 -- 172.21.15.31:0/3480791383 learned_addr learned my addr 172.21.15.31:0/3480791383 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:37.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f034c96f640 1 --1- 172.21.15.31:0/3480791383 >> v1:172.21.15.33:6789/0 conn(0x7f0348153c80 0x7f034815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:53912/0 (socket says 172.21.15.31:53912) 2023-10-30T13:36:37.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f0347fff640 1 --2- 172.21.15.31:0/3480791383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c4a10 0x7f034815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:37.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f03477fe640 1 -- 172.21.15.31:0/3480791383 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096491035 0 0) 0x7f034815d620 con 0x7f0348153c80 2023-10-30T13:36:37.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f03477fe640 1 -- 172.21.15.31:0/3480791383 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0338003530 con 0x7f0348153c80 2023-10-30T13:36:37.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f03477fe640 1 -- 172.21.15.31:0/3480791383 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761564037 0 0) 0x7f0348159f20 con 0x7f03480c42a0 2023-10-30T13:36:37.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f03477fe640 1 -- 172.21.15.31:0/3480791383 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f034815d620 con 0x7f03480c42a0 2023-10-30T13:36:37.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f0347fff640 1 -- 172.21.15.31:0/3480791383 >> v1:172.21.15.33:6789/0 conn(0x7f0348153c80 legacy=0x7f034815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:37.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f0347fff640 1 -- 172.21.15.31:0/3480791383 >> v1:172.21.15.31:6789/0 conn(0x7f03480c42a0 legacy=0x7f03480ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:37.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.524+0000 7f0347fff640 1 -- 172.21.15.31:0/3480791383 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034815d900 con 0x7f03480c4a10 2023-10-30T13:36:37.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.528+0000 7f0347fff640 1 --2- 172.21.15.31:0/3480791383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c4a10 0x7f034815cf90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f03340097f0 tx=0x7f033401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=851f29a1ed5242ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:37.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f03477fe640 1 -- 172.21.15.31:0/3480791383 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03340a89b0 con 0x7f03480c4a10 2023-10-30T13:36:37.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f03477fe640 1 -- 172.21.15.31:0/3480791383 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03340a8b10 con 0x7f03480c4a10 2023-10-30T13:36:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f03477fe640 1 -- 172.21.15.31:0/3480791383 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03340a8e80 con 0x7f03480c4a10 2023-10-30T13:36:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f034ebd3640 1 -- 172.21.15.31:0/3480791383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c4a10 msgr2=0x7f034815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f034ebd3640 1 --2- 172.21.15.31:0/3480791383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c4a10 0x7f034815cf90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f03340097f0 tx=0x7f033401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f034ebd3640 1 -- 172.21.15.31:0/3480791383 shutdown_connections 2023-10-30T13:36:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f034ebd3640 1 --2- 172.21.15.31:0/3480791383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c4a10 0x7f034815cf90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f034ebd3640 1 -- 172.21.15.31:0/3480791383 >> 172.21.15.31:0/3480791383 conn(0x7f03480715c0 msgr2=0x7f03480719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f034ebd3640 1 -- 172.21.15.31:0/3480791383 shutdown_connections 2023-10-30T13:36:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.532+0000 7f034ebd3640 1 -- 172.21.15.31:0/3480791383 wait complete. 2023-10-30T13:36:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 Processor -- start 2023-10-30T13:36:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 -- start start 2023-10-30T13:36:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c42a0 0x7f03480ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03480c4a10 0x7f034814ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348153c80 0x7f034814f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034814e950 con 0x7f03480c4a10 2023-10-30T13:36:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034815c520 con 0x7f0348153c80 2023-10-30T13:36:37.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034ebd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034815c770 con 0x7f03480c42a0 2023-10-30T13:36:37.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03480c4a10 0x7f034814ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:37.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034c96f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c42a0 0x7f03480ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:37.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03480c4a10 0x7f034814ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52654/0 (socket says 172.21.15.31:52654) 2023-10-30T13:36:37.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 -- 172.21.15.31:0/2531012070 learned_addr learned my addr 172.21.15.31:0/2531012070 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:37.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f034d170640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348153c80 0x7f034814f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:37.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 -- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c42a0 msgr2=0x7f03480ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:37.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c42a0 0x7f03480ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 -- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348153c80 msgr2=0x7f034814f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:37.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348153c80 0x7f034814f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.536+0000 7f0347fff640 1 -- 172.21.15.31:0/2531012070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334009420 con 0x7f03480c4a10 2023-10-30T13:36:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.540+0000 7f0347fff640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03480c4a10 0x7f034814ed70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f03340b3040 tx=0x7f033401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.540+0000 7f03457fa640 1 -- 172.21.15.31:0/2531012070 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03340bd070 con 0x7f03480c4a10 2023-10-30T13:36:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.540+0000 7f03457fa640 1 -- 172.21.15.31:0/2531012070 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0334003620 con 0x7f03480c4a10 2023-10-30T13:36:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.540+0000 7f03457fa640 1 -- 172.21.15.31:0/2531012070 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0334003990 con 0x7f03480c4a10 2023-10-30T13:36:37.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.540+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034814dfb0 con 0x7f03480c4a10 2023-10-30T13:36:37.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.540+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034814e570 con 0x7f03480c4a10 2023-10-30T13:36:37.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.540+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034814e730 con 0x7f03480c4a10 2023-10-30T13:36:37.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.544+0000 7f03457fa640 1 -- 172.21.15.31:0/2531012070 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03340b7040 con 0x7f03480c4a10 2023-10-30T13:36:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.544+0000 7f03457fa640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f032c0664a0 0x7f032c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.544+0000 7f03457fa640 1 -- 172.21.15.31:0/2531012070 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f0334121c40 con 0x7f03480c4a10 2023-10-30T13:36:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.544+0000 7f034c96f640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f032c0664a0 0x7f032c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:37.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.544+0000 7f034c96f640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f032c0664a0 0x7f032c068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f033c007fd0 tx=0x7f033c007d60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.552+0000 7f03457fa640 1 -- 172.21.15.31:0/2531012070 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03340ef310 con 0x7f03480c4a10 2023-10-30T13:36:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.936+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f034804e170 con 0x7f03480c4a10 2023-10-30T13:36:37.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.936+0000 7f03457fa640 1 -- 172.21.15.31:0/2531012070 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5730 (secure 0 0 0) 0x7f03340f2930 con 0x7f03480c4a10 2023-10-30T13:36:37.941 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:37.942 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:31.769646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:resolve","state_seq":24,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2023-10-30T13:36:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f032c0664a0 msgr2=0x7f032c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f032c0664a0 0x7f032c068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f033c007fd0 tx=0x7f033c007d60 comp rx=0 tx=0).stop 2023-10-30T13:36:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03480c4a10 msgr2=0x7f034814ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:37.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03480c4a10 0x7f034814ed70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f03340b3040 tx=0x7f033401ba50 comp rx=0 tx=0).stop 2023-10-30T13:36:37.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 shutdown_connections 2023-10-30T13:36:37.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f032c0664a0 0x7f032c068960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348153c80 0x7f034814f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03480c4a10 0x7f034814ed70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 --2- 172.21.15.31:0/2531012070 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f03480c42a0 0x7f03480ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 >> 172.21.15.31:0/2531012070 conn(0x7f03480715c0 msgr2=0x7f03481515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 shutdown_connections 2023-10-30T13:36:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:37.940+0000 7f034ebd3640 1 -- 172.21.15.31:0/2531012070 wait complete. 2023-10-30T13:36:37.973 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:39.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:40.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.208+0000 7fcbb0fd3640 1 Processor -- start 2023-10-30T13:36:40.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.208+0000 7fcbb0fd3640 1 -- start start 2023-10-30T13:36:40.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.208+0000 7fcbb0fd3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fcbac06c420 0x7fcbac166f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:40.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.208+0000 7fcbb0fd3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac070c10 con 0x7fcbac163a30 2023-10-30T13:36:40.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.208+0000 7fcbb0fd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac070ef0 con 0x7fcbac06fdf0 2023-10-30T13:36:40.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.212+0000 7fcbb0fd3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac071b70 con 0x7fcbac06c420 2023-10-30T13:36:40.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.212+0000 7fcbaa59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fcbac163a30 0x7fcbac0c55f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:39294/0 (socket says 172.21.15.31:39294) 2023-10-30T13:36:40.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.212+0000 7fcbaa59c640 1 -- 172.21.15.31:0/483323777 learned_addr learned my addr 172.21.15.31:0/483323777 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:40.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.212+0000 7fcbaad9d640 1 --2- 172.21.15.31:0/483323777 >> v2:172.21.15.31:3300/0 conn(0x7fcbac06c420 0x7fcbac166f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:40.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.212+0000 7fcba959a640 1 -- 172.21.15.31:0/483323777 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224533259 0 0) 0x7fcbac070c10 con 0x7fcbac163a30 2023-10-30T13:36:40.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.220+0000 7fcba959a640 1 -- 172.21.15.31:0/483323777 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb94003510 con 0x7fcbac163a30 2023-10-30T13:36:40.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.220+0000 7fcba959a640 1 -- 172.21.15.31:0/483323777 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709638724 0 0) 0x7fcbac070ef0 con 0x7fcbac06fdf0 2023-10-30T13:36:40.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.220+0000 7fcba959a640 1 -- 172.21.15.31:0/483323777 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbac070c10 con 0x7fcbac06fdf0 2023-10-30T13:36:40.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.220+0000 7fcbaad9d640 1 -- 172.21.15.31:0/483323777 >> v1:172.21.15.33:6789/0 conn(0x7fcbac06fdf0 legacy=0x7fcbac06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:40.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.220+0000 7fcbaad9d640 1 -- 172.21.15.31:0/483323777 >> v1:172.21.15.31:6789/0 conn(0x7fcbac163a30 legacy=0x7fcbac0c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:40.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.220+0000 7fcbaad9d640 1 -- 172.21.15.31:0/483323777 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0711d0 con 0x7fcbac06c420 2023-10-30T13:36:40.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.224+0000 7fcbaad9d640 1 --2- 172.21.15.31:0/483323777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06c420 0x7fcbac166f40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009da0 tx=0x7fcba401f100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0c02db00f1b262 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:40.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcba959a640 1 -- 172.21.15.31:0/483323777 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4003ba0 con 0x7fcbac06c420 2023-10-30T13:36:40.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcba959a640 1 -- 172.21.15.31:0/483323777 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba40ab050 con 0x7fcbac06c420 2023-10-30T13:36:40.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcba959a640 1 -- 172.21.15.31:0/483323777 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba40090d0 con 0x7fcbac06c420 2023-10-30T13:36:40.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/483323777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06c420 msgr2=0x7fcbac166f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:40.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/483323777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06c420 0x7fcbac166f40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009da0 tx=0x7fcba401f100 comp rx=0 tx=0).stop 2023-10-30T13:36:40.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/483323777 shutdown_connections 2023-10-30T13:36:40.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/483323777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06c420 0x7fcbac166f40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:40.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/483323777 >> 172.21.15.31:0/483323777 conn(0x7fcbac0cac70 msgr2=0x7fcbac0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:40.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/483323777 shutdown_connections 2023-10-30T13:36:40.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.228+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/483323777 wait complete. 2023-10-30T13:36:40.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 Processor -- start 2023-10-30T13:36:40.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 -- start start 2023-10-30T13:36:40.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcbac06c420 0x7fcbac164e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:40.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06fdf0 0x7fcbac157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:40.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbac163a30 0x7fcbac0ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:40.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0c8f30 con 0x7fcbac06fdf0 2023-10-30T13:36:40.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac165b20 con 0x7fcbac163a30 2023-10-30T13:36:40.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbb0fd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1645a0 con 0x7fcbac06c420 2023-10-30T13:36:40.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbaad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbac163a30 0x7fcbac0ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:40.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbaa59c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcbac06c420 0x7fcbac164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:40.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcba9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06fdf0 0x7fcbac157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:40.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbaad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbac163a30 0x7fcbac0ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:42000/0 (socket says 172.21.15.31:42000) 2023-10-30T13:36:40.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbaad9d640 1 -- 172.21.15.31:0/3053155257 learned_addr learned my addr 172.21.15.31:0/3053155257 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcba9d9b640 1 -- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcbac06c420 msgr2=0x7fcbac164e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcba9d9b640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcbac06c420 0x7fcbac164e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcba9d9b640 1 -- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbac163a30 msgr2=0x7fcbac0ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcba9d9b640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbac163a30 0x7fcbac0ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcba9d9b640 1 -- 172.21.15.31:0/3053155257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba4009a80 con 0x7fcbac06fdf0 2023-10-30T13:36:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.232+0000 7fcbaad9d640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbac163a30 0x7fcbac0ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.236+0000 7fcba9d9b640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06fdf0 0x7fcbac157690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcba000c3c0 tx=0x7fcba000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.236+0000 7fcb9affd640 1 -- 172.21.15.31:0/3053155257 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00052d0 con 0x7fcbac06fdf0 2023-10-30T13:36:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.236+0000 7fcb9affd640 1 -- 172.21.15.31:0/3053155257 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba0005450 con 0x7fcbac06fdf0 2023-10-30T13:36:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.236+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac0c9140 con 0x7fcbac06fdf0 2023-10-30T13:36:40.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.236+0000 7fcb9affd640 1 -- 172.21.15.31:0/3053155257 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0003840 con 0x7fcbac06fdf0 2023-10-30T13:36:40.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.236+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac166070 con 0x7fcbac06fdf0 2023-10-30T13:36:40.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.236+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac071f70 con 0x7fcbac06fdf0 2023-10-30T13:36:40.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.240+0000 7fcb9affd640 1 -- 172.21.15.31:0/3053155257 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcba00039a0 con 0x7fcbac06fdf0 2023-10-30T13:36:40.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.240+0000 7fcb9affd640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcb8c0664a0 0x7fcb8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:40.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.240+0000 7fcb9affd640 1 -- 172.21.15.31:0/3053155257 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fcba00833f0 con 0x7fcbac06fdf0 2023-10-30T13:36:40.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.240+0000 7fcbaa59c640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcb8c0664a0 0x7fcb8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:40.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.240+0000 7fcbaa59c640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcb8c0664a0 0x7fcb8c068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c000f70 tx=0x7fcb9c007d90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.248+0000 7fcb9affd640 1 -- 172.21.15.31:0/3053155257 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcbac071f70 con 0x7fcbac06fdf0 2023-10-30T13:36:40.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.632+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbac071f70 con 0x7fcbac06fdf0 2023-10-30T13:36:40.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.632+0000 7fcb9affd640 1 -- 172.21.15.31:0/3053155257 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5732 (secure 0 0 0) 0x7fcbac071f70 con 0x7fcbac06fdf0 2023-10-30T13:36:40.636 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:40.636 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:37.765774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:40.636 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 16 2023-10-30T13:36:40.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcb8c0664a0 msgr2=0x7fcb8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:40.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcb8c0664a0 0x7fcb8c068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c000f70 tx=0x7fcb9c007d90 comp rx=0 tx=0).stop 2023-10-30T13:36:40.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06fdf0 msgr2=0x7fcbac157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:40.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06fdf0 0x7fcbac157690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcba000c3c0 tx=0x7fcba000c880 comp rx=0 tx=0).stop 2023-10-30T13:36:40.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 shutdown_connections 2023-10-30T13:36:40.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcb8c0664a0 0x7fcb8c068960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:40.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbac163a30 0x7fcbac0ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:40.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac06fdf0 0x7fcbac157690 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:40.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 --2- 172.21.15.31:0/3053155257 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcbac06c420 0x7fcbac164e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:40.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 >> 172.21.15.31:0/3053155257 conn(0x7fcbac0cac70 msgr2=0x7fcbac158970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:40.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 shutdown_connections 2023-10-30T13:36:40.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:40.636+0000 7fcbb0fd3640 1 -- 172.21.15.31:0/3053155257 wait complete. 2023-10-30T13:36:40.668 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:42.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:42.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79ca7a7640 1 Processor -- start 2023-10-30T13:36:42.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79ca7a7640 1 -- start start 2023-10-30T13:36:42.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79ca7a7640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f79c4153b50 0x7f79c41597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:42.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79ca7a7640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c40c62b0 0x7f79c40ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:42.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79ca7a7640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c415a770 con 0x7f79c40cb180 2023-10-30T13:36:42.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79ca7a7640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c4159cf0 con 0x7f79c4153b50 2023-10-30T13:36:42.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79ca7a7640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c406aa30 con 0x7f79c40c62b0 2023-10-30T13:36:42.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c97a5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f79c4153b50 0x7f79c41597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:42.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c97a5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f79c4153b50 0x7f79c41597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52688/0 (socket says 172.21.15.31:52688) 2023-10-30T13:36:42.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c97a5640 1 -- 172.21.15.31:0/2607988603 learned_addr learned my addr 172.21.15.31:0/2607988603 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:42.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c8fa4640 1 --2- 172.21.15.31:0/2607988603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c40c62b0 0x7f79c40ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c97a5640 1 -- 172.21.15.31:0/2607988603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c40c62b0 msgr2=0x7f79c40ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c97a5640 1 --2- 172.21.15.31:0/2607988603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c40c62b0 0x7f79c40ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c97a5640 1 -- 172.21.15.31:0/2607988603 >> v1:172.21.15.33:6789/0 conn(0x7f79c40cb180 legacy=0x7f79c406ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.816+0000 7f79c97a5640 1 -- 172.21.15.31:0/2607988603 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c40c9d40 con 0x7f79c4153b50 2023-10-30T13:36:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79c97a5640 1 --2- 172.21.15.31:0/2607988603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c4153b50 0x7f79c41597b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009870 tx=0x7f79c001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848ead4653eb3bbf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79bbfff640 1 -- 172.21.15.31:0/2607988603 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c00ab600 con 0x7f79c4153b50 2023-10-30T13:36:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79bbfff640 1 -- 172.21.15.31:0/2607988603 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c00a7050 con 0x7f79c4153b50 2023-10-30T13:36:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79bbfff640 1 -- 172.21.15.31:0/2607988603 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c00ab980 con 0x7f79c4153b50 2023-10-30T13:36:42.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2607988603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c4153b50 msgr2=0x7f79c41597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:42.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2607988603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c4153b50 0x7f79c41597b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009870 tx=0x7f79c001c540 comp rx=0 tx=0).stop 2023-10-30T13:36:42.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2607988603 shutdown_connections 2023-10-30T13:36:42.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2607988603 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c40c62b0 0x7f79c40ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2607988603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c4153b50 0x7f79c41597b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2607988603 >> 172.21.15.31:0/2607988603 conn(0x7f79c4065a80 msgr2=0x7f79c4065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2607988603 shutdown_connections 2023-10-30T13:36:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.820+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2607988603 wait complete. 2023-10-30T13:36:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 Processor -- start 2023-10-30T13:36:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 -- start start 2023-10-30T13:36:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c40c62b0 0x7f79c41f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c40cb180 0x7f79c4159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c4153b50 0x7f79c4147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c40c24d0 con 0x7f79c40cb180 2023-10-30T13:36:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c414ae40 con 0x7f79c40c62b0 2023-10-30T13:36:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c4152910 con 0x7f79c4153b50 2023-10-30T13:36:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c40cb180 0x7f79c4159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c40cb180 0x7f79c4159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52694/0 (socket says 172.21.15.31:52694) 2023-10-30T13:36:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 -- 172.21.15.31:0/2923911397 learned_addr learned my addr 172.21.15.31:0/2923911397 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c97a5640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c40c62b0 0x7f79c41f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c9fa6640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c4153b50 0x7f79c4147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 -- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c4153b50 msgr2=0x7f79c4147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c4153b50 0x7f79c4147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 -- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c40c62b0 msgr2=0x7f79c41f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c40c62b0 0x7f79c41f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 -- 172.21.15.31:0/2923911397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00094d0 con 0x7f79c40cb180 2023-10-30T13:36:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c9fa6640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c4153b50 0x7f79c4147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79c8fa4640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c40cb180 0x7f79c4159230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f79b400d7d0 tx=0x7f79b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79b9ffb640 1 -- 172.21.15.31:0/2923911397 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4018070 con 0x7f79c40cb180 2023-10-30T13:36:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79b9ffb640 1 -- 172.21.15.31:0/2923911397 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b4004020 con 0x7f79c40cb180 2023-10-30T13:36:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79b9ffb640 1 -- 172.21.15.31:0/2923911397 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b40041d0 con 0x7f79c40cb180 2023-10-30T13:36:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c40c7e50 con 0x7f79c40cb180 2023-10-30T13:36:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.824+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c406a650 con 0x7f79c40cb180 2023-10-30T13:36:42.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.828+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c40c5dc0 con 0x7f79c40cb180 2023-10-30T13:36:42.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.828+0000 7f79b9ffb640 1 -- 172.21.15.31:0/2923911397 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79b4004330 con 0x7f79c40cb180 2023-10-30T13:36:42.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.828+0000 7f79b9ffb640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f79a80664d0 0x7f79a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:42.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.828+0000 7f79b9ffb640 1 -- 172.21.15.31:0/2923911397 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f79b4083df0 con 0x7f79c40cb180 2023-10-30T13:36:42.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.828+0000 7f79c97a5640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f79a80664d0 0x7f79a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.828+0000 7f79c97a5640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f79a80664d0 0x7f79a8068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79c406ad60 tx=0x7f79c0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:42.832+0000 7f79b9ffb640 1 -- 172.21.15.31:0/2923911397 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f79b40510e0 con 0x7f79c40cb180 2023-10-30T13:36:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.040+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79c4067540 con 0x7f79c40cb180 2023-10-30T13:36:43.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.040+0000 7f79b9ffb640 1 -- 172.21.15.31:0/2923911397 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f79b4054700 con 0x7f79c40cb180 2023-10-30T13:36:43.044 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:43.044 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:37.765774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 16 2023-10-30T13:36:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f79a80664d0 msgr2=0x7f79a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f79a80664d0 0x7f79a8068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79c406ad60 tx=0x7f79c0008040 comp rx=0 tx=0).stop 2023-10-30T13:36:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c40cb180 msgr2=0x7f79c4159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c40cb180 0x7f79c4159230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f79b400d7d0 tx=0x7f79b400dc90 comp rx=0 tx=0).stop 2023-10-30T13:36:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 shutdown_connections 2023-10-30T13:36:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f79a80664d0 0x7f79a8068990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f79c4153b50 0x7f79c4147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c40cb180 0x7f79c4159230 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 --2- 172.21.15.31:0/2923911397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c40c62b0 0x7f79c41f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 >> 172.21.15.31:0/2923911397 conn(0x7f79c4065a80 msgr2=0x7f79c414f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 shutdown_connections 2023-10-30T13:36:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:43.044+0000 7f79ca7a7640 1 -- 172.21.15.31:0/2923911397 wait complete. 2023-10-30T13:36:43.067 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4272": { "addr": "172.21.15.31:6837/1956809670", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1956809670, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1956809670, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:reconnect", "state_seq": 26 }, "gid_4276": { "addr": "172.21.15.33:6837/3402459191", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 3402459191, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 3402459191, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4276, "incarnation": 10, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:36:37.765774+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4276, "mds_2": 4272 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:36:45.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:45.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd35d1d640 1 Processor -- start 2023-10-30T13:36:45.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd35d1d640 1 -- start start 2023-10-30T13:36:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd35d1d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7efd300c4a10 0x7efd3015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd35d1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd300c42a0 0x7efd300ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd35d1d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd30159f20 con 0x7efd30153c80 2023-10-30T13:36:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd35d1d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3015dec0 con 0x7efd300c4a10 2023-10-30T13:36:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd35d1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd301599a0 con 0x7efd300c42a0 2023-10-30T13:36:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2effd640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7efd300c4a10 0x7efd3015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd300c42a0 0x7efd300ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2effd640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7efd300c4a10 0x7efd3015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:48676/0 (socket says 172.21.15.31:48676) 2023-10-30T13:36:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd300c42a0 0x7efd300ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:42926/0 (socket says 172.21.15.31:42926) 2023-10-30T13:36:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2effd640 1 -- 172.21.15.31:0/529008119 learned_addr learned my addr 172.21.15.31:0/529008119 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2f7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7efd30153c80 0x7efd3015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:34730/0 (socket says 172.21.15.31:34730) 2023-10-30T13:36:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2ffff640 1 -- 172.21.15.31:0/529008119 >> v2:172.21.15.33:3300/0 conn(0x7efd300c4a10 msgr2=0x7efd3015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2ffff640 1 --2- 172.21.15.31:0/529008119 >> v2:172.21.15.33:3300/0 conn(0x7efd300c4a10 0x7efd3015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2ffff640 1 -- 172.21.15.31:0/529008119 >> v1:172.21.15.33:6789/0 conn(0x7efd30153c80 legacy=0x7efd3015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.212+0000 7efd2ffff640 1 -- 172.21.15.31:0/529008119 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3015d620 con 0x7efd300c42a0 2023-10-30T13:36:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd2ffff640 1 --2- 172.21.15.31:0/529008119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd300c42a0 0x7efd300ca1e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd28009d90 tx=0x7efd2801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c46ee8a38a06db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:45.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd2e7fc640 1 -- 172.21.15.31:0/529008119 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd280af600 con 0x7efd300c42a0 2023-10-30T13:36:45.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd2e7fc640 1 -- 172.21.15.31:0/529008119 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd280ab050 con 0x7efd300c42a0 2023-10-30T13:36:45.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd2e7fc640 1 -- 172.21.15.31:0/529008119 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd280af980 con 0x7efd300c42a0 2023-10-30T13:36:45.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 -- 172.21.15.31:0/529008119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd300c42a0 msgr2=0x7efd300ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:45.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 --2- 172.21.15.31:0/529008119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd300c42a0 0x7efd300ca1e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd28009d90 tx=0x7efd2801f130 comp rx=0 tx=0).stop 2023-10-30T13:36:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 -- 172.21.15.31:0/529008119 shutdown_connections 2023-10-30T13:36:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 --2- 172.21.15.31:0/529008119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd300c42a0 0x7efd300ca1e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 --2- 172.21.15.31:0/529008119 >> v2:172.21.15.33:3300/0 conn(0x7efd300c4a10 0x7efd3015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 -- 172.21.15.31:0/529008119 >> 172.21.15.31:0/529008119 conn(0x7efd300715c0 msgr2=0x7efd300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 -- 172.21.15.31:0/529008119 shutdown_connections 2023-10-30T13:36:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.216+0000 7efd35d1d640 1 -- 172.21.15.31:0/529008119 wait complete. 2023-10-30T13:36:45.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 Processor -- start 2023-10-30T13:36:45.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 -- start start 2023-10-30T13:36:45.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd300c42a0 0x7efd3015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:45.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd300c4a10 0x7efd300c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd30153c80 0x7efd3015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3015c210 con 0x7efd300c42a0 2023-10-30T13:36:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd301ef280 con 0x7efd300c4a10 2023-10-30T13:36:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd301ef4f0 con 0x7efd30153c80 2023-10-30T13:36:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd300c42a0 0x7efd3015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd300c42a0 0x7efd3015ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48232/0 (socket says 172.21.15.31:48232) 2023-10-30T13:36:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 -- 172.21.15.31:0/3580180692 learned_addr learned my addr 172.21.15.31:0/3580180692 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2ffff640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd30153c80 0x7efd3015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2effd640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd300c4a10 0x7efd300c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 -- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd30153c80 msgr2=0x7efd3015c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd30153c80 0x7efd3015c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 -- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd300c4a10 msgr2=0x7efd300c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd300c4a10 0x7efd300c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 -- 172.21.15.31:0/3580180692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28009a70 con 0x7efd300c42a0 2023-10-30T13:36:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2effd640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd300c4a10 0x7efd300c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd2f7fe640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd300c42a0 0x7efd3015ab60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd2000c430 tx=0x7efd2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd17fff640 1 -- 172.21.15.31:0/3580180692 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd200074e0 con 0x7efd300c42a0 2023-10-30T13:36:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd17fff640 1 -- 172.21.15.31:0/3580180692 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd20007640 con 0x7efd300c42a0 2023-10-30T13:36:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd17fff640 1 -- 172.21.15.31:0/3580180692 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd20004a10 con 0x7efd300c42a0 2023-10-30T13:36:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.220+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3015bea0 con 0x7efd300c42a0 2023-10-30T13:36:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.224+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3015b1c0 con 0x7efd300c42a0 2023-10-30T13:36:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.224+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3015bd00 con 0x7efd300c42a0 2023-10-30T13:36:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.224+0000 7efd17fff640 1 -- 172.21.15.31:0/3580180692 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd20004b70 con 0x7efd300c42a0 2023-10-30T13:36:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.224+0000 7efd17fff640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.224+0000 7efd17fff640 1 -- 172.21.15.31:0/3580180692 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7efd20083980 con 0x7efd300c42a0 2023-10-30T13:36:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.224+0000 7efd2effd640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.228+0000 7efd2effd640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efd301575a0 tx=0x7efd24008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.228+0000 7efd17fff640 1 -- 172.21.15.31:0/3580180692 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd3015bd00 con 0x7efd300c42a0 2023-10-30T13:36:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.420+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd3015bd00 con 0x7efd300c42a0 2023-10-30T13:36:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.420+0000 7efd17fff640 1 -- 172.21.15.31:0/3580180692 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5729 (secure 0 0 0) 0x7efd3015bd00 con 0x7efd300c42a0 2023-10-30T13:36:45.425 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:45.425 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:43.766457+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 17 2023-10-30T13:36:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 msgr2=0x7efd1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efd301575a0 tx=0x7efd24008040 comp rx=0 tx=0).stop 2023-10-30T13:36:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd300c42a0 msgr2=0x7efd3015ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd300c42a0 0x7efd3015ab60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd2000c430 tx=0x7efd2000c8f0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 shutdown_connections 2023-10-30T13:36:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd30153c80 0x7efd3015c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd300c4a10 0x7efd300c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 --2- 172.21.15.31:0/3580180692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd300c42a0 0x7efd3015ab60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 >> 172.21.15.31:0/3580180692 conn(0x7efd300715c0 msgr2=0x7efd301587d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 shutdown_connections 2023-10-30T13:36:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:45.424+0000 7efd35d1d640 1 -- 172.21.15.31:0/3580180692 wait complete. 2023-10-30T13:36:45.447 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:45.731 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 65% (7896/12071) Updating files: 66% (7967/12071) Updating files: 67% (8088/12071) Updating files: 68% (8209/12071) Updating files: 69% (8329/12071) Updating files: 70% (8450/12071) Updating files: 71% (8571/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 75% (9054/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 78% (9416/12071) Updating files: 79% (9537/12071) Updating files: 80% (9657/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 86% (10382/12071) Updating files: 87% (10502/12071) Updating files: 88% (10623/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T13:36:46.301 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to '00c1c13cea21a686b046584596fbeac5315a7d71'. 2023-10-30T13:36:46.301 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:36:46.301 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T13:36:46.301 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T13:36:46.302 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T13:36:46.302 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:36:46.302 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T13:36:46.302 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T13:36:46.302 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:36:46.303 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2023-10-30T13:36:46.303 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:36:46.303 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2023-10-30T13:36:46.303 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:36:46.303 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2023-10-30T13:36:46.303 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:36:46.304 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T13:36:46.304 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:36:46.304 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at 00c1c13cea2 mds/MDBalancer: ignore queued callbacks if MDS is not active 2023-10-30T13:36:46.306 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T13:36:46.358 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T13:36:46.361 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T13:36:46.361 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T13:36:47.192 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T13:36:47.283 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T13:36:47.340 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T13:36:47.343 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T13:36:47.343 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T13:36:47.398 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T13:36:47.404 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:36:47.404 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T13:36:47.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:47.458 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-30T13:36:47.460 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-30T13:36:47.460 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-30T13:36:47.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.588+0000 7f22a3d65640 1 Processor -- start 2023-10-30T13:36:47.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a3d65640 1 -- start start 2023-10-30T13:36:47.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a3d65640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f229c153c80 0x7f229c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:47.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a3d65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c42a0 0x7f229c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:47.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a3d65640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c159f20 con 0x7f229c0c4a10 2023-10-30T13:36:47.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a3d65640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c15dec0 con 0x7f229c153c80 2023-10-30T13:36:47.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a3d65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c1599a0 con 0x7f229c0c42a0 2023-10-30T13:36:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a2302640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c42a0 0x7f229c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a1b01640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f229c153c80 0x7f229c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a2302640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c42a0 0x7f229c0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:42948/0 (socket says 172.21.15.31:42948) 2023-10-30T13:36:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a2302640 1 -- 172.21.15.31:0/1184569286 learned_addr learned my addr 172.21.15.31:0/1184569286 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a1b01640 1 -- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c42a0 msgr2=0x7f229c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:47.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a1b01640 1 --2- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c42a0 0x7f229c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a1b01640 1 -- 172.21.15.31:0/1184569286 >> v1:172.21.15.33:6789/0 conn(0x7f229c0c4a10 legacy=0x7f229c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:47.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a1b01640 1 -- 172.21.15.31:0/1184569286 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c15d620 con 0x7f229c153c80 2023-10-30T13:36:47.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.592+0000 7f22a2302640 1 --2- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c42a0 0x7f229c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:47.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a1b01640 1 --2- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c15cf90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22980097c0 tx=0x7f229801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73333ffe8691797a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a0aff640 1 -- 172.21.15.31:0/1184569286 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980a8970 con 0x7f229c153c80 2023-10-30T13:36:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a0aff640 1 -- 172.21.15.31:0/1184569286 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22980a8ad0 con 0x7f229c153c80 2023-10-30T13:36:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a0aff640 1 -- 172.21.15.31:0/1184569286 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980a8e60 con 0x7f229c153c80 2023-10-30T13:36:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 -- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 msgr2=0x7f229c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 --2- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c15cf90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22980097c0 tx=0x7f229801ec70 comp rx=0 tx=0).stop 2023-10-30T13:36:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 -- 172.21.15.31:0/1184569286 shutdown_connections 2023-10-30T13:36:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 --2- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c42a0 0x7f229c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 --2- 172.21.15.31:0/1184569286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c15cf90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 -- 172.21.15.31:0/1184569286 >> 172.21.15.31:0/1184569286 conn(0x7f229c0715c0 msgr2=0x7f229c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 -- 172.21.15.31:0/1184569286 shutdown_connections 2023-10-30T13:36:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.596+0000 7f22a3d65640 1 -- 172.21.15.31:0/1184569286 wait complete. 2023-10-30T13:36:47.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 Processor -- start 2023-10-30T13:36:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 -- start start 2023-10-30T13:36:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229c0c42a0 0x7f229c151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c4a10 0x7f229c0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c0c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c154950 con 0x7f229c0c42a0 2023-10-30T13:36:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c156e20 con 0x7f229c153c80 2023-10-30T13:36:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c1f23e0 con 0x7f229c0c4a10 2023-10-30T13:36:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229c0c42a0 0x7f229c151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229c0c42a0 0x7f229c151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48240/0 (socket says 172.21.15.31:48240) 2023-10-30T13:36:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 -- 172.21.15.31:0/3219421210 learned_addr learned my addr 172.21.15.31:0/3219421210 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a2302640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c0c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:47.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1300640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c4a10 0x7f229c0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:47.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 -- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c4a10 msgr2=0x7f229c0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:47.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c4a10 0x7f229c0c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 -- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 msgr2=0x7f229c0c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:47.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c0c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 -- 172.21.15.31:0/3219421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2294009b30 con 0x7f229c0c42a0 2023-10-30T13:36:47.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a2302640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c0c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:47.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a1b01640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229c0c42a0 0x7f229c151940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f22980b3040 tx=0x7f229801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:47.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22927fc640 1 -- 172.21.15.31:0/3219421210 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980b9630 con 0x7f229c0c42a0 2023-10-30T13:36:47.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22927fc640 1 -- 172.21.15.31:0/3219421210 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22980b97b0 con 0x7f229c0c42a0 2023-10-30T13:36:47.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22927fc640 1 -- 172.21.15.31:0/3219421210 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980b9b60 con 0x7f229c0c42a0 2023-10-30T13:36:47.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2298009420 con 0x7f229c0c42a0 2023-10-30T13:36:47.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c154e40 con 0x7f229c0c42a0 2023-10-30T13:36:47.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.600+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c04e170 con 0x7f229c0c42a0 2023-10-30T13:36:47.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.604+0000 7f22927fc640 1 -- 172.21.15.31:0/3219421210 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22980b7040 con 0x7f229c0c42a0 2023-10-30T13:36:47.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.604+0000 7f22927fc640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f22800664a0 0x7f2280068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:47.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.604+0000 7f22927fc640 1 -- 172.21.15.31:0/3219421210 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f2298121db0 con 0x7f229c0c42a0 2023-10-30T13:36:47.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.604+0000 7f22a1300640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f22800664a0 0x7f2280068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:47.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.604+0000 7f22a1300640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f22800664a0 0x7f2280068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f229c0ca7b0 tx=0x7f228c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:47.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.608+0000 7f22927fc640 1 -- 172.21.15.31:0/3219421210 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2298126070 con 0x7f229c0c42a0 2023-10-30T13:36:47.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.796+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f229c0cad40 con 0x7f229c0c42a0 2023-10-30T13:36:47.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22927fc640 1 -- 172.21.15.31:0/3219421210 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5729 (secure 0 0 0) 0x7f22980b5020 con 0x7f229c0c42a0 2023-10-30T13:36:47.803 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:47.803 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:43.766457+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:47.803 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 17 2023-10-30T13:36:47.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f22800664a0 msgr2=0x7f2280068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:47.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f22800664a0 0x7f2280068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f229c0ca7b0 tx=0x7f228c009040 comp rx=0 tx=0).stop 2023-10-30T13:36:47.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229c0c42a0 msgr2=0x7f229c151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:47.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229c0c42a0 0x7f229c151940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f22980b3040 tx=0x7f229801ba10 comp rx=0 tx=0).stop 2023-10-30T13:36:47.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 shutdown_connections 2023-10-30T13:36:47.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f22800664a0 0x7f2280068960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f229c153c80 0x7f229c0c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f229c0c4a10 0x7f229c0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 --2- 172.21.15.31:0/3219421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229c0c42a0 0x7f229c151940 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:47.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 >> 172.21.15.31:0/3219421210 conn(0x7f229c0715c0 msgr2=0x7f229c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:47.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 shutdown_connections 2023-10-30T13:36:47.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:47.800+0000 7f22a3d65640 1 -- 172.21.15.31:0/3219421210 wait complete. 2023-10-30T13:36:47.824 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:48.816 INFO:tasks.workunit.client.0.smithi031.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-30T13:36:49.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:49.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a7005640 1 Processor -- start 2023-10-30T13:36:49.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a7005640 1 -- start start 2023-10-30T13:36:49.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a7005640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe6a00c4a10 0x7fe6a015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:49.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a7005640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:49.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a7005640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a0159f20 con 0x7fe6a0153c80 2023-10-30T13:36:49.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a7005640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a015dec0 con 0x7fe6a00c4a10 2023-10-30T13:36:49.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a7005640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a01599a0 con 0x7fe6a00c42a0 2023-10-30T13:36:49.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe69ffff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe6a00c4a10 0x7fe6a015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:49.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe69ffff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe6a00c4a10 0x7fe6a015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:48720/0 (socket says 172.21.15.31:48720) 2023-10-30T13:36:49.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a4da1640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fe6a0153c80 0x7fe6a015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:34744/0 (socket says 172.21.15.31:34744) 2023-10-30T13:36:49.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe69ffff640 1 -- 172.21.15.31:0/1203876921 learned_addr learned my addr 172.21.15.31:0/1203876921 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:49.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a55a2640 1 --2- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:49.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe69ffff640 1 -- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 msgr2=0x7fe6a00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:49.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe69ffff640 1 --2- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:49.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe69ffff640 1 -- 172.21.15.31:0/1203876921 >> v1:172.21.15.33:6789/0 conn(0x7fe6a0153c80 legacy=0x7fe6a015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:49.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe69ffff640 1 -- 172.21.15.31:0/1203876921 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a015d620 con 0x7fe6a00c4a10 2023-10-30T13:36:49.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.968+0000 7fe6a55a2640 1 --2- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe69ffff640 1 --2- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a015cf90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009870 tx=0x7fe68c01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92e6ca8e9062eec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe69f7fe640 1 -- 172.21.15.31:0/1203876921 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0a8a30 con 0x7fe6a00c4a10 2023-10-30T13:36:49.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe69f7fe640 1 -- 172.21.15.31:0/1203876921 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c0a8b90 con 0x7fe6a00c4a10 2023-10-30T13:36:49.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe69f7fe640 1 -- 172.21.15.31:0/1203876921 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0b0440 con 0x7fe6a00c4a10 2023-10-30T13:36:49.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 -- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 msgr2=0x7fe6a015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 --2- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a015cf90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009870 tx=0x7fe68c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:36:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 -- 172.21.15.31:0/1203876921 shutdown_connections 2023-10-30T13:36:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 --2- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 --2- 172.21.15.31:0/1203876921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a015cf90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 -- 172.21.15.31:0/1203876921 >> 172.21.15.31:0/1203876921 conn(0x7fe6a00715c0 msgr2=0x7fe6a00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:49.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 -- 172.21.15.31:0/1203876921 shutdown_connections 2023-10-30T13:36:49.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.972+0000 7fe6a7005640 1 -- 172.21.15.31:0/1203876921 wait complete. 2023-10-30T13:36:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 Processor -- start 2023-10-30T13:36:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 -- start start 2023-10-30T13:36:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a00c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6a0153c80 0x7fe6a015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a015b330 con 0x7fe6a0153c80 2023-10-30T13:36:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a015cc30 con 0x7fe6a00c4a10 2023-10-30T13:36:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a015cf10 con 0x7fe6a00c42a0 2023-10-30T13:36:49.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6a0153c80 0x7fe6a015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6a0153c80 0x7fe6a015bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48246/0 (socket says 172.21.15.31:48246) 2023-10-30T13:36:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 -- 172.21.15.31:0/2666611260 learned_addr learned my addr 172.21.15.31:0/2666611260 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a4da1640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe69ffff640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a00c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 -- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 msgr2=0x7fe6a015b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a015b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 -- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 msgr2=0x7fe6a00c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a00c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 -- 172.21.15.31:0/2666611260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698009a70 con 0x7fe6a0153c80 2023-10-30T13:36:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe69ffff640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a00c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a55a2640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6a0153c80 0x7fe6a015bf80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe69800efc0 tx=0x7fe698009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe69d7fa640 1 -- 172.21.15.31:0/2666611260 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698009d70 con 0x7fe6a0153c80 2023-10-30T13:36:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe69d7fa640 1 -- 172.21.15.31:0/2666611260 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69800f040 con 0x7fe6a0153c80 2023-10-30T13:36:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe69d7fa640 1 -- 172.21.15.31:0/2666611260 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6980039a0 con 0x7fe6a0153c80 2023-10-30T13:36:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c0094d0 con 0x7fe6a0153c80 2023-10-30T13:36:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.976+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a015afc0 con 0x7fe6a0153c80 2023-10-30T13:36:49.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.980+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a015b4e0 con 0x7fe6a0153c80 2023-10-30T13:36:49.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.980+0000 7fe69d7fa640 1 -- 172.21.15.31:0/2666611260 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe6980144a0 con 0x7fe6a0153c80 2023-10-30T13:36:49.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.980+0000 7fe69d7fa640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6840664a0 0x7fe684068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:49.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.980+0000 7fe69d7fa640 1 -- 172.21.15.31:0/2666611260 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fe698084cf0 con 0x7fe6a0153c80 2023-10-30T13:36:49.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.980+0000 7fe6a4da1640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6840664a0 0x7fe684068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:49.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.980+0000 7fe6a4da1640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6840664a0 0x7fe684068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe694000f70 tx=0x7fe694007cf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:49.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:49.984+0000 7fe69d7fa640 1 -- 172.21.15.31:0/2666611260 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe6a015b4e0 con 0x7fe6a0153c80 2023-10-30T13:36:50.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.196+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6a015b4e0 con 0x7fe6a0153c80 2023-10-30T13:36:50.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.196+0000 7fe69d7fa640 1 -- 172.21.15.31:0/2666611260 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5729 (secure 0 0 0) 0x7fe6a015b4e0 con 0x7fe6a0153c80 2023-10-30T13:36:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:50.200 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:48.802383+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:50.201 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 18 2023-10-30T13:36:50.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.196+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6840664a0 msgr2=0x7fe684068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:50.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.196+0000 7fe6a7005640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6840664a0 0x7fe684068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe694000f70 tx=0x7fe694007cf0 comp rx=0 tx=0).stop 2023-10-30T13:36:50.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.196+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6a0153c80 msgr2=0x7fe6a015bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:50.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.196+0000 7fe6a7005640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6a0153c80 0x7fe6a015bf80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe69800efc0 tx=0x7fe698009f90 comp rx=0 tx=0).stop 2023-10-30T13:36:50.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 shutdown_connections 2023-10-30T13:36:50.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6840664a0 0x7fe684068960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:50.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6a0153c80 0x7fe6a015bf80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:50.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6a00c4a10 0x7fe6a00c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:50.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 --2- 172.21.15.31:0/2666611260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6a00c42a0 0x7fe6a015b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:50.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 >> 172.21.15.31:0/2666611260 conn(0x7fe6a00715c0 msgr2=0x7fe6a00719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:50.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 shutdown_connections 2023-10-30T13:36:50.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:50.200+0000 7fe6a7005640 1 -- 172.21.15.31:0/2666611260 wait complete. 2023-10-30T13:36:50.222 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=2, replacing gid=4273 2023-10-30T13:36:50.222 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.d 2023-10-30T13:36:51.461 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir foo 2023-10-30T13:36:51.474 INFO:tasks.workunit.client.0.smithi031.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-30T13:36:51.576 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir foo/.snap/barsnap 2023-10-30T13:36:53.848 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-30T13:36:53.869 INFO:tasks.workunit.client.0.smithi031.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-30T13:36:54.475 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo passed: files are identical 2023-10-30T13:36:54.475 INFO:tasks.workunit.client.0.smithi031.stdout:passed: files are identical 2023-10-30T13:36:54.476 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir foo/.snap/barsnap 2023-10-30T13:36:55.008 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:36:55.009 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2023-10-30T13:36:55.010 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:36:55.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:36:55.060 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-30T13:36:55.061 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-30T13:36:55.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:55.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:55.128 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 1 2023-10-30T13:36:55.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 2 2023-10-30T13:36:55.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 3 2023-10-30T13:36:55.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ '[' -e file4 ']' 2023-10-30T13:36:55.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir .snap/snap1 2023-10-30T13:36:55.151 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 4 2023-10-30T13:36:55.154 INFO:tasks.workunit.client.0.smithi031.stderr:++ ls 2023-10-30T13:36:55.161 INFO:tasks.workunit.client.0.smithi031.stderr:+ now='file1 2023-10-30T13:36:55.161 INFO:tasks.workunit.client.0.smithi031.stderr:file2 2023-10-30T13:36:55.161 INFO:tasks.workunit.client.0.smithi031.stderr:file3 2023-10-30T13:36:55.161 INFO:tasks.workunit.client.0.smithi031.stderr:file4' 2023-10-30T13:36:55.162 INFO:tasks.workunit.client.0.smithi031.stderr:++ ls .snap/snap1 2023-10-30T13:36:55.171 INFO:tasks.workunit.client.0.smithi031.stderr:+ then='file1 2023-10-30T13:36:55.171 INFO:tasks.workunit.client.0.smithi031.stderr:file2 2023-10-30T13:36:55.172 INFO:tasks.workunit.client.0.smithi031.stderr:file3' 2023-10-30T13:36:55.172 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir .snap/snap1 2023-10-30T13:36:55.182 INFO:tasks.workunit.client.0.smithi031.stderr:+ '[' 'file1 2023-10-30T13:36:55.182 INFO:tasks.workunit.client.0.smithi031.stderr:file2 2023-10-30T13:36:55.182 INFO:tasks.workunit.client.0.smithi031.stderr:file3 2023-10-30T13:36:55.182 INFO:tasks.workunit.client.0.smithi031.stderr:file4' = 'file1 2023-10-30T13:36:55.182 INFO:tasks.workunit.client.0.smithi031.stderr:file2 2023-10-30T13:36:55.183 INFO:tasks.workunit.client.0.smithi031.stderr:file3' ']' 2023-10-30T13:36:55.183 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 1 2023-10-30T13:36:55.191 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 2 2023-10-30T13:36:55.200 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 3 2023-10-30T13:36:55.210 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir .snap/snap1 2023-10-30T13:36:55.220 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 4 2023-10-30T13:36:55.229 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir .snap/snap1 2023-10-30T13:36:55.240 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm file1 file2 file3 file4 2023-10-30T13:36:55.248 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2023-10-30T13:36:55.249 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:36:55.249 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:36:55.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:36:55.321 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-30T13:36:55.322 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-30T13:36:55.389 INFO:tasks.workunit.client.0.smithi031.stdout:Create dir 100 to 199 ... 2023-10-30T13:36:55.391 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 100 2023-10-30T13:36:55.400 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 101 2023-10-30T13:36:55.422 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 102 2023-10-30T13:36:55.444 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 103 2023-10-30T13:36:55.465 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 104 2023-10-30T13:36:55.712 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 105 2023-10-30T13:36:56.223 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:36:56.224 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:36:56.224 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:36:56.227 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:36:56.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:56.307 INFO:tasks.ceph.mds.d.smithi033.stdout:starting mds.d at 2023-10-30T13:36:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.375+0000 7fa09fcea640 1 Processor -- start 2023-10-30T13:36:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.375+0000 7fa09fcea640 1 -- start start 2023-10-30T13:36:56.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.375+0000 7fa09fcea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa0980c2c10 0x7fa0980cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09fcea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa098151aa0 con 0x7fa0980ccf60 2023-10-30T13:36:56.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09fcea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa098158af0 con 0x7fa0981516c0 2023-10-30T13:36:56.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09fcea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0980c74a0 con 0x7fa0980c2c10 2023-10-30T13:36:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fa0980ccf60 0x7fa0980c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:50636/0 (socket says 172.21.15.31:50636) 2023-10-30T13:36:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 -- 172.21.15.31:0/1114070218 learned_addr learned my addr 172.21.15.31:0/1114070218 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09e287640 1 --2- 172.21.15.31:0/1114070218 >> v2:172.21.15.33:3300/0 conn(0x7fa0980c2c10 0x7fa0980cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653500033 0 0) 0x7fa098151aa0 con 0x7fa0980ccf60 2023-10-30T13:36:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa084003530 con 0x7fa0980ccf60 2023-10-30T13:36:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2958814158 0 0) 0x7fa084003530 con 0x7fa0980ccf60 2023-10-30T13:36:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 >> v2:172.21.15.33:3300/0 conn(0x7fa0980c2c10 msgr2=0x7fa0980cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 --2- 172.21.15.31:0/1114070218 >> v2:172.21.15.33:3300/0 conn(0x7fa0980c2c10 0x7fa0980cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 >> v1:172.21.15.33:6789/0 conn(0x7fa0981516c0 legacy=0x7fa0980c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa098158510 con 0x7fa0980ccf60 2023-10-30T13:36:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fa088001680 con 0x7fa0980ccf60 2023-10-30T13:36:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09e287640 1 --2- 172.21.15.31:0/1114070218 >> v2:172.21.15.33:3300/0 conn(0x7fa0980c2c10 0x7fa0980cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 >> v1:172.21.15.31:6789/0 conn(0x7fa0980ccf60 legacy=0x7fa0980c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa084003980 0x7fa084003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0840047d0 0x7fa084004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa084005190 0x7fa08400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0980c74a0 con 0x7fa0840047d0 2023-10-30T13:36:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08400c030 con 0x7fa084003980 2023-10-30T13:36:56.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08400c1f0 con 0x7fa084005190 2023-10-30T13:36:56.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0840047d0 0x7fa084004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09da86640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa084003980 0x7fa084003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09e287640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa084005190 0x7fa08400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 -- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa084005190 msgr2=0x7fa08400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:56.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa084005190 0x7fa08400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 -- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa084003980 msgr2=0x7fa084003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:56.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa084003980 0x7fa084003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09e287640 1 -- 172.21.15.31:0/1114070218 reap_dead start 2023-10-30T13:36:56.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.379+0000 7fa09d285640 1 -- 172.21.15.31:0/1114070218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa084003530 con 0x7fa0840047d0 2023-10-30T13:36:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09d285640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0840047d0 0x7fa084004c50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa08800c1c0 tx=0x7fa08801f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454431096acf153f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa088005ae0 con 0x7fa0840047d0 2023-10-30T13:36:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0880ab070 con 0x7fa0840047d0 2023-10-30T13:36:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09ca84640 1 -- 172.21.15.31:0/1114070218 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa088007070 con 0x7fa0840047d0 2023-10-30T13:36:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 -- 172.21.15.31:0/1114070218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0840047d0 msgr2=0x7fa084004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0840047d0 0x7fa084004c50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa08800c1c0 tx=0x7fa08801f310 comp rx=0 tx=0).stop 2023-10-30T13:36:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 -- 172.21.15.31:0/1114070218 shutdown_connections 2023-10-30T13:36:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 --2- 172.21.15.31:0/1114070218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0840047d0 0x7fa084004c50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 -- 172.21.15.31:0/1114070218 >> 172.21.15.31:0/1114070218 conn(0x7fa098071fb0 msgr2=0x7fa0980723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 -- 172.21.15.31:0/1114070218 shutdown_connections 2023-10-30T13:36:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 -- 172.21.15.31:0/1114070218 wait complete. 2023-10-30T13:36:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 Processor -- start 2023-10-30T13:36:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.383+0000 7fa09fcea640 1 -- start start 2023-10-30T13:36:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa0840047d0 0x7fa0980c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0980c5580 0x7fa098070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09814fbc0 0x7fa098150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa098152460 con 0x7fa09814fbc0 2023-10-30T13:36:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa098153910 con 0x7fa0980c5580 2023-10-30T13:36:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa098153bb0 con 0x7fa0840047d0 2023-10-30T13:36:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09814fbc0 0x7fa098150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09814fbc0 0x7fa098150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57652/0 (socket says 172.21.15.31:57652) 2023-10-30T13:36:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 -- 172.21.15.31:0/3095181925 learned_addr learned my addr 172.21.15.31:0/3095181925 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09d285640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0980c5580 0x7fa098070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09da86640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa0840047d0 0x7fa0980c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 -- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa0840047d0 msgr2=0x7fa0980c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa0840047d0 0x7fa0980c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 -- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0980c5580 msgr2=0x7fa098070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0980c5580 0x7fa098070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 -- 172.21.15.31:0/3095181925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa098153e90 con 0x7fa09814fbc0 2023-10-30T13:36:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09e287640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09814fbc0 0x7fa098150070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa098155140 tx=0x7fa090009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa08e7fc640 1 -- 172.21.15.31:0/3095181925 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09000e070 con 0x7fa09814fbc0 2023-10-30T13:36:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa08e7fc640 1 -- 172.21.15.31:0/3095181925 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa090011e60 con 0x7fa09814fbc0 2023-10-30T13:36:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0980c4d80 con 0x7fa09814fbc0 2023-10-30T13:36:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa08e7fc640 1 -- 172.21.15.31:0/3095181925 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900039a0 con 0x7fa09814fbc0 2023-10-30T13:36:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0981573d0 con 0x7fa09814fbc0 2023-10-30T13:36:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09804e170 con 0x7fa09814fbc0 2023-10-30T13:36:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.387+0000 7fa08e7fc640 1 -- 172.21.15.31:0/3095181925 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa090003b00 con 0x7fa09814fbc0 2023-10-30T13:36:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.391+0000 7fa08e7fc640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa07c0664a0 0x7fa07c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.391+0000 7fa08e7fc640 1 -- 172.21.15.31:0/3095181925 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fa0900847b0 con 0x7fa09814fbc0 2023-10-30T13:36:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.391+0000 7fa09da86640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa07c0664a0 0x7fa07c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.391+0000 7fa09da86640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa07c0664a0 0x7fa07c068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa094007fd0 tx=0x7fa094009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.395+0000 7fa08e7fc640 1 -- 172.21.15.31:0/3095181925 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa090051b30 con 0x7fa09814fbc0 2023-10-30T13:36:56.407 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 106 2023-10-30T13:36:56.431 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 107 2023-10-30T13:36:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.583+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0980c7b70 con 0x7fa09814fbc0 2023-10-30T13:36:56.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa08e7fc640 1 -- 172.21.15.31:0/3095181925 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5729 (secure 0 0 0) 0x7fa090055150 con 0x7fa09814fbc0 2023-10-30T13:36:56.590 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:56.591 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:48.802383+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 18 2023-10-30T13:36:56.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa07c0664a0 msgr2=0x7fa07c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa07c0664a0 0x7fa07c068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa094007fd0 tx=0x7fa094009040 comp rx=0 tx=0).stop 2023-10-30T13:36:56.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09814fbc0 msgr2=0x7fa098150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09814fbc0 0x7fa098150070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa098155140 tx=0x7fa090009f90 comp rx=0 tx=0).stop 2023-10-30T13:36:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 shutdown_connections 2023-10-30T13:36:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa07c0664a0 0x7fa07c068960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09814fbc0 0x7fa098150070 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0980c5580 0x7fa098070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 --2- 172.21.15.31:0/3095181925 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa0840047d0 0x7fa0980c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 >> 172.21.15.31:0/3095181925 conn(0x7fa098071fb0 msgr2=0x7fa0980cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 shutdown_connections 2023-10-30T13:36:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:56.587+0000 7fa09fcea640 1 -- 172.21.15.31:0/3095181925 wait complete. 2023-10-30T13:36:56.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:36:56.943 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 108 2023-10-30T13:36:56.966 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 109 2023-10-30T13:36:56.987 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 110 2023-10-30T13:36:57.008 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 111 2023-10-30T13:36:57.029 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 112 2023-10-30T13:36:57.048 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 113 2023-10-30T13:36:57.067 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 114 2023-10-30T13:36:57.084 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 115 2023-10-30T13:36:57.100 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 116 2023-10-30T13:36:57.116 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 117 2023-10-30T13:36:57.132 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 118 2023-10-30T13:36:57.147 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 119 2023-10-30T13:36:57.191 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 120 2023-10-30T13:36:57.208 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 121 2023-10-30T13:36:57.227 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 122 2023-10-30T13:36:57.245 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 123 2023-10-30T13:36:57.511 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 124 2023-10-30T13:36:57.533 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 125 2023-10-30T13:36:57.556 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 126 2023-10-30T13:36:57.823 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 127 2023-10-30T13:36:57.846 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 128 2023-10-30T13:36:57.869 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 129 2023-10-30T13:36:57.892 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 130 2023-10-30T13:36:57.915 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 131 2023-10-30T13:36:57.934 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 132 2023-10-30T13:36:57.953 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 133 2023-10-30T13:36:57.970 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 134 2023-10-30T13:36:57.988 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 135 2023-10-30T13:36:58.003 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 136 2023-10-30T13:36:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 137 2023-10-30T13:36:58.035 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 138 2023-10-30T13:36:58.049 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 139 2023-10-30T13:36:58.064 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 140 2023-10-30T13:36:58.078 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 141 2023-10-30T13:36:58.092 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 142 2023-10-30T13:36:58.106 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 143 2023-10-30T13:36:58.119 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 144 2023-10-30T13:36:58.132 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 145 2023-10-30T13:36:58.145 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 146 2023-10-30T13:36:58.157 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 147 2023-10-30T13:36:58.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:58.720 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 148 2023-10-30T13:36:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2aa52ea640 1 Processor -- start 2023-10-30T13:36:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2aa52ea640 1 -- start start 2023-10-30T13:36:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2aa52ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2aa0153a10 0x7f2aa00c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2aa52ea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa015d0a0 con 0x7f2aa00718d0 2023-10-30T13:36:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2aa52ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa015d380 con 0x7f2aa0071cb0 2023-10-30T13:36:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2aa52ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00cca10 con 0x7f2aa0153a10 2023-10-30T13:36:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2a9effd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2aa00718d0 0x7f2aa015a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:50650/0 (socket says 172.21.15.31:50650) 2023-10-30T13:36:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2a9effd640 1 -- 172.21.15.31:0/1517707810 learned_addr learned my addr 172.21.15.31:0/1517707810 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:36:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.763+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/1517707810 >> v2:172.21.15.33:3300/0 conn(0x7f2aa0153a10 0x7f2aa00c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097901810 0 0) 0x7f2aa015d0a0 con 0x7f2aa00718d0 2023-10-30T13:36:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8c003530 con 0x7f2aa00718d0 2023-10-30T13:36:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3982013152 0 0) 0x7f2a8c003530 con 0x7f2aa00718d0 2023-10-30T13:36:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 >> v2:172.21.15.33:3300/0 conn(0x7f2aa0153a10 msgr2=0x7f2aa00c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 --2- 172.21.15.31:0/1517707810 >> v2:172.21.15.33:3300/0 conn(0x7f2aa0153a10 0x7f2aa00c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 >> v1:172.21.15.33:6789/0 conn(0x7f2aa0071cb0 legacy=0x7f2aa0153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa015d660 con 0x7f2aa00718d0 2023-10-30T13:36:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f2a90001680 con 0x7f2aa00718d0 2023-10-30T13:36:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 >> v1:172.21.15.31:6789/0 conn(0x7f2aa00718d0 legacy=0x7f2aa015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c003980 0x7f2a8c003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a8c004380 0x7f2a8c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a8c005050 0x7f2a8c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00cca10 con 0x7f2a8c005050 2023-10-30T13:36:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c00bf00 con 0x7f2a8c004380 2023-10-30T13:36:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c00c0c0 con 0x7f2a8c003980 2023-10-30T13:36:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a8c005050 0x7f2a8c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9effd640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c003980 0x7f2a8c003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a97fff640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a8c004380 0x7f2a8c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c003980 msgr2=0x7f2a8c003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c003980 0x7f2a8c003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a8c004380 msgr2=0x7f2a8c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a8c004380 0x7f2a8c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/1517707810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c003530 con 0x7f2a8c005050 2023-10-30T13:36:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.767+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/1517707810 reap_dead start 2023-10-30T13:36:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a8c005050 0x7f2a8c00b990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00c9f50 tx=0x7f2a9801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e592f7d6915538f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a980a5c50 con 0x7f2a8c005050 2023-10-30T13:36:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9800f040 con 0x7f2a8c005050 2023-10-30T13:36:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2a9e7fc640 1 -- 172.21.15.31:0/1517707810 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a980ad440 con 0x7f2a8c005050 2023-10-30T13:36:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- 172.21.15.31:0/1517707810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a8c005050 msgr2=0x7f2a8c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a8c005050 0x7f2a8c00b990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00c9f50 tx=0x7f2a9801cc40 comp rx=0 tx=0).stop 2023-10-30T13:36:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- 172.21.15.31:0/1517707810 shutdown_connections 2023-10-30T13:36:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/1517707810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a8c005050 0x7f2a8c00b990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- 172.21.15.31:0/1517707810 >> 172.21.15.31:0/1517707810 conn(0x7f2aa006ab00 msgr2=0x7f2aa0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- 172.21.15.31:0/1517707810 shutdown_connections 2023-10-30T13:36:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- 172.21.15.31:0/1517707810 wait complete. 2023-10-30T13:36:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 Processor -- start 2023-10-30T13:36:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- start start 2023-10-30T13:36:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c005050 0x7f2aa015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aa014d0f0 0x7f2aa0158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aa00cfa90 0x7f2aa01f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0159930 con 0x7f2aa00cfa90 2023-10-30T13:36:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00cd940 con 0x7f2aa014d0f0 2023-10-30T13:36:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.771+0000 7f2aa52ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa01581c0 con 0x7f2a8c005050 2023-10-30T13:36:58.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aa00cfa90 0x7f2aa01f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aa00cfa90 0x7f2aa01f1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57666/0 (socket says 172.21.15.31:57666) 2023-10-30T13:36:58.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/725137120 learned_addr learned my addr 172.21.15.31:0/725137120 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:36:58.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9effd640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c005050 0x7f2aa015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a97fff640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aa014d0f0 0x7f2aa0158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c005050 msgr2=0x7f2aa015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c005050 0x7f2aa015c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aa014d0f0 msgr2=0x7f2aa0158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aa014d0f0 0x7f2aa0158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 -- 172.21.15.31:0/725137120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0157d30 con 0x7f2aa00cfa90 2023-10-30T13:36:58.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a9f7fe640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aa00cfa90 0x7f2aa01f1840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a980011f0 tx=0x7f2a98009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a977fe640 1 -- 172.21.15.31:0/725137120 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a980adcc0 con 0x7f2aa00cfa90 2023-10-30T13:36:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a977fe640 1 -- 172.21.15.31:0/725137120 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a980ade20 con 0x7f2aa00cfa90 2023-10-30T13:36:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a977fe640 1 -- 172.21.15.31:0/725137120 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a980b6610 con 0x7f2aa00cfa90 2023-10-30T13:36:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa01eee60 con 0x7f2aa00cfa90 2023-10-30T13:36:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa0157400 con 0x7f2aa00cfa90 2023-10-30T13:36:58.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa004e170 con 0x7f2aa00cfa90 2023-10-30T13:36:58.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a977fe640 1 -- 172.21.15.31:0/725137120 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2a980b4040 con 0x7f2aa00cfa90 2023-10-30T13:36:58.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.775+0000 7f2a977fe640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2a800664d0 0x7f2a80068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.779+0000 7f2a977fe640 1 -- 172.21.15.31:0/725137120 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f2a9811e150 con 0x7f2aa00cfa90 2023-10-30T13:36:58.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.779+0000 7f2a9effd640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2a800664d0 0x7f2a80068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.779+0000 7f2a9effd640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2a800664d0 0x7f2a80068990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2a900059a0 tx=0x7f2a90007040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:58.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.783+0000 7f2a977fe640 1 -- 172.21.15.31:0/725137120 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2a980eb500 con 0x7f2aa00cfa90 2023-10-30T13:36:58.816 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 149 2023-10-30T13:36:58.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.971+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa00c9f80 con 0x7f2aa00cfa90 2023-10-30T13:36:58.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.971+0000 7f2a977fe640 1 -- 172.21.15.31:0/725137120 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6520 (secure 0 0 0) 0x7f2a980bc020 con 0x7f2aa00cfa90 2023-10-30T13:36:58.978 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:36:58.978 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:36:48.802383+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:58.978 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 19 2023-10-30T13:36:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2a800664d0 msgr2=0x7f2a80068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2a800664d0 0x7f2a80068990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2a900059a0 tx=0x7f2a90007040 comp rx=0 tx=0).stop 2023-10-30T13:36:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aa00cfa90 msgr2=0x7f2aa01f1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aa00cfa90 0x7f2aa01f1840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a980011f0 tx=0x7f2a98009750 comp rx=0 tx=0).stop 2023-10-30T13:36:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 shutdown_connections 2023-10-30T13:36:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2a800664d0 0x7f2a80068990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aa00cfa90 0x7f2aa01f1840 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aa014d0f0 0x7f2aa0158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 --2- 172.21.15.31:0/725137120 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2a8c005050 0x7f2aa015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 >> 172.21.15.31:0/725137120 conn(0x7f2aa006ab00 msgr2=0x7f2aa00c4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:58.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 shutdown_connections 2023-10-30T13:36:58.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:36:58.975+0000 7f2aa52ea640 1 -- 172.21.15.31:0/725137120 wait complete. 2023-10-30T13:36:58.999 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:36:58.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2023-10-30T13:36:59.046 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 150 2023-10-30T13:36:59.069 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 151 2023-10-30T13:36:59.092 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 152 2023-10-30T13:36:59.115 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 153 2023-10-30T13:36:59.579 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 154 2023-10-30T13:36:59.611 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 155 2023-10-30T13:36:59.646 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 156 2023-10-30T13:37:00.438 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 157 2023-10-30T13:37:00.460 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 158 2023-10-30T13:37:00.482 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 159 2023-10-30T13:37:00.506 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 160 2023-10-30T13:37:00.540 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 161 2023-10-30T13:37:00.572 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 162 2023-10-30T13:37:00.600 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 163 2023-10-30T13:37:00.625 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 164 2023-10-30T13:37:00.649 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 165 2023-10-30T13:37:00.918 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 166 2023-10-30T13:37:00.956 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 167 2023-10-30T13:37:00.981 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 168 2023-10-30T13:37:01.004 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 169 2023-10-30T13:37:01.024 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 170 2023-10-30T13:37:01.044 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 171 2023-10-30T13:37:01.062 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 172 2023-10-30T13:37:01.080 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 173 2023-10-30T13:37:01.097 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 174 2023-10-30T13:37:01.113 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 175 2023-10-30T13:37:01.128 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 176 2023-10-30T13:37:01.142 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 177 2023-10-30T13:37:01.156 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 178 2023-10-30T13:37:01.169 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 179 2023-10-30T13:37:01.182 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 180 2023-10-30T13:37:01.196 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 181 2023-10-30T13:37:01.210 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 182 2023-10-30T13:37:01.223 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 183 2023-10-30T13:37:01.235 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 184 2023-10-30T13:37:01.247 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 185 2023-10-30T13:37:01.260 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 186 2023-10-30T13:37:01.272 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 187 2023-10-30T13:37:01.283 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 188 2023-10-30T13:37:01.545 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 189 2023-10-30T13:37:01.586 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 190 2023-10-30T13:37:01.624 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 191 2023-10-30T13:37:01.657 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 192 2023-10-30T13:37:01.678 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 193 2023-10-30T13:37:02.208 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 194 2023-10-30T13:37:02.231 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 195 2023-10-30T13:37:02.707 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 196 2023-10-30T13:37:02.731 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 197 2023-10-30T13:37:02.754 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 198 2023-10-30T13:37:02.777 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 199 2023-10-30T13:37:02.799 INFO:tasks.workunit.client.0.smithi031.stdout:Take first snapshot .snap/test1 2023-10-30T13:37:03.005 INFO:tasks.workunit.client.0.smithi031.stdout:Create dir 200 to 299 ... 2023-10-30T13:37:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 200 2023-10-30T13:37:03.017 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 201 2023-10-30T13:37:03.039 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 202 2023-10-30T13:37:03.060 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 203 2023-10-30T13:37:03.082 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 204 2023-10-30T13:37:03.103 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 205 2023-10-30T13:37:03.122 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 206 2023-10-30T13:37:03.140 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 207 2023-10-30T13:37:03.157 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 208 2023-10-30T13:37:03.173 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 209 2023-10-30T13:37:03.189 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 210 2023-10-30T13:37:03.204 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 211 2023-10-30T13:37:03.218 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 212 2023-10-30T13:37:03.232 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 213 2023-10-30T13:37:03.245 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 214 2023-10-30T13:37:03.507 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 215 2023-10-30T13:37:03.527 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 216 2023-10-30T13:37:03.790 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 217 2023-10-30T13:37:03.811 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 218 2023-10-30T13:37:03.832 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 219 2023-10-30T13:37:03.937 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 220 2023-10-30T13:37:04.705 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 221 2023-10-30T13:37:04.830 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 222 2023-10-30T13:37:04.851 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 223 2023-10-30T13:37:04.879 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 224 2023-10-30T13:37:04.901 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 225 2023-10-30T13:37:04.922 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 226 2023-10-30T13:37:04.943 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 227 2023-10-30T13:37:04.961 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 228 2023-10-30T13:37:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 229 2023-10-30T13:37:04.998 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 230 2023-10-30T13:37:05.014 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 231 2023-10-30T13:37:05.030 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 232 2023-10-30T13:37:05.045 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 233 2023-10-30T13:37:05.728 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 234 2023-10-30T13:37:05.967 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 235 2023-10-30T13:37:06.087 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 236 2023-10-30T13:37:06.110 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 237 2023-10-30T13:37:06.560 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 238 2023-10-30T13:37:06.583 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 239 2023-10-30T13:37:06.605 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 240 2023-10-30T13:37:06.627 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 241 2023-10-30T13:37:06.648 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 242 2023-10-30T13:37:06.669 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 243 2023-10-30T13:37:06.688 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 244 2023-10-30T13:37:06.706 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 245 2023-10-30T13:37:06.722 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 246 2023-10-30T13:37:06.739 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 247 2023-10-30T13:37:06.755 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 248 2023-10-30T13:37:06.773 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 249 2023-10-30T13:37:06.792 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 250 2023-10-30T13:37:09.365 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 251 2023-10-30T13:37:09.439 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 252 2023-10-30T13:37:09.461 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 253 2023-10-30T13:37:09.483 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 254 2023-10-30T13:37:09.502 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 255 2023-10-30T13:37:09.522 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 256 2023-10-30T13:37:09.541 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 257 2023-10-30T13:37:09.560 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 258 2023-10-30T13:37:09.576 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 259 2023-10-30T13:37:09.591 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 260 2023-10-30T13:37:09.606 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 261 2023-10-30T13:37:09.621 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 262 2023-10-30T13:37:09.634 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 263 2023-10-30T13:37:09.649 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 264 2023-10-30T13:37:09.663 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 265 2023-10-30T13:37:09.924 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 266 2023-10-30T13:37:10.186 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 267 2023-10-30T13:37:10.235 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 268 2023-10-30T13:37:10.255 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 269 2023-10-30T13:37:10.278 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 270 2023-10-30T13:37:10.299 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 271 2023-10-30T13:37:10.318 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 272 2023-10-30T13:37:10.336 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 273 2023-10-30T13:37:10.354 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 274 2023-10-30T13:37:10.370 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 275 2023-10-30T13:37:10.385 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 276 2023-10-30T13:37:10.400 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 277 2023-10-30T13:37:10.416 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 278 2023-10-30T13:37:10.430 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 279 2023-10-30T13:37:10.444 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 280 2023-10-30T13:37:10.458 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 281 2023-10-30T13:37:10.472 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 282 2023-10-30T13:37:10.485 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 283 2023-10-30T13:37:10.498 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 284 2023-10-30T13:37:10.511 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 285 2023-10-30T13:37:10.522 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 286 2023-10-30T13:37:10.534 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 287 2023-10-30T13:37:10.546 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 288 2023-10-30T13:37:10.558 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 289 2023-10-30T13:37:10.569 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 290 2023-10-30T13:37:10.720 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 291 2023-10-30T13:37:11.117 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 292 2023-10-30T13:37:11.139 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 293 2023-10-30T13:37:11.409 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 294 2023-10-30T13:37:11.436 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 295 2023-10-30T13:37:11.468 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 296 2023-10-30T13:37:11.501 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 297 2023-10-30T13:37:11.534 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 298 2023-10-30T13:37:11.571 INFO:tasks.workunit.client.0.smithi031.stdout: create dir 299 2023-10-30T13:37:11.612 INFO:tasks.workunit.client.0.smithi031.stdout:Create a snapshot in every first level dir ... 2023-10-30T13:37:11.927 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/.snap/snap-subdir-test 2023-10-30T13:37:11.953 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/30 file after the snapshot 2023-10-30T13:37:11.956 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/31 file after the snapshot 2023-10-30T13:37:11.956 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/32 file after the snapshot 2023-10-30T13:37:11.956 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/33 file after the snapshot 2023-10-30T13:37:11.956 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/34 file after the snapshot 2023-10-30T13:37:11.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/35 file after the snapshot 2023-10-30T13:37:11.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/36 file after the snapshot 2023-10-30T13:37:11.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/37 file after the snapshot 2023-10-30T13:37:11.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/38 file after the snapshot 2023-10-30T13:37:11.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 100/39 file after the snapshot 2023-10-30T13:37:11.958 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/.snap/snap-subdir-test 2023-10-30T13:37:11.998 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/30 file after the snapshot 2023-10-30T13:37:12.000 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/31 file after the snapshot 2023-10-30T13:37:12.001 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/32 file after the snapshot 2023-10-30T13:37:12.001 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/33 file after the snapshot 2023-10-30T13:37:12.001 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/34 file after the snapshot 2023-10-30T13:37:12.001 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/35 file after the snapshot 2023-10-30T13:37:12.002 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/36 file after the snapshot 2023-10-30T13:37:12.002 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/37 file after the snapshot 2023-10-30T13:37:12.002 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/38 file after the snapshot 2023-10-30T13:37:12.002 INFO:tasks.workunit.client.0.smithi031.stdout: create 101/39 file after the snapshot 2023-10-30T13:37:12.002 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/.snap/snap-subdir-test 2023-10-30T13:37:12.056 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/30 file after the snapshot 2023-10-30T13:37:12.059 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/31 file after the snapshot 2023-10-30T13:37:12.059 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/32 file after the snapshot 2023-10-30T13:37:12.059 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/33 file after the snapshot 2023-10-30T13:37:12.059 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/34 file after the snapshot 2023-10-30T13:37:12.060 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/35 file after the snapshot 2023-10-30T13:37:12.060 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/36 file after the snapshot 2023-10-30T13:37:12.060 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/37 file after the snapshot 2023-10-30T13:37:12.060 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/38 file after the snapshot 2023-10-30T13:37:12.060 INFO:tasks.workunit.client.0.smithi031.stdout: create 102/39 file after the snapshot 2023-10-30T13:37:12.061 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/.snap/snap-subdir-test 2023-10-30T13:37:12.110 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/30 file after the snapshot 2023-10-30T13:37:12.392 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/31 file after the snapshot 2023-10-30T13:37:12.393 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/32 file after the snapshot 2023-10-30T13:37:12.393 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/33 file after the snapshot 2023-10-30T13:37:12.394 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/34 file after the snapshot 2023-10-30T13:37:12.394 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/35 file after the snapshot 2023-10-30T13:37:12.394 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/36 file after the snapshot 2023-10-30T13:37:12.395 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/37 file after the snapshot 2023-10-30T13:37:12.395 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/38 file after the snapshot 2023-10-30T13:37:12.396 INFO:tasks.workunit.client.0.smithi031.stdout: create 103/39 file after the snapshot 2023-10-30T13:37:12.396 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/.snap/snap-subdir-test 2023-10-30T13:37:12.464 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/30 file after the snapshot 2023-10-30T13:37:12.468 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/31 file after the snapshot 2023-10-30T13:37:12.468 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/32 file after the snapshot 2023-10-30T13:37:12.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/33 file after the snapshot 2023-10-30T13:37:12.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/34 file after the snapshot 2023-10-30T13:37:12.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/35 file after the snapshot 2023-10-30T13:37:12.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/36 file after the snapshot 2023-10-30T13:37:12.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/37 file after the snapshot 2023-10-30T13:37:12.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/38 file after the snapshot 2023-10-30T13:37:12.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 104/39 file after the snapshot 2023-10-30T13:37:12.471 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/.snap/snap-subdir-test 2023-10-30T13:37:12.544 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/30 file after the snapshot 2023-10-30T13:37:12.554 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/31 file after the snapshot 2023-10-30T13:37:12.554 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/32 file after the snapshot 2023-10-30T13:37:12.554 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/33 file after the snapshot 2023-10-30T13:37:12.554 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/34 file after the snapshot 2023-10-30T13:37:12.554 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/35 file after the snapshot 2023-10-30T13:37:12.555 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/36 file after the snapshot 2023-10-30T13:37:12.555 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/37 file after the snapshot 2023-10-30T13:37:12.555 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/38 file after the snapshot 2023-10-30T13:37:12.555 INFO:tasks.workunit.client.0.smithi031.stdout: create 105/39 file after the snapshot 2023-10-30T13:37:12.555 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/.snap/snap-subdir-test 2023-10-30T13:37:13.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/30 file after the snapshot 2023-10-30T13:37:13.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/31 file after the snapshot 2023-10-30T13:37:13.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/32 file after the snapshot 2023-10-30T13:37:13.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/33 file after the snapshot 2023-10-30T13:37:13.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/34 file after the snapshot 2023-10-30T13:37:13.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/35 file after the snapshot 2023-10-30T13:37:13.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/36 file after the snapshot 2023-10-30T13:37:13.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/37 file after the snapshot 2023-10-30T13:37:13.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/38 file after the snapshot 2023-10-30T13:37:13.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 106/39 file after the snapshot 2023-10-30T13:37:13.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/.snap/snap-subdir-test 2023-10-30T13:37:13.644 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/30 file after the snapshot 2023-10-30T13:37:13.648 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/31 file after the snapshot 2023-10-30T13:37:13.648 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/32 file after the snapshot 2023-10-30T13:37:13.648 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/33 file after the snapshot 2023-10-30T13:37:13.648 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/34 file after the snapshot 2023-10-30T13:37:13.649 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/35 file after the snapshot 2023-10-30T13:37:13.649 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/36 file after the snapshot 2023-10-30T13:37:13.649 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/37 file after the snapshot 2023-10-30T13:37:13.649 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/38 file after the snapshot 2023-10-30T13:37:13.649 INFO:tasks.workunit.client.0.smithi031.stdout: create 107/39 file after the snapshot 2023-10-30T13:37:13.649 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/.snap/snap-subdir-test 2023-10-30T13:37:13.720 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/30 file after the snapshot 2023-10-30T13:37:13.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/31 file after the snapshot 2023-10-30T13:37:13.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/32 file after the snapshot 2023-10-30T13:37:13.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/33 file after the snapshot 2023-10-30T13:37:13.724 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/34 file after the snapshot 2023-10-30T13:37:13.724 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/35 file after the snapshot 2023-10-30T13:37:13.724 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/36 file after the snapshot 2023-10-30T13:37:13.724 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/37 file after the snapshot 2023-10-30T13:37:13.724 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/38 file after the snapshot 2023-10-30T13:37:13.725 INFO:tasks.workunit.client.0.smithi031.stdout: create 108/39 file after the snapshot 2023-10-30T13:37:13.725 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/.snap/snap-subdir-test 2023-10-30T13:37:14.037 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/30 file after the snapshot 2023-10-30T13:37:14.042 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/31 file after the snapshot 2023-10-30T13:37:14.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/32 file after the snapshot 2023-10-30T13:37:14.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/33 file after the snapshot 2023-10-30T13:37:14.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/34 file after the snapshot 2023-10-30T13:37:14.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/35 file after the snapshot 2023-10-30T13:37:14.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/36 file after the snapshot 2023-10-30T13:37:14.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/37 file after the snapshot 2023-10-30T13:37:14.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/38 file after the snapshot 2023-10-30T13:37:14.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 109/39 file after the snapshot 2023-10-30T13:37:14.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/.snap/snap-subdir-test 2023-10-30T13:37:14.114 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/30 file after the snapshot 2023-10-30T13:37:14.119 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/31 file after the snapshot 2023-10-30T13:37:14.119 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/32 file after the snapshot 2023-10-30T13:37:14.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/33 file after the snapshot 2023-10-30T13:37:14.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/34 file after the snapshot 2023-10-30T13:37:14.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/35 file after the snapshot 2023-10-30T13:37:14.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/36 file after the snapshot 2023-10-30T13:37:14.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/37 file after the snapshot 2023-10-30T13:37:14.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/38 file after the snapshot 2023-10-30T13:37:14.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 110/39 file after the snapshot 2023-10-30T13:37:14.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/.snap/snap-subdir-test 2023-10-30T13:37:14.201 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/30 file after the snapshot 2023-10-30T13:37:14.204 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/31 file after the snapshot 2023-10-30T13:37:14.205 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/32 file after the snapshot 2023-10-30T13:37:14.205 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/33 file after the snapshot 2023-10-30T13:37:14.205 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/34 file after the snapshot 2023-10-30T13:37:14.205 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/35 file after the snapshot 2023-10-30T13:37:14.205 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/36 file after the snapshot 2023-10-30T13:37:14.206 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/37 file after the snapshot 2023-10-30T13:37:14.206 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/38 file after the snapshot 2023-10-30T13:37:14.206 INFO:tasks.workunit.client.0.smithi031.stdout: create 111/39 file after the snapshot 2023-10-30T13:37:14.206 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/.snap/snap-subdir-test 2023-10-30T13:37:14.277 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/30 file after the snapshot 2023-10-30T13:37:14.281 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/31 file after the snapshot 2023-10-30T13:37:14.281 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/32 file after the snapshot 2023-10-30T13:37:14.282 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/33 file after the snapshot 2023-10-30T13:37:14.282 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/34 file after the snapshot 2023-10-30T13:37:14.282 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/35 file after the snapshot 2023-10-30T13:37:14.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/36 file after the snapshot 2023-10-30T13:37:14.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/37 file after the snapshot 2023-10-30T13:37:14.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/38 file after the snapshot 2023-10-30T13:37:14.284 INFO:tasks.workunit.client.0.smithi031.stdout: create 112/39 file after the snapshot 2023-10-30T13:37:14.284 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/.snap/snap-subdir-test 2023-10-30T13:37:15.003 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/30 file after the snapshot 2023-10-30T13:37:15.005 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/31 file after the snapshot 2023-10-30T13:37:15.006 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/32 file after the snapshot 2023-10-30T13:37:15.006 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/33 file after the snapshot 2023-10-30T13:37:15.006 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/34 file after the snapshot 2023-10-30T13:37:15.006 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/35 file after the snapshot 2023-10-30T13:37:15.007 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/36 file after the snapshot 2023-10-30T13:37:15.007 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/37 file after the snapshot 2023-10-30T13:37:15.007 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/38 file after the snapshot 2023-10-30T13:37:15.007 INFO:tasks.workunit.client.0.smithi031.stdout: create 113/39 file after the snapshot 2023-10-30T13:37:15.008 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/.snap/snap-subdir-test 2023-10-30T13:37:15.100 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/30 file after the snapshot 2023-10-30T13:37:15.105 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/31 file after the snapshot 2023-10-30T13:37:15.106 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/32 file after the snapshot 2023-10-30T13:37:15.106 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/33 file after the snapshot 2023-10-30T13:37:15.106 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/34 file after the snapshot 2023-10-30T13:37:15.106 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/35 file after the snapshot 2023-10-30T13:37:15.107 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/36 file after the snapshot 2023-10-30T13:37:15.107 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/37 file after the snapshot 2023-10-30T13:37:15.107 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/38 file after the snapshot 2023-10-30T13:37:15.107 INFO:tasks.workunit.client.0.smithi031.stdout: create 114/39 file after the snapshot 2023-10-30T13:37:15.107 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/.snap/snap-subdir-test 2023-10-30T13:37:15.172 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/30 file after the snapshot 2023-10-30T13:37:15.175 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/31 file after the snapshot 2023-10-30T13:37:15.175 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/32 file after the snapshot 2023-10-30T13:37:15.176 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/33 file after the snapshot 2023-10-30T13:37:15.176 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/34 file after the snapshot 2023-10-30T13:37:15.176 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/35 file after the snapshot 2023-10-30T13:37:15.176 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/36 file after the snapshot 2023-10-30T13:37:15.176 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/37 file after the snapshot 2023-10-30T13:37:15.177 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/38 file after the snapshot 2023-10-30T13:37:15.177 INFO:tasks.workunit.client.0.smithi031.stdout: create 115/39 file after the snapshot 2023-10-30T13:37:15.177 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/.snap/snap-subdir-test 2023-10-30T13:37:15.252 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/30 file after the snapshot 2023-10-30T13:37:15.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/31 file after the snapshot 2023-10-30T13:37:15.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/32 file after the snapshot 2023-10-30T13:37:15.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/33 file after the snapshot 2023-10-30T13:37:15.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/34 file after the snapshot 2023-10-30T13:37:15.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/35 file after the snapshot 2023-10-30T13:37:15.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/36 file after the snapshot 2023-10-30T13:37:15.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/37 file after the snapshot 2023-10-30T13:37:15.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/38 file after the snapshot 2023-10-30T13:37:15.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 116/39 file after the snapshot 2023-10-30T13:37:15.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/.snap/snap-subdir-test 2023-10-30T13:37:15.766 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/30 file after the snapshot 2023-10-30T13:37:15.771 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/31 file after the snapshot 2023-10-30T13:37:15.771 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/32 file after the snapshot 2023-10-30T13:37:15.771 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/33 file after the snapshot 2023-10-30T13:37:15.772 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/34 file after the snapshot 2023-10-30T13:37:15.772 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/35 file after the snapshot 2023-10-30T13:37:15.772 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/36 file after the snapshot 2023-10-30T13:37:15.772 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/37 file after the snapshot 2023-10-30T13:37:15.772 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/38 file after the snapshot 2023-10-30T13:37:15.773 INFO:tasks.workunit.client.0.smithi031.stdout: create 117/39 file after the snapshot 2023-10-30T13:37:15.773 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/.snap/snap-subdir-test 2023-10-30T13:37:15.846 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/30 file after the snapshot 2023-10-30T13:37:15.849 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/31 file after the snapshot 2023-10-30T13:37:15.850 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/32 file after the snapshot 2023-10-30T13:37:15.850 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/33 file after the snapshot 2023-10-30T13:37:15.850 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/34 file after the snapshot 2023-10-30T13:37:15.850 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/35 file after the snapshot 2023-10-30T13:37:15.850 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/36 file after the snapshot 2023-10-30T13:37:15.850 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/37 file after the snapshot 2023-10-30T13:37:15.851 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/38 file after the snapshot 2023-10-30T13:37:15.851 INFO:tasks.workunit.client.0.smithi031.stdout: create 118/39 file after the snapshot 2023-10-30T13:37:15.851 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/.snap/snap-subdir-test 2023-10-30T13:37:15.920 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/30 file after the snapshot 2023-10-30T13:37:15.925 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/31 file after the snapshot 2023-10-30T13:37:15.925 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/32 file after the snapshot 2023-10-30T13:37:15.925 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/33 file after the snapshot 2023-10-30T13:37:15.925 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/34 file after the snapshot 2023-10-30T13:37:15.925 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/35 file after the snapshot 2023-10-30T13:37:15.926 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/36 file after the snapshot 2023-10-30T13:37:15.926 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/37 file after the snapshot 2023-10-30T13:37:15.926 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/38 file after the snapshot 2023-10-30T13:37:15.926 INFO:tasks.workunit.client.0.smithi031.stdout: create 119/39 file after the snapshot 2023-10-30T13:37:15.927 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/.snap/snap-subdir-test 2023-10-30T13:37:16.014 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/30 file after the snapshot 2023-10-30T13:37:16.018 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/31 file after the snapshot 2023-10-30T13:37:16.018 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/32 file after the snapshot 2023-10-30T13:37:16.018 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/33 file after the snapshot 2023-10-30T13:37:16.018 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/34 file after the snapshot 2023-10-30T13:37:16.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/35 file after the snapshot 2023-10-30T13:37:16.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/36 file after the snapshot 2023-10-30T13:37:16.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/37 file after the snapshot 2023-10-30T13:37:16.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/38 file after the snapshot 2023-10-30T13:37:16.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 120/39 file after the snapshot 2023-10-30T13:37:16.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/.snap/snap-subdir-test 2023-10-30T13:37:16.090 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/30 file after the snapshot 2023-10-30T13:37:16.099 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/31 file after the snapshot 2023-10-30T13:37:16.099 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/32 file after the snapshot 2023-10-30T13:37:16.099 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/33 file after the snapshot 2023-10-30T13:37:16.099 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/34 file after the snapshot 2023-10-30T13:37:16.099 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/35 file after the snapshot 2023-10-30T13:37:16.100 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/36 file after the snapshot 2023-10-30T13:37:16.100 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/37 file after the snapshot 2023-10-30T13:37:16.100 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/38 file after the snapshot 2023-10-30T13:37:16.100 INFO:tasks.workunit.client.0.smithi031.stdout: create 121/39 file after the snapshot 2023-10-30T13:37:16.100 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/.snap/snap-subdir-test 2023-10-30T13:37:16.186 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/30 file after the snapshot 2023-10-30T13:37:16.190 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/31 file after the snapshot 2023-10-30T13:37:16.190 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/32 file after the snapshot 2023-10-30T13:37:16.190 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/33 file after the snapshot 2023-10-30T13:37:16.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/34 file after the snapshot 2023-10-30T13:37:16.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/35 file after the snapshot 2023-10-30T13:37:16.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/36 file after the snapshot 2023-10-30T13:37:16.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/37 file after the snapshot 2023-10-30T13:37:16.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/38 file after the snapshot 2023-10-30T13:37:16.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 122/39 file after the snapshot 2023-10-30T13:37:16.192 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/.snap/snap-subdir-test 2023-10-30T13:37:16.279 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/30 file after the snapshot 2023-10-30T13:37:16.282 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/31 file after the snapshot 2023-10-30T13:37:16.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/32 file after the snapshot 2023-10-30T13:37:16.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/33 file after the snapshot 2023-10-30T13:37:16.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/34 file after the snapshot 2023-10-30T13:37:16.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/35 file after the snapshot 2023-10-30T13:37:16.283 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/36 file after the snapshot 2023-10-30T13:37:16.284 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/37 file after the snapshot 2023-10-30T13:37:16.284 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/38 file after the snapshot 2023-10-30T13:37:16.284 INFO:tasks.workunit.client.0.smithi031.stdout: create 123/39 file after the snapshot 2023-10-30T13:37:16.284 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/.snap/snap-subdir-test 2023-10-30T13:37:16.350 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/30 file after the snapshot 2023-10-30T13:37:16.353 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/31 file after the snapshot 2023-10-30T13:37:16.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/32 file after the snapshot 2023-10-30T13:37:16.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/33 file after the snapshot 2023-10-30T13:37:16.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/34 file after the snapshot 2023-10-30T13:37:16.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/35 file after the snapshot 2023-10-30T13:37:16.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/36 file after the snapshot 2023-10-30T13:37:16.355 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/37 file after the snapshot 2023-10-30T13:37:16.355 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/38 file after the snapshot 2023-10-30T13:37:16.355 INFO:tasks.workunit.client.0.smithi031.stdout: create 124/39 file after the snapshot 2023-10-30T13:37:16.355 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/.snap/snap-subdir-test 2023-10-30T13:37:16.422 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/30 file after the snapshot 2023-10-30T13:37:16.425 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/31 file after the snapshot 2023-10-30T13:37:16.425 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/32 file after the snapshot 2023-10-30T13:37:16.426 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/33 file after the snapshot 2023-10-30T13:37:16.426 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/34 file after the snapshot 2023-10-30T13:37:16.426 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/35 file after the snapshot 2023-10-30T13:37:16.426 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/36 file after the snapshot 2023-10-30T13:37:16.427 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/37 file after the snapshot 2023-10-30T13:37:16.427 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/38 file after the snapshot 2023-10-30T13:37:16.427 INFO:tasks.workunit.client.0.smithi031.stdout: create 125/39 file after the snapshot 2023-10-30T13:37:16.427 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/.snap/snap-subdir-test 2023-10-30T13:37:16.495 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/30 file after the snapshot 2023-10-30T13:37:16.498 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/31 file after the snapshot 2023-10-30T13:37:16.498 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/32 file after the snapshot 2023-10-30T13:37:16.499 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/33 file after the snapshot 2023-10-30T13:37:16.499 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/34 file after the snapshot 2023-10-30T13:37:16.499 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/35 file after the snapshot 2023-10-30T13:37:16.499 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/36 file after the snapshot 2023-10-30T13:37:16.499 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/37 file after the snapshot 2023-10-30T13:37:16.500 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/38 file after the snapshot 2023-10-30T13:37:16.500 INFO:tasks.workunit.client.0.smithi031.stdout: create 126/39 file after the snapshot 2023-10-30T13:37:16.500 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/.snap/snap-subdir-test 2023-10-30T13:37:16.563 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/30 file after the snapshot 2023-10-30T13:37:16.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/31 file after the snapshot 2023-10-30T13:37:16.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/32 file after the snapshot 2023-10-30T13:37:16.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/33 file after the snapshot 2023-10-30T13:37:16.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/34 file after the snapshot 2023-10-30T13:37:16.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/35 file after the snapshot 2023-10-30T13:37:16.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/36 file after the snapshot 2023-10-30T13:37:16.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/37 file after the snapshot 2023-10-30T13:37:16.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/38 file after the snapshot 2023-10-30T13:37:16.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 127/39 file after the snapshot 2023-10-30T13:37:16.570 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/.snap/snap-subdir-test 2023-10-30T13:37:16.651 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/30 file after the snapshot 2023-10-30T13:37:16.655 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/31 file after the snapshot 2023-10-30T13:37:16.655 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/32 file after the snapshot 2023-10-30T13:37:16.655 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/33 file after the snapshot 2023-10-30T13:37:16.655 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/34 file after the snapshot 2023-10-30T13:37:16.656 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/35 file after the snapshot 2023-10-30T13:37:16.656 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/36 file after the snapshot 2023-10-30T13:37:16.656 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/37 file after the snapshot 2023-10-30T13:37:16.656 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/38 file after the snapshot 2023-10-30T13:37:16.656 INFO:tasks.workunit.client.0.smithi031.stdout: create 128/39 file after the snapshot 2023-10-30T13:37:16.657 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/.snap/snap-subdir-test 2023-10-30T13:37:16.726 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/30 file after the snapshot 2023-10-30T13:37:16.729 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/31 file after the snapshot 2023-10-30T13:37:16.729 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/32 file after the snapshot 2023-10-30T13:37:16.729 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/33 file after the snapshot 2023-10-30T13:37:16.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/34 file after the snapshot 2023-10-30T13:37:16.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/35 file after the snapshot 2023-10-30T13:37:16.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/36 file after the snapshot 2023-10-30T13:37:16.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/37 file after the snapshot 2023-10-30T13:37:16.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/38 file after the snapshot 2023-10-30T13:37:16.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 129/39 file after the snapshot 2023-10-30T13:37:16.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/.snap/snap-subdir-test 2023-10-30T13:37:16.794 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/30 file after the snapshot 2023-10-30T13:37:16.799 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/31 file after the snapshot 2023-10-30T13:37:16.799 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/32 file after the snapshot 2023-10-30T13:37:16.799 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/33 file after the snapshot 2023-10-30T13:37:16.799 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/34 file after the snapshot 2023-10-30T13:37:16.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/35 file after the snapshot 2023-10-30T13:37:16.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/36 file after the snapshot 2023-10-30T13:37:16.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/37 file after the snapshot 2023-10-30T13:37:16.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/38 file after the snapshot 2023-10-30T13:37:16.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 130/39 file after the snapshot 2023-10-30T13:37:16.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/.snap/snap-subdir-test 2023-10-30T13:37:16.869 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/30 file after the snapshot 2023-10-30T13:37:16.872 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/31 file after the snapshot 2023-10-30T13:37:16.873 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/32 file after the snapshot 2023-10-30T13:37:16.873 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/33 file after the snapshot 2023-10-30T13:37:16.873 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/34 file after the snapshot 2023-10-30T13:37:16.873 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/35 file after the snapshot 2023-10-30T13:37:16.873 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/36 file after the snapshot 2023-10-30T13:37:16.873 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/37 file after the snapshot 2023-10-30T13:37:16.874 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/38 file after the snapshot 2023-10-30T13:37:16.874 INFO:tasks.workunit.client.0.smithi031.stdout: create 131/39 file after the snapshot 2023-10-30T13:37:16.874 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/.snap/snap-subdir-test 2023-10-30T13:37:16.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/30 file after the snapshot 2023-10-30T13:37:16.947 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/31 file after the snapshot 2023-10-30T13:37:16.948 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/32 file after the snapshot 2023-10-30T13:37:16.948 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/33 file after the snapshot 2023-10-30T13:37:16.948 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/34 file after the snapshot 2023-10-30T13:37:16.948 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/35 file after the snapshot 2023-10-30T13:37:16.948 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/36 file after the snapshot 2023-10-30T13:37:16.949 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/37 file after the snapshot 2023-10-30T13:37:16.949 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/38 file after the snapshot 2023-10-30T13:37:16.949 INFO:tasks.workunit.client.0.smithi031.stdout: create 132/39 file after the snapshot 2023-10-30T13:37:16.949 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/.snap/snap-subdir-test 2023-10-30T13:37:17.262 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/30 file after the snapshot 2023-10-30T13:37:17.265 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/31 file after the snapshot 2023-10-30T13:37:17.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/32 file after the snapshot 2023-10-30T13:37:17.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/33 file after the snapshot 2023-10-30T13:37:17.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/34 file after the snapshot 2023-10-30T13:37:17.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/35 file after the snapshot 2023-10-30T13:37:17.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/36 file after the snapshot 2023-10-30T13:37:17.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/37 file after the snapshot 2023-10-30T13:37:17.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/38 file after the snapshot 2023-10-30T13:37:17.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 133/39 file after the snapshot 2023-10-30T13:37:17.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/.snap/snap-subdir-test 2023-10-30T13:37:17.335 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/30 file after the snapshot 2023-10-30T13:37:17.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/31 file after the snapshot 2023-10-30T13:37:17.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/32 file after the snapshot 2023-10-30T13:37:17.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/33 file after the snapshot 2023-10-30T13:37:17.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/34 file after the snapshot 2023-10-30T13:37:17.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/35 file after the snapshot 2023-10-30T13:37:17.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/36 file after the snapshot 2023-10-30T13:37:17.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/37 file after the snapshot 2023-10-30T13:37:17.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/38 file after the snapshot 2023-10-30T13:37:17.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 134/39 file after the snapshot 2023-10-30T13:37:17.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/.snap/snap-subdir-test 2023-10-30T13:37:17.702 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/30 file after the snapshot 2023-10-30T13:37:17.733 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/31 file after the snapshot 2023-10-30T13:37:17.734 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/32 file after the snapshot 2023-10-30T13:37:17.734 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/33 file after the snapshot 2023-10-30T13:37:17.734 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/34 file after the snapshot 2023-10-30T13:37:17.734 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/35 file after the snapshot 2023-10-30T13:37:17.734 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/36 file after the snapshot 2023-10-30T13:37:17.734 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/37 file after the snapshot 2023-10-30T13:37:17.735 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/38 file after the snapshot 2023-10-30T13:37:17.735 INFO:tasks.workunit.client.0.smithi031.stdout: create 135/39 file after the snapshot 2023-10-30T13:37:17.735 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/.snap/snap-subdir-test 2023-10-30T13:37:17.773 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/30 file after the snapshot 2023-10-30T13:37:17.776 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/31 file after the snapshot 2023-10-30T13:37:17.776 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/32 file after the snapshot 2023-10-30T13:37:17.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/33 file after the snapshot 2023-10-30T13:37:17.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/34 file after the snapshot 2023-10-30T13:37:17.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/35 file after the snapshot 2023-10-30T13:37:17.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/36 file after the snapshot 2023-10-30T13:37:17.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/37 file after the snapshot 2023-10-30T13:37:17.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/38 file after the snapshot 2023-10-30T13:37:17.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 136/39 file after the snapshot 2023-10-30T13:37:17.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/.snap/snap-subdir-test 2023-10-30T13:37:18.757 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/30 file after the snapshot 2023-10-30T13:37:18.760 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/31 file after the snapshot 2023-10-30T13:37:18.760 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/32 file after the snapshot 2023-10-30T13:37:18.760 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/33 file after the snapshot 2023-10-30T13:37:18.761 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/34 file after the snapshot 2023-10-30T13:37:18.761 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/35 file after the snapshot 2023-10-30T13:37:18.761 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/36 file after the snapshot 2023-10-30T13:37:18.761 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/37 file after the snapshot 2023-10-30T13:37:18.761 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/38 file after the snapshot 2023-10-30T13:37:18.762 INFO:tasks.workunit.client.0.smithi031.stdout: create 137/39 file after the snapshot 2023-10-30T13:37:18.762 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/.snap/snap-subdir-test 2023-10-30T13:37:18.829 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/30 file after the snapshot 2023-10-30T13:37:18.834 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/31 file after the snapshot 2023-10-30T13:37:18.834 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/32 file after the snapshot 2023-10-30T13:37:18.834 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/33 file after the snapshot 2023-10-30T13:37:18.835 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/34 file after the snapshot 2023-10-30T13:37:18.835 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/35 file after the snapshot 2023-10-30T13:37:18.835 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/36 file after the snapshot 2023-10-30T13:37:18.835 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/37 file after the snapshot 2023-10-30T13:37:18.835 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/38 file after the snapshot 2023-10-30T13:37:18.836 INFO:tasks.workunit.client.0.smithi031.stdout: create 138/39 file after the snapshot 2023-10-30T13:37:18.836 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/.snap/snap-subdir-test 2023-10-30T13:37:18.923 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/30 file after the snapshot 2023-10-30T13:37:18.926 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/31 file after the snapshot 2023-10-30T13:37:18.927 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/32 file after the snapshot 2023-10-30T13:37:18.927 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/33 file after the snapshot 2023-10-30T13:37:18.927 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/34 file after the snapshot 2023-10-30T13:37:18.927 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/35 file after the snapshot 2023-10-30T13:37:18.927 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/36 file after the snapshot 2023-10-30T13:37:18.928 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/37 file after the snapshot 2023-10-30T13:37:18.928 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/38 file after the snapshot 2023-10-30T13:37:18.928 INFO:tasks.workunit.client.0.smithi031.stdout: create 139/39 file after the snapshot 2023-10-30T13:37:18.928 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/.snap/snap-subdir-test 2023-10-30T13:37:19.715 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/30 file after the snapshot 2023-10-30T13:37:19.720 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/31 file after the snapshot 2023-10-30T13:37:19.720 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/32 file after the snapshot 2023-10-30T13:37:19.720 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/33 file after the snapshot 2023-10-30T13:37:19.720 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/34 file after the snapshot 2023-10-30T13:37:19.721 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/35 file after the snapshot 2023-10-30T13:37:19.721 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/36 file after the snapshot 2023-10-30T13:37:19.721 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/37 file after the snapshot 2023-10-30T13:37:19.722 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/38 file after the snapshot 2023-10-30T13:37:19.722 INFO:tasks.workunit.client.0.smithi031.stdout: create 140/39 file after the snapshot 2023-10-30T13:37:19.722 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/.snap/snap-subdir-test 2023-10-30T13:37:19.806 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/30 file after the snapshot 2023-10-30T13:37:19.810 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/31 file after the snapshot 2023-10-30T13:37:19.810 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/32 file after the snapshot 2023-10-30T13:37:19.810 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/33 file after the snapshot 2023-10-30T13:37:19.811 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/34 file after the snapshot 2023-10-30T13:37:19.811 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/35 file after the snapshot 2023-10-30T13:37:19.811 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/36 file after the snapshot 2023-10-30T13:37:19.811 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/37 file after the snapshot 2023-10-30T13:37:19.811 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/38 file after the snapshot 2023-10-30T13:37:19.812 INFO:tasks.workunit.client.0.smithi031.stdout: create 141/39 file after the snapshot 2023-10-30T13:37:19.812 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/.snap/snap-subdir-test 2023-10-30T13:37:19.876 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/30 file after the snapshot 2023-10-30T13:37:19.889 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/31 file after the snapshot 2023-10-30T13:37:19.891 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/32 file after the snapshot 2023-10-30T13:37:19.891 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/33 file after the snapshot 2023-10-30T13:37:19.891 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/34 file after the snapshot 2023-10-30T13:37:19.891 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/35 file after the snapshot 2023-10-30T13:37:19.891 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/36 file after the snapshot 2023-10-30T13:37:19.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/37 file after the snapshot 2023-10-30T13:37:19.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/38 file after the snapshot 2023-10-30T13:37:19.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 142/39 file after the snapshot 2023-10-30T13:37:19.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/.snap/snap-subdir-test 2023-10-30T13:37:20.276 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/30 file after the snapshot 2023-10-30T13:37:20.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/31 file after the snapshot 2023-10-30T13:37:20.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/32 file after the snapshot 2023-10-30T13:37:20.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/33 file after the snapshot 2023-10-30T13:37:20.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/34 file after the snapshot 2023-10-30T13:37:20.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/35 file after the snapshot 2023-10-30T13:37:20.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/36 file after the snapshot 2023-10-30T13:37:20.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/37 file after the snapshot 2023-10-30T13:37:20.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/38 file after the snapshot 2023-10-30T13:37:20.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 143/39 file after the snapshot 2023-10-30T13:37:20.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/.snap/snap-subdir-test 2023-10-30T13:37:20.833 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/30 file after the snapshot 2023-10-30T13:37:20.838 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/31 file after the snapshot 2023-10-30T13:37:20.839 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/32 file after the snapshot 2023-10-30T13:37:20.839 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/33 file after the snapshot 2023-10-30T13:37:20.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/34 file after the snapshot 2023-10-30T13:37:20.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/35 file after the snapshot 2023-10-30T13:37:20.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/36 file after the snapshot 2023-10-30T13:37:20.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/37 file after the snapshot 2023-10-30T13:37:20.841 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/38 file after the snapshot 2023-10-30T13:37:20.841 INFO:tasks.workunit.client.0.smithi031.stdout: create 144/39 file after the snapshot 2023-10-30T13:37:20.841 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/.snap/snap-subdir-test 2023-10-30T13:37:20.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/30 file after the snapshot 2023-10-30T13:37:20.897 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/31 file after the snapshot 2023-10-30T13:37:20.897 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/32 file after the snapshot 2023-10-30T13:37:20.897 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/33 file after the snapshot 2023-10-30T13:37:20.897 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/34 file after the snapshot 2023-10-30T13:37:20.898 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/35 file after the snapshot 2023-10-30T13:37:20.898 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/36 file after the snapshot 2023-10-30T13:37:20.898 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/37 file after the snapshot 2023-10-30T13:37:20.898 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/38 file after the snapshot 2023-10-30T13:37:20.898 INFO:tasks.workunit.client.0.smithi031.stdout: create 145/39 file after the snapshot 2023-10-30T13:37:20.899 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/.snap/snap-subdir-test 2023-10-30T13:37:20.952 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/30 file after the snapshot 2023-10-30T13:37:21.368 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/31 file after the snapshot 2023-10-30T13:37:21.369 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/32 file after the snapshot 2023-10-30T13:37:21.369 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/33 file after the snapshot 2023-10-30T13:37:21.369 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/34 file after the snapshot 2023-10-30T13:37:21.369 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/35 file after the snapshot 2023-10-30T13:37:21.369 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/36 file after the snapshot 2023-10-30T13:37:21.369 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/37 file after the snapshot 2023-10-30T13:37:21.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/38 file after the snapshot 2023-10-30T13:37:21.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 146/39 file after the snapshot 2023-10-30T13:37:21.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/.snap/snap-subdir-test 2023-10-30T13:37:21.405 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/30 file after the snapshot 2023-10-30T13:37:21.415 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/31 file after the snapshot 2023-10-30T13:37:21.417 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/32 file after the snapshot 2023-10-30T13:37:21.418 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/33 file after the snapshot 2023-10-30T13:37:21.418 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/34 file after the snapshot 2023-10-30T13:37:21.418 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/35 file after the snapshot 2023-10-30T13:37:21.418 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/36 file after the snapshot 2023-10-30T13:37:21.418 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/37 file after the snapshot 2023-10-30T13:37:21.419 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/38 file after the snapshot 2023-10-30T13:37:21.419 INFO:tasks.workunit.client.0.smithi031.stdout: create 147/39 file after the snapshot 2023-10-30T13:37:21.419 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/.snap/snap-subdir-test 2023-10-30T13:37:21.466 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/30 file after the snapshot 2023-10-30T13:37:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/31 file after the snapshot 2023-10-30T13:37:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/32 file after the snapshot 2023-10-30T13:37:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/33 file after the snapshot 2023-10-30T13:37:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/34 file after the snapshot 2023-10-30T13:37:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/35 file after the snapshot 2023-10-30T13:37:21.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/36 file after the snapshot 2023-10-30T13:37:21.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/37 file after the snapshot 2023-10-30T13:37:21.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/38 file after the snapshot 2023-10-30T13:37:21.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 148/39 file after the snapshot 2023-10-30T13:37:21.470 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/.snap/snap-subdir-test 2023-10-30T13:37:21.515 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/30 file after the snapshot 2023-10-30T13:37:21.518 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/31 file after the snapshot 2023-10-30T13:37:21.518 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/32 file after the snapshot 2023-10-30T13:37:21.518 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/33 file after the snapshot 2023-10-30T13:37:21.519 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/34 file after the snapshot 2023-10-30T13:37:21.519 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/35 file after the snapshot 2023-10-30T13:37:21.519 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/36 file after the snapshot 2023-10-30T13:37:21.519 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/37 file after the snapshot 2023-10-30T13:37:21.519 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/38 file after the snapshot 2023-10-30T13:37:21.519 INFO:tasks.workunit.client.0.smithi031.stdout: create 149/39 file after the snapshot 2023-10-30T13:37:21.520 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/.snap/snap-subdir-test 2023-10-30T13:37:21.629 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/30 file after the snapshot 2023-10-30T13:37:21.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/31 file after the snapshot 2023-10-30T13:37:21.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/32 file after the snapshot 2023-10-30T13:37:21.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/33 file after the snapshot 2023-10-30T13:37:21.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/34 file after the snapshot 2023-10-30T13:37:21.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/35 file after the snapshot 2023-10-30T13:37:21.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/36 file after the snapshot 2023-10-30T13:37:21.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/37 file after the snapshot 2023-10-30T13:37:21.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/38 file after the snapshot 2023-10-30T13:37:21.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 150/39 file after the snapshot 2023-10-30T13:37:21.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/.snap/snap-subdir-test 2023-10-30T13:37:21.683 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/30 file after the snapshot 2023-10-30T13:37:21.686 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/31 file after the snapshot 2023-10-30T13:37:21.686 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/32 file after the snapshot 2023-10-30T13:37:21.686 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/33 file after the snapshot 2023-10-30T13:37:21.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/34 file after the snapshot 2023-10-30T13:37:21.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/35 file after the snapshot 2023-10-30T13:37:21.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/36 file after the snapshot 2023-10-30T13:37:21.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/37 file after the snapshot 2023-10-30T13:37:21.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/38 file after the snapshot 2023-10-30T13:37:21.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 151/39 file after the snapshot 2023-10-30T13:37:21.688 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/.snap/snap-subdir-test 2023-10-30T13:37:21.724 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/30 file after the snapshot 2023-10-30T13:37:21.727 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/31 file after the snapshot 2023-10-30T13:37:21.727 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/32 file after the snapshot 2023-10-30T13:37:21.727 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/33 file after the snapshot 2023-10-30T13:37:21.727 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/34 file after the snapshot 2023-10-30T13:37:21.728 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/35 file after the snapshot 2023-10-30T13:37:21.728 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/36 file after the snapshot 2023-10-30T13:37:21.728 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/37 file after the snapshot 2023-10-30T13:37:21.728 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/38 file after the snapshot 2023-10-30T13:37:21.728 INFO:tasks.workunit.client.0.smithi031.stdout: create 152/39 file after the snapshot 2023-10-30T13:37:21.728 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/.snap/snap-subdir-test 2023-10-30T13:37:22.041 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/30 file after the snapshot 2023-10-30T13:37:22.296 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/31 file after the snapshot 2023-10-30T13:37:22.297 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/32 file after the snapshot 2023-10-30T13:37:22.297 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/33 file after the snapshot 2023-10-30T13:37:22.297 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/34 file after the snapshot 2023-10-30T13:37:22.297 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/35 file after the snapshot 2023-10-30T13:37:22.297 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/36 file after the snapshot 2023-10-30T13:37:22.298 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/37 file after the snapshot 2023-10-30T13:37:22.298 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/38 file after the snapshot 2023-10-30T13:37:22.298 INFO:tasks.workunit.client.0.smithi031.stdout: create 153/39 file after the snapshot 2023-10-30T13:37:22.298 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/.snap/snap-subdir-test 2023-10-30T13:37:22.361 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/30 file after the snapshot 2023-10-30T13:37:22.364 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/31 file after the snapshot 2023-10-30T13:37:22.364 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/32 file after the snapshot 2023-10-30T13:37:22.364 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/33 file after the snapshot 2023-10-30T13:37:22.364 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/34 file after the snapshot 2023-10-30T13:37:22.364 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/35 file after the snapshot 2023-10-30T13:37:22.365 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/36 file after the snapshot 2023-10-30T13:37:22.365 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/37 file after the snapshot 2023-10-30T13:37:22.365 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/38 file after the snapshot 2023-10-30T13:37:22.365 INFO:tasks.workunit.client.0.smithi031.stdout: create 154/39 file after the snapshot 2023-10-30T13:37:22.365 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/.snap/snap-subdir-test 2023-10-30T13:37:22.431 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/30 file after the snapshot 2023-10-30T13:37:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/31 file after the snapshot 2023-10-30T13:37:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/32 file after the snapshot 2023-10-30T13:37:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/33 file after the snapshot 2023-10-30T13:37:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/34 file after the snapshot 2023-10-30T13:37:22.438 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/35 file after the snapshot 2023-10-30T13:37:22.438 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/36 file after the snapshot 2023-10-30T13:37:22.438 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/37 file after the snapshot 2023-10-30T13:37:22.438 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/38 file after the snapshot 2023-10-30T13:37:22.438 INFO:tasks.workunit.client.0.smithi031.stdout: create 155/39 file after the snapshot 2023-10-30T13:37:22.439 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/.snap/snap-subdir-test 2023-10-30T13:37:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/30 file after the snapshot 2023-10-30T13:37:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/31 file after the snapshot 2023-10-30T13:37:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/32 file after the snapshot 2023-10-30T13:37:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/33 file after the snapshot 2023-10-30T13:37:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/34 file after the snapshot 2023-10-30T13:37:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/35 file after the snapshot 2023-10-30T13:37:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/36 file after the snapshot 2023-10-30T13:37:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/37 file after the snapshot 2023-10-30T13:37:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/38 file after the snapshot 2023-10-30T13:37:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: create 156/39 file after the snapshot 2023-10-30T13:37:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/.snap/snap-subdir-test 2023-10-30T13:37:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/30 file after the snapshot 2023-10-30T13:37:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/31 file after the snapshot 2023-10-30T13:37:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/32 file after the snapshot 2023-10-30T13:37:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/33 file after the snapshot 2023-10-30T13:37:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/34 file after the snapshot 2023-10-30T13:37:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/35 file after the snapshot 2023-10-30T13:37:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/36 file after the snapshot 2023-10-30T13:37:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/37 file after the snapshot 2023-10-30T13:37:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/38 file after the snapshot 2023-10-30T13:37:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: create 157/39 file after the snapshot 2023-10-30T13:37:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/.snap/snap-subdir-test 2023-10-30T13:37:22.628 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/30 file after the snapshot 2023-10-30T13:37:22.631 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/31 file after the snapshot 2023-10-30T13:37:22.631 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/32 file after the snapshot 2023-10-30T13:37:22.631 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/33 file after the snapshot 2023-10-30T13:37:22.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/34 file after the snapshot 2023-10-30T13:37:22.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/35 file after the snapshot 2023-10-30T13:37:22.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/36 file after the snapshot 2023-10-30T13:37:22.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/37 file after the snapshot 2023-10-30T13:37:22.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/38 file after the snapshot 2023-10-30T13:37:22.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 158/39 file after the snapshot 2023-10-30T13:37:22.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/.snap/snap-subdir-test 2023-10-30T13:37:22.878 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/30 file after the snapshot 2023-10-30T13:37:23.130 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/31 file after the snapshot 2023-10-30T13:37:23.131 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/32 file after the snapshot 2023-10-30T13:37:23.131 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/33 file after the snapshot 2023-10-30T13:37:23.131 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/34 file after the snapshot 2023-10-30T13:37:23.131 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/35 file after the snapshot 2023-10-30T13:37:23.132 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/36 file after the snapshot 2023-10-30T13:37:23.132 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/37 file after the snapshot 2023-10-30T13:37:23.132 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/38 file after the snapshot 2023-10-30T13:37:23.132 INFO:tasks.workunit.client.0.smithi031.stdout: create 159/39 file after the snapshot 2023-10-30T13:37:23.132 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/.snap/snap-subdir-test 2023-10-30T13:37:23.195 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/30 file after the snapshot 2023-10-30T13:37:23.198 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/31 file after the snapshot 2023-10-30T13:37:23.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/32 file after the snapshot 2023-10-30T13:37:23.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/33 file after the snapshot 2023-10-30T13:37:23.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/34 file after the snapshot 2023-10-30T13:37:23.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/35 file after the snapshot 2023-10-30T13:37:23.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/36 file after the snapshot 2023-10-30T13:37:23.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/37 file after the snapshot 2023-10-30T13:37:23.200 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/38 file after the snapshot 2023-10-30T13:37:23.200 INFO:tasks.workunit.client.0.smithi031.stdout: create 160/39 file after the snapshot 2023-10-30T13:37:23.200 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/.snap/snap-subdir-test 2023-10-30T13:37:23.264 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/30 file after the snapshot 2023-10-30T13:37:23.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/31 file after the snapshot 2023-10-30T13:37:23.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/32 file after the snapshot 2023-10-30T13:37:23.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/33 file after the snapshot 2023-10-30T13:37:23.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/34 file after the snapshot 2023-10-30T13:37:23.267 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/35 file after the snapshot 2023-10-30T13:37:23.268 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/36 file after the snapshot 2023-10-30T13:37:23.268 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/37 file after the snapshot 2023-10-30T13:37:23.268 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/38 file after the snapshot 2023-10-30T13:37:23.268 INFO:tasks.workunit.client.0.smithi031.stdout: create 161/39 file after the snapshot 2023-10-30T13:37:23.268 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/.snap/snap-subdir-test 2023-10-30T13:37:23.336 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/30 file after the snapshot 2023-10-30T13:37:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/31 file after the snapshot 2023-10-30T13:37:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/32 file after the snapshot 2023-10-30T13:37:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/33 file after the snapshot 2023-10-30T13:37:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/34 file after the snapshot 2023-10-30T13:37:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/35 file after the snapshot 2023-10-30T13:37:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/36 file after the snapshot 2023-10-30T13:37:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/37 file after the snapshot 2023-10-30T13:37:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/38 file after the snapshot 2023-10-30T13:37:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: create 162/39 file after the snapshot 2023-10-30T13:37:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/.snap/snap-subdir-test 2023-10-30T13:37:23.545 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/30 file after the snapshot 2023-10-30T13:37:23.549 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/31 file after the snapshot 2023-10-30T13:37:23.550 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/32 file after the snapshot 2023-10-30T13:37:23.550 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/33 file after the snapshot 2023-10-30T13:37:23.550 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/34 file after the snapshot 2023-10-30T13:37:23.550 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/35 file after the snapshot 2023-10-30T13:37:23.551 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/36 file after the snapshot 2023-10-30T13:37:23.551 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/37 file after the snapshot 2023-10-30T13:37:23.552 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/38 file after the snapshot 2023-10-30T13:37:23.552 INFO:tasks.workunit.client.0.smithi031.stdout: create 163/39 file after the snapshot 2023-10-30T13:37:23.552 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/.snap/snap-subdir-test 2023-10-30T13:37:23.616 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/30 file after the snapshot 2023-10-30T13:37:23.618 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/31 file after the snapshot 2023-10-30T13:37:23.619 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/32 file after the snapshot 2023-10-30T13:37:23.619 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/33 file after the snapshot 2023-10-30T13:37:23.619 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/34 file after the snapshot 2023-10-30T13:37:23.619 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/35 file after the snapshot 2023-10-30T13:37:23.620 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/36 file after the snapshot 2023-10-30T13:37:23.620 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/37 file after the snapshot 2023-10-30T13:37:23.621 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/38 file after the snapshot 2023-10-30T13:37:23.621 INFO:tasks.workunit.client.0.smithi031.stdout: create 164/39 file after the snapshot 2023-10-30T13:37:23.621 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/.snap/snap-subdir-test 2023-10-30T13:37:23.710 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/30 file after the snapshot 2023-10-30T13:37:23.715 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/31 file after the snapshot 2023-10-30T13:37:23.715 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/32 file after the snapshot 2023-10-30T13:37:23.715 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/33 file after the snapshot 2023-10-30T13:37:23.716 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/34 file after the snapshot 2023-10-30T13:37:23.716 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/35 file after the snapshot 2023-10-30T13:37:23.716 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/36 file after the snapshot 2023-10-30T13:37:23.716 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/37 file after the snapshot 2023-10-30T13:37:23.716 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/38 file after the snapshot 2023-10-30T13:37:23.717 INFO:tasks.workunit.client.0.smithi031.stdout: create 165/39 file after the snapshot 2023-10-30T13:37:23.717 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/.snap/snap-subdir-test 2023-10-30T13:37:25.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:25.168 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:25.318 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/30 file after the snapshot 2023-10-30T13:37:25.353 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/31 file after the snapshot 2023-10-30T13:37:25.353 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/32 file after the snapshot 2023-10-30T13:37:25.353 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/33 file after the snapshot 2023-10-30T13:37:25.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/34 file after the snapshot 2023-10-30T13:37:25.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/35 file after the snapshot 2023-10-30T13:37:25.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/36 file after the snapshot 2023-10-30T13:37:25.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/37 file after the snapshot 2023-10-30T13:37:25.354 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/38 file after the snapshot 2023-10-30T13:37:25.355 INFO:tasks.workunit.client.0.smithi031.stdout: create 166/39 file after the snapshot 2023-10-30T13:37:25.355 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/.snap/snap-subdir-test 2023-10-30T13:37:27.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/30 file after the snapshot 2023-10-30T13:37:27.259 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/31 file after the snapshot 2023-10-30T13:37:27.259 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/32 file after the snapshot 2023-10-30T13:37:27.259 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/33 file after the snapshot 2023-10-30T13:37:27.260 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/34 file after the snapshot 2023-10-30T13:37:27.260 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/35 file after the snapshot 2023-10-30T13:37:27.260 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/36 file after the snapshot 2023-10-30T13:37:27.260 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/37 file after the snapshot 2023-10-30T13:37:27.260 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/38 file after the snapshot 2023-10-30T13:37:27.260 INFO:tasks.workunit.client.0.smithi031.stdout: create 167/39 file after the snapshot 2023-10-30T13:37:27.261 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/.snap/snap-subdir-test 2023-10-30T13:37:27.417 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/30 file after the snapshot 2023-10-30T13:37:27.420 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/31 file after the snapshot 2023-10-30T13:37:27.420 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/32 file after the snapshot 2023-10-30T13:37:27.421 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/33 file after the snapshot 2023-10-30T13:37:27.421 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/34 file after the snapshot 2023-10-30T13:37:27.421 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/35 file after the snapshot 2023-10-30T13:37:27.421 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/36 file after the snapshot 2023-10-30T13:37:27.421 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/37 file after the snapshot 2023-10-30T13:37:27.421 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/38 file after the snapshot 2023-10-30T13:37:27.422 INFO:tasks.workunit.client.0.smithi031.stdout: create 168/39 file after the snapshot 2023-10-30T13:37:27.422 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/.snap/snap-subdir-test 2023-10-30T13:37:27.544 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/30 file after the snapshot 2023-10-30T13:37:27.548 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/31 file after the snapshot 2023-10-30T13:37:27.551 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/32 file after the snapshot 2023-10-30T13:37:27.551 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/33 file after the snapshot 2023-10-30T13:37:27.551 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/34 file after the snapshot 2023-10-30T13:37:27.551 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/35 file after the snapshot 2023-10-30T13:37:27.551 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/36 file after the snapshot 2023-10-30T13:37:27.552 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/37 file after the snapshot 2023-10-30T13:37:27.552 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/38 file after the snapshot 2023-10-30T13:37:27.552 INFO:tasks.workunit.client.0.smithi031.stdout: create 169/39 file after the snapshot 2023-10-30T13:37:27.552 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/.snap/snap-subdir-test 2023-10-30T13:37:27.699 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/30 file after the snapshot 2023-10-30T13:37:27.702 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/31 file after the snapshot 2023-10-30T13:37:27.703 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/32 file after the snapshot 2023-10-30T13:37:27.703 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/33 file after the snapshot 2023-10-30T13:37:27.703 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/34 file after the snapshot 2023-10-30T13:37:27.703 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/35 file after the snapshot 2023-10-30T13:37:27.703 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/36 file after the snapshot 2023-10-30T13:37:27.703 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/37 file after the snapshot 2023-10-30T13:37:27.704 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/38 file after the snapshot 2023-10-30T13:37:27.704 INFO:tasks.workunit.client.0.smithi031.stdout: create 170/39 file after the snapshot 2023-10-30T13:37:27.704 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/.snap/snap-subdir-test 2023-10-30T13:37:27.728 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/30 file after the snapshot 2023-10-30T13:37:27.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/31 file after the snapshot 2023-10-30T13:37:27.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/32 file after the snapshot 2023-10-30T13:37:27.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/33 file after the snapshot 2023-10-30T13:37:27.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/34 file after the snapshot 2023-10-30T13:37:27.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/35 file after the snapshot 2023-10-30T13:37:27.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/36 file after the snapshot 2023-10-30T13:37:27.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/37 file after the snapshot 2023-10-30T13:37:27.732 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/38 file after the snapshot 2023-10-30T13:37:27.732 INFO:tasks.workunit.client.0.smithi031.stdout: create 171/39 file after the snapshot 2023-10-30T13:37:27.732 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/.snap/snap-subdir-test 2023-10-30T13:37:27.762 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/30 file after the snapshot 2023-10-30T13:37:27.764 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/31 file after the snapshot 2023-10-30T13:37:27.764 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/32 file after the snapshot 2023-10-30T13:37:27.764 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/33 file after the snapshot 2023-10-30T13:37:27.764 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/34 file after the snapshot 2023-10-30T13:37:27.764 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/35 file after the snapshot 2023-10-30T13:37:27.765 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/36 file after the snapshot 2023-10-30T13:37:27.765 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/37 file after the snapshot 2023-10-30T13:37:27.765 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/38 file after the snapshot 2023-10-30T13:37:27.765 INFO:tasks.workunit.client.0.smithi031.stdout: create 172/39 file after the snapshot 2023-10-30T13:37:27.765 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/.snap/snap-subdir-test 2023-10-30T13:37:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/30 file after the snapshot 2023-10-30T13:37:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/31 file after the snapshot 2023-10-30T13:37:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/32 file after the snapshot 2023-10-30T13:37:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/33 file after the snapshot 2023-10-30T13:37:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/34 file after the snapshot 2023-10-30T13:37:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/35 file after the snapshot 2023-10-30T13:37:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/36 file after the snapshot 2023-10-30T13:37:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/37 file after the snapshot 2023-10-30T13:37:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/38 file after the snapshot 2023-10-30T13:37:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 173/39 file after the snapshot 2023-10-30T13:37:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/.snap/snap-subdir-test 2023-10-30T13:37:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/30 file after the snapshot 2023-10-30T13:37:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/31 file after the snapshot 2023-10-30T13:37:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/32 file after the snapshot 2023-10-30T13:37:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/33 file after the snapshot 2023-10-30T13:37:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/34 file after the snapshot 2023-10-30T13:37:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/35 file after the snapshot 2023-10-30T13:37:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/36 file after the snapshot 2023-10-30T13:37:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/37 file after the snapshot 2023-10-30T13:37:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/38 file after the snapshot 2023-10-30T13:37:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: create 174/39 file after the snapshot 2023-10-30T13:37:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/.snap/snap-subdir-test 2023-10-30T13:37:27.899 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/30 file after the snapshot 2023-10-30T13:37:27.900 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/31 file after the snapshot 2023-10-30T13:37:27.901 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/32 file after the snapshot 2023-10-30T13:37:27.901 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/33 file after the snapshot 2023-10-30T13:37:27.901 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/34 file after the snapshot 2023-10-30T13:37:27.901 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/35 file after the snapshot 2023-10-30T13:37:27.901 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/36 file after the snapshot 2023-10-30T13:37:27.902 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/37 file after the snapshot 2023-10-30T13:37:27.902 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/38 file after the snapshot 2023-10-30T13:37:27.902 INFO:tasks.workunit.client.0.smithi031.stdout: create 175/39 file after the snapshot 2023-10-30T13:37:27.902 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/.snap/snap-subdir-test 2023-10-30T13:37:27.939 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/30 file after the snapshot 2023-10-30T13:37:27.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/31 file after the snapshot 2023-10-30T13:37:27.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/32 file after the snapshot 2023-10-30T13:37:27.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/33 file after the snapshot 2023-10-30T13:37:27.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/34 file after the snapshot 2023-10-30T13:37:27.943 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/35 file after the snapshot 2023-10-30T13:37:27.943 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/36 file after the snapshot 2023-10-30T13:37:27.943 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/37 file after the snapshot 2023-10-30T13:37:27.943 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/38 file after the snapshot 2023-10-30T13:37:27.943 INFO:tasks.workunit.client.0.smithi031.stdout: create 176/39 file after the snapshot 2023-10-30T13:37:27.944 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/.snap/snap-subdir-test 2023-10-30T13:37:27.983 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/30 file after the snapshot 2023-10-30T13:37:27.984 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/31 file after the snapshot 2023-10-30T13:37:27.985 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/32 file after the snapshot 2023-10-30T13:37:27.985 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/33 file after the snapshot 2023-10-30T13:37:27.985 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/34 file after the snapshot 2023-10-30T13:37:27.985 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/35 file after the snapshot 2023-10-30T13:37:27.985 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/36 file after the snapshot 2023-10-30T13:37:27.986 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/37 file after the snapshot 2023-10-30T13:37:27.986 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/38 file after the snapshot 2023-10-30T13:37:27.986 INFO:tasks.workunit.client.0.smithi031.stdout: create 177/39 file after the snapshot 2023-10-30T13:37:27.986 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/.snap/snap-subdir-test 2023-10-30T13:37:28.023 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/30 file after the snapshot 2023-10-30T13:37:28.026 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/31 file after the snapshot 2023-10-30T13:37:28.026 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/32 file after the snapshot 2023-10-30T13:37:28.026 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/33 file after the snapshot 2023-10-30T13:37:28.026 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/34 file after the snapshot 2023-10-30T13:37:28.027 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/35 file after the snapshot 2023-10-30T13:37:28.027 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/36 file after the snapshot 2023-10-30T13:37:28.027 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/37 file after the snapshot 2023-10-30T13:37:28.027 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/38 file after the snapshot 2023-10-30T13:37:28.027 INFO:tasks.workunit.client.0.smithi031.stdout: create 178/39 file after the snapshot 2023-10-30T13:37:28.028 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/.snap/snap-subdir-test 2023-10-30T13:37:28.082 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/30 file after the snapshot 2023-10-30T13:37:28.085 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/31 file after the snapshot 2023-10-30T13:37:28.086 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/32 file after the snapshot 2023-10-30T13:37:28.086 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/33 file after the snapshot 2023-10-30T13:37:28.086 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/34 file after the snapshot 2023-10-30T13:37:28.086 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/35 file after the snapshot 2023-10-30T13:37:28.086 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/36 file after the snapshot 2023-10-30T13:37:28.087 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/37 file after the snapshot 2023-10-30T13:37:28.087 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/38 file after the snapshot 2023-10-30T13:37:28.087 INFO:tasks.workunit.client.0.smithi031.stdout: create 179/39 file after the snapshot 2023-10-30T13:37:28.087 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/.snap/snap-subdir-test 2023-10-30T13:37:28.142 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/30 file after the snapshot 2023-10-30T13:37:28.146 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/31 file after the snapshot 2023-10-30T13:37:28.146 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/32 file after the snapshot 2023-10-30T13:37:28.146 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/33 file after the snapshot 2023-10-30T13:37:28.147 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/34 file after the snapshot 2023-10-30T13:37:28.147 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/35 file after the snapshot 2023-10-30T13:37:28.147 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/36 file after the snapshot 2023-10-30T13:37:28.147 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/37 file after the snapshot 2023-10-30T13:37:28.147 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/38 file after the snapshot 2023-10-30T13:37:28.147 INFO:tasks.workunit.client.0.smithi031.stdout: create 180/39 file after the snapshot 2023-10-30T13:37:28.148 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/.snap/snap-subdir-test 2023-10-30T13:37:28.185 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/30 file after the snapshot 2023-10-30T13:37:28.186 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/31 file after the snapshot 2023-10-30T13:37:28.187 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/32 file after the snapshot 2023-10-30T13:37:28.187 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/33 file after the snapshot 2023-10-30T13:37:28.187 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/34 file after the snapshot 2023-10-30T13:37:28.187 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/35 file after the snapshot 2023-10-30T13:37:28.187 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/36 file after the snapshot 2023-10-30T13:37:28.188 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/37 file after the snapshot 2023-10-30T13:37:28.188 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/38 file after the snapshot 2023-10-30T13:37:28.188 INFO:tasks.workunit.client.0.smithi031.stdout: create 181/39 file after the snapshot 2023-10-30T13:37:28.188 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/.snap/snap-subdir-test 2023-10-30T13:37:28.226 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/30 file after the snapshot 2023-10-30T13:37:28.229 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/31 file after the snapshot 2023-10-30T13:37:28.229 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/32 file after the snapshot 2023-10-30T13:37:28.229 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/33 file after the snapshot 2023-10-30T13:37:28.230 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/34 file after the snapshot 2023-10-30T13:37:28.230 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/35 file after the snapshot 2023-10-30T13:37:28.230 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/36 file after the snapshot 2023-10-30T13:37:28.230 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/37 file after the snapshot 2023-10-30T13:37:28.230 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/38 file after the snapshot 2023-10-30T13:37:28.231 INFO:tasks.workunit.client.0.smithi031.stdout: create 182/39 file after the snapshot 2023-10-30T13:37:28.231 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/.snap/snap-subdir-test 2023-10-30T13:37:28.610 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/30 file after the snapshot 2023-10-30T13:37:28.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/31 file after the snapshot 2023-10-30T13:37:28.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/32 file after the snapshot 2023-10-30T13:37:28.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/33 file after the snapshot 2023-10-30T13:37:28.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/34 file after the snapshot 2023-10-30T13:37:28.613 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/35 file after the snapshot 2023-10-30T13:37:28.613 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/36 file after the snapshot 2023-10-30T13:37:28.613 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/37 file after the snapshot 2023-10-30T13:37:28.613 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/38 file after the snapshot 2023-10-30T13:37:28.613 INFO:tasks.workunit.client.0.smithi031.stdout: create 183/39 file after the snapshot 2023-10-30T13:37:28.614 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/.snap/snap-subdir-test 2023-10-30T13:37:28.642 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/30 file after the snapshot 2023-10-30T13:37:28.646 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/31 file after the snapshot 2023-10-30T13:37:28.646 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/32 file after the snapshot 2023-10-30T13:37:28.646 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/33 file after the snapshot 2023-10-30T13:37:28.646 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/34 file after the snapshot 2023-10-30T13:37:28.647 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/35 file after the snapshot 2023-10-30T13:37:28.647 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/36 file after the snapshot 2023-10-30T13:37:28.647 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/37 file after the snapshot 2023-10-30T13:37:28.647 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/38 file after the snapshot 2023-10-30T13:37:28.647 INFO:tasks.workunit.client.0.smithi031.stdout: create 184/39 file after the snapshot 2023-10-30T13:37:28.648 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/.snap/snap-subdir-test 2023-10-30T13:37:28.704 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/30 file after the snapshot 2023-10-30T13:37:28.707 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/31 file after the snapshot 2023-10-30T13:37:28.707 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/32 file after the snapshot 2023-10-30T13:37:28.707 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/33 file after the snapshot 2023-10-30T13:37:28.707 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/34 file after the snapshot 2023-10-30T13:37:28.708 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/35 file after the snapshot 2023-10-30T13:37:28.708 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/36 file after the snapshot 2023-10-30T13:37:28.708 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/37 file after the snapshot 2023-10-30T13:37:28.708 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/38 file after the snapshot 2023-10-30T13:37:28.708 INFO:tasks.workunit.client.0.smithi031.stdout: create 185/39 file after the snapshot 2023-10-30T13:37:28.709 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/.snap/snap-subdir-test 2023-10-30T13:37:29.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/30 file after the snapshot 2023-10-30T13:37:29.913 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/31 file after the snapshot 2023-10-30T13:37:29.913 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/32 file after the snapshot 2023-10-30T13:37:29.913 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/33 file after the snapshot 2023-10-30T13:37:29.913 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/34 file after the snapshot 2023-10-30T13:37:29.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/35 file after the snapshot 2023-10-30T13:37:29.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/36 file after the snapshot 2023-10-30T13:37:29.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/37 file after the snapshot 2023-10-30T13:37:29.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/38 file after the snapshot 2023-10-30T13:37:29.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 186/39 file after the snapshot 2023-10-30T13:37:29.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/.snap/snap-subdir-test 2023-10-30T13:37:30.128 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/30 file after the snapshot 2023-10-30T13:37:30.129 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/31 file after the snapshot 2023-10-30T13:37:30.130 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/32 file after the snapshot 2023-10-30T13:37:30.130 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/33 file after the snapshot 2023-10-30T13:37:30.130 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/34 file after the snapshot 2023-10-30T13:37:30.130 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/35 file after the snapshot 2023-10-30T13:37:30.130 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/36 file after the snapshot 2023-10-30T13:37:30.130 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/37 file after the snapshot 2023-10-30T13:37:30.131 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/38 file after the snapshot 2023-10-30T13:37:30.131 INFO:tasks.workunit.client.0.smithi031.stdout: create 187/39 file after the snapshot 2023-10-30T13:37:30.131 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/.snap/snap-subdir-test 2023-10-30T13:37:30.159 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/30 file after the snapshot 2023-10-30T13:37:30.162 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/31 file after the snapshot 2023-10-30T13:37:30.162 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/32 file after the snapshot 2023-10-30T13:37:30.162 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/33 file after the snapshot 2023-10-30T13:37:30.162 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/34 file after the snapshot 2023-10-30T13:37:30.163 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/35 file after the snapshot 2023-10-30T13:37:30.163 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/36 file after the snapshot 2023-10-30T13:37:30.163 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/37 file after the snapshot 2023-10-30T13:37:30.163 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/38 file after the snapshot 2023-10-30T13:37:30.163 INFO:tasks.workunit.client.0.smithi031.stdout: create 188/39 file after the snapshot 2023-10-30T13:37:30.164 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/.snap/snap-subdir-test 2023-10-30T13:37:30.215 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/30 file after the snapshot 2023-10-30T13:37:30.218 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/31 file after the snapshot 2023-10-30T13:37:30.218 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/32 file after the snapshot 2023-10-30T13:37:30.218 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/33 file after the snapshot 2023-10-30T13:37:30.218 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/34 file after the snapshot 2023-10-30T13:37:30.219 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/35 file after the snapshot 2023-10-30T13:37:30.219 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/36 file after the snapshot 2023-10-30T13:37:30.219 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/37 file after the snapshot 2023-10-30T13:37:30.219 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/38 file after the snapshot 2023-10-30T13:37:30.219 INFO:tasks.workunit.client.0.smithi031.stdout: create 189/39 file after the snapshot 2023-10-30T13:37:30.220 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/.snap/snap-subdir-test 2023-10-30T13:37:30.254 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/30 file after the snapshot 2023-10-30T13:37:30.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/31 file after the snapshot 2023-10-30T13:37:30.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/32 file after the snapshot 2023-10-30T13:37:30.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/33 file after the snapshot 2023-10-30T13:37:30.256 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/34 file after the snapshot 2023-10-30T13:37:30.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/35 file after the snapshot 2023-10-30T13:37:30.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/36 file after the snapshot 2023-10-30T13:37:30.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/37 file after the snapshot 2023-10-30T13:37:30.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/38 file after the snapshot 2023-10-30T13:37:30.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 190/39 file after the snapshot 2023-10-30T13:37:30.257 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/.snap/snap-subdir-test 2023-10-30T13:37:30.291 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/30 file after the snapshot 2023-10-30T13:37:30.294 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/31 file after the snapshot 2023-10-30T13:37:30.294 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/32 file after the snapshot 2023-10-30T13:37:30.295 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/33 file after the snapshot 2023-10-30T13:37:30.295 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/34 file after the snapshot 2023-10-30T13:37:30.295 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/35 file after the snapshot 2023-10-30T13:37:30.295 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/36 file after the snapshot 2023-10-30T13:37:30.295 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/37 file after the snapshot 2023-10-30T13:37:30.295 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/38 file after the snapshot 2023-10-30T13:37:30.296 INFO:tasks.workunit.client.0.smithi031.stdout: create 191/39 file after the snapshot 2023-10-30T13:37:30.296 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/.snap/snap-subdir-test 2023-10-30T13:37:30.352 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/30 file after the snapshot 2023-10-30T13:37:30.355 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/31 file after the snapshot 2023-10-30T13:37:30.356 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/32 file after the snapshot 2023-10-30T13:37:30.356 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/33 file after the snapshot 2023-10-30T13:37:30.356 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/34 file after the snapshot 2023-10-30T13:37:30.356 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/35 file after the snapshot 2023-10-30T13:37:30.356 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/36 file after the snapshot 2023-10-30T13:37:30.357 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/37 file after the snapshot 2023-10-30T13:37:30.357 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/38 file after the snapshot 2023-10-30T13:37:30.357 INFO:tasks.workunit.client.0.smithi031.stdout: create 192/39 file after the snapshot 2023-10-30T13:37:30.357 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/.snap/snap-subdir-test 2023-10-30T13:37:30.383 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/30 file after the snapshot 2023-10-30T13:37:30.384 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/31 file after the snapshot 2023-10-30T13:37:30.384 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/32 file after the snapshot 2023-10-30T13:37:30.385 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/33 file after the snapshot 2023-10-30T13:37:30.385 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/34 file after the snapshot 2023-10-30T13:37:30.385 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/35 file after the snapshot 2023-10-30T13:37:30.385 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/36 file after the snapshot 2023-10-30T13:37:30.385 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/37 file after the snapshot 2023-10-30T13:37:30.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/38 file after the snapshot 2023-10-30T13:37:30.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 193/39 file after the snapshot 2023-10-30T13:37:30.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/.snap/snap-subdir-test 2023-10-30T13:37:30.663 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/30 file after the snapshot 2023-10-30T13:37:30.664 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/31 file after the snapshot 2023-10-30T13:37:30.664 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/32 file after the snapshot 2023-10-30T13:37:30.665 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/33 file after the snapshot 2023-10-30T13:37:30.665 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/34 file after the snapshot 2023-10-30T13:37:30.665 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/35 file after the snapshot 2023-10-30T13:37:30.665 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/36 file after the snapshot 2023-10-30T13:37:30.665 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/37 file after the snapshot 2023-10-30T13:37:30.665 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/38 file after the snapshot 2023-10-30T13:37:30.666 INFO:tasks.workunit.client.0.smithi031.stdout: create 194/39 file after the snapshot 2023-10-30T13:37:30.666 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/.snap/snap-subdir-test 2023-10-30T13:37:30.692 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/30 file after the snapshot 2023-10-30T13:37:30.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/31 file after the snapshot 2023-10-30T13:37:30.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/32 file after the snapshot 2023-10-30T13:37:30.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/33 file after the snapshot 2023-10-30T13:37:30.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/34 file after the snapshot 2023-10-30T13:37:30.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/35 file after the snapshot 2023-10-30T13:37:30.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/36 file after the snapshot 2023-10-30T13:37:30.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/37 file after the snapshot 2023-10-30T13:37:30.696 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/38 file after the snapshot 2023-10-30T13:37:30.696 INFO:tasks.workunit.client.0.smithi031.stdout: create 195/39 file after the snapshot 2023-10-30T13:37:30.696 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/.snap/snap-subdir-test 2023-10-30T13:37:30.719 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/30 file after the snapshot 2023-10-30T13:37:30.722 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/31 file after the snapshot 2023-10-30T13:37:30.722 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/32 file after the snapshot 2023-10-30T13:37:30.722 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/33 file after the snapshot 2023-10-30T13:37:30.722 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/34 file after the snapshot 2023-10-30T13:37:30.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/35 file after the snapshot 2023-10-30T13:37:30.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/36 file after the snapshot 2023-10-30T13:37:30.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/37 file after the snapshot 2023-10-30T13:37:30.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/38 file after the snapshot 2023-10-30T13:37:30.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 196/39 file after the snapshot 2023-10-30T13:37:30.723 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/.snap/snap-subdir-test 2023-10-30T13:37:30.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/30 file after the snapshot 2023-10-30T13:37:30.780 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/31 file after the snapshot 2023-10-30T13:37:30.780 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/32 file after the snapshot 2023-10-30T13:37:30.780 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/33 file after the snapshot 2023-10-30T13:37:30.780 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/34 file after the snapshot 2023-10-30T13:37:30.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/35 file after the snapshot 2023-10-30T13:37:30.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/36 file after the snapshot 2023-10-30T13:37:30.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/37 file after the snapshot 2023-10-30T13:37:30.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/38 file after the snapshot 2023-10-30T13:37:30.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 197/39 file after the snapshot 2023-10-30T13:37:30.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/.snap/snap-subdir-test 2023-10-30T13:37:30.814 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/30 file after the snapshot 2023-10-30T13:37:30.815 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/31 file after the snapshot 2023-10-30T13:37:30.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/32 file after the snapshot 2023-10-30T13:37:30.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/33 file after the snapshot 2023-10-30T13:37:30.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/34 file after the snapshot 2023-10-30T13:37:30.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/35 file after the snapshot 2023-10-30T13:37:30.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/36 file after the snapshot 2023-10-30T13:37:30.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/37 file after the snapshot 2023-10-30T13:37:30.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/38 file after the snapshot 2023-10-30T13:37:30.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 198/39 file after the snapshot 2023-10-30T13:37:30.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/.snap/snap-subdir-test 2023-10-30T13:37:30.844 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/30 file after the snapshot 2023-10-30T13:37:30.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/31 file after the snapshot 2023-10-30T13:37:30.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/32 file after the snapshot 2023-10-30T13:37:30.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/33 file after the snapshot 2023-10-30T13:37:30.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/34 file after the snapshot 2023-10-30T13:37:30.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/35 file after the snapshot 2023-10-30T13:37:30.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/36 file after the snapshot 2023-10-30T13:37:30.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/37 file after the snapshot 2023-10-30T13:37:30.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/38 file after the snapshot 2023-10-30T13:37:30.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 199/39 file after the snapshot 2023-10-30T13:37:30.849 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/.snap/snap-subdir-test 2023-10-30T13:37:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/30 file after the snapshot 2023-10-30T13:37:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/31 file after the snapshot 2023-10-30T13:37:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/32 file after the snapshot 2023-10-30T13:37:30.885 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/33 file after the snapshot 2023-10-30T13:37:30.885 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/34 file after the snapshot 2023-10-30T13:37:30.885 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/35 file after the snapshot 2023-10-30T13:37:30.885 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/36 file after the snapshot 2023-10-30T13:37:30.885 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/37 file after the snapshot 2023-10-30T13:37:30.885 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/38 file after the snapshot 2023-10-30T13:37:30.886 INFO:tasks.workunit.client.0.smithi031.stdout: create 200/39 file after the snapshot 2023-10-30T13:37:30.886 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/.snap/snap-subdir-test 2023-10-30T13:37:30.916 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/30 file after the snapshot 2023-10-30T13:37:30.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/31 file after the snapshot 2023-10-30T13:37:30.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/32 file after the snapshot 2023-10-30T13:37:30.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/33 file after the snapshot 2023-10-30T13:37:30.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/34 file after the snapshot 2023-10-30T13:37:30.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/35 file after the snapshot 2023-10-30T13:37:30.920 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/36 file after the snapshot 2023-10-30T13:37:30.920 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/37 file after the snapshot 2023-10-30T13:37:30.920 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/38 file after the snapshot 2023-10-30T13:37:30.920 INFO:tasks.workunit.client.0.smithi031.stdout: create 201/39 file after the snapshot 2023-10-30T13:37:30.920 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/.snap/snap-subdir-test 2023-10-30T13:37:30.954 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/30 file after the snapshot 2023-10-30T13:37:30.956 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/31 file after the snapshot 2023-10-30T13:37:30.956 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/32 file after the snapshot 2023-10-30T13:37:30.956 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/33 file after the snapshot 2023-10-30T13:37:30.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/34 file after the snapshot 2023-10-30T13:37:30.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/35 file after the snapshot 2023-10-30T13:37:30.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/36 file after the snapshot 2023-10-30T13:37:30.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/37 file after the snapshot 2023-10-30T13:37:30.957 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/38 file after the snapshot 2023-10-30T13:37:30.958 INFO:tasks.workunit.client.0.smithi031.stdout: create 202/39 file after the snapshot 2023-10-30T13:37:30.958 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/.snap/snap-subdir-test 2023-10-30T13:37:31.490 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/30 file after the snapshot 2023-10-30T13:37:31.492 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/31 file after the snapshot 2023-10-30T13:37:31.492 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/32 file after the snapshot 2023-10-30T13:37:31.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/33 file after the snapshot 2023-10-30T13:37:31.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/34 file after the snapshot 2023-10-30T13:37:31.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/35 file after the snapshot 2023-10-30T13:37:31.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/36 file after the snapshot 2023-10-30T13:37:31.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/37 file after the snapshot 2023-10-30T13:37:31.494 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/38 file after the snapshot 2023-10-30T13:37:31.494 INFO:tasks.workunit.client.0.smithi031.stdout: create 203/39 file after the snapshot 2023-10-30T13:37:31.494 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/.snap/snap-subdir-test 2023-10-30T13:37:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/30 file after the snapshot 2023-10-30T13:37:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/31 file after the snapshot 2023-10-30T13:37:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/32 file after the snapshot 2023-10-30T13:37:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/33 file after the snapshot 2023-10-30T13:37:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/34 file after the snapshot 2023-10-30T13:37:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/35 file after the snapshot 2023-10-30T13:37:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/36 file after the snapshot 2023-10-30T13:37:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/37 file after the snapshot 2023-10-30T13:37:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/38 file after the snapshot 2023-10-30T13:37:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 204/39 file after the snapshot 2023-10-30T13:37:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/.snap/snap-subdir-test 2023-10-30T13:37:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/30 file after the snapshot 2023-10-30T13:37:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/31 file after the snapshot 2023-10-30T13:37:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/32 file after the snapshot 2023-10-30T13:37:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/33 file after the snapshot 2023-10-30T13:37:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/34 file after the snapshot 2023-10-30T13:37:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/35 file after the snapshot 2023-10-30T13:37:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/36 file after the snapshot 2023-10-30T13:37:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/37 file after the snapshot 2023-10-30T13:37:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/38 file after the snapshot 2023-10-30T13:37:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 205/39 file after the snapshot 2023-10-30T13:37:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/.snap/snap-subdir-test 2023-10-30T13:37:31.621 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/30 file after the snapshot 2023-10-30T13:37:31.624 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/31 file after the snapshot 2023-10-30T13:37:31.624 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/32 file after the snapshot 2023-10-30T13:37:31.624 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/33 file after the snapshot 2023-10-30T13:37:31.625 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/34 file after the snapshot 2023-10-30T13:37:31.625 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/35 file after the snapshot 2023-10-30T13:37:31.625 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/36 file after the snapshot 2023-10-30T13:37:31.626 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/37 file after the snapshot 2023-10-30T13:37:31.626 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/38 file after the snapshot 2023-10-30T13:37:31.626 INFO:tasks.workunit.client.0.smithi031.stdout: create 206/39 file after the snapshot 2023-10-30T13:37:31.626 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/.snap/snap-subdir-test 2023-10-30T13:37:31.659 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/30 file after the snapshot 2023-10-30T13:37:31.661 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/31 file after the snapshot 2023-10-30T13:37:31.661 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/32 file after the snapshot 2023-10-30T13:37:31.661 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/33 file after the snapshot 2023-10-30T13:37:31.661 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/34 file after the snapshot 2023-10-30T13:37:31.662 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/35 file after the snapshot 2023-10-30T13:37:31.662 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/36 file after the snapshot 2023-10-30T13:37:31.662 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/37 file after the snapshot 2023-10-30T13:37:31.662 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/38 file after the snapshot 2023-10-30T13:37:31.662 INFO:tasks.workunit.client.0.smithi031.stdout: create 207/39 file after the snapshot 2023-10-30T13:37:31.663 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/.snap/snap-subdir-test 2023-10-30T13:37:31.690 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/30 file after the snapshot 2023-10-30T13:37:31.693 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/31 file after the snapshot 2023-10-30T13:37:31.693 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/32 file after the snapshot 2023-10-30T13:37:31.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/33 file after the snapshot 2023-10-30T13:37:31.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/34 file after the snapshot 2023-10-30T13:37:31.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/35 file after the snapshot 2023-10-30T13:37:31.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/36 file after the snapshot 2023-10-30T13:37:31.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/37 file after the snapshot 2023-10-30T13:37:31.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/38 file after the snapshot 2023-10-30T13:37:31.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 208/39 file after the snapshot 2023-10-30T13:37:31.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/.snap/snap-subdir-test 2023-10-30T13:37:31.742 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/30 file after the snapshot 2023-10-30T13:37:31.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/31 file after the snapshot 2023-10-30T13:37:31.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/32 file after the snapshot 2023-10-30T13:37:31.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/33 file after the snapshot 2023-10-30T13:37:31.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/34 file after the snapshot 2023-10-30T13:37:31.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/35 file after the snapshot 2023-10-30T13:37:31.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/36 file after the snapshot 2023-10-30T13:37:31.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/37 file after the snapshot 2023-10-30T13:37:31.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/38 file after the snapshot 2023-10-30T13:37:31.747 INFO:tasks.workunit.client.0.smithi031.stdout: create 209/39 file after the snapshot 2023-10-30T13:37:31.747 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/.snap/snap-subdir-test 2023-10-30T13:37:31.795 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/30 file after the snapshot 2023-10-30T13:37:31.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/31 file after the snapshot 2023-10-30T13:37:31.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/32 file after the snapshot 2023-10-30T13:37:31.800 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/33 file after the snapshot 2023-10-30T13:37:31.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/34 file after the snapshot 2023-10-30T13:37:31.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/35 file after the snapshot 2023-10-30T13:37:31.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/36 file after the snapshot 2023-10-30T13:37:31.801 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/37 file after the snapshot 2023-10-30T13:37:31.802 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/38 file after the snapshot 2023-10-30T13:37:31.802 INFO:tasks.workunit.client.0.smithi031.stdout: create 210/39 file after the snapshot 2023-10-30T13:37:31.802 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/.snap/snap-subdir-test 2023-10-30T13:37:32.234 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/30 file after the snapshot 2023-10-30T13:37:32.237 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/31 file after the snapshot 2023-10-30T13:37:32.237 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/32 file after the snapshot 2023-10-30T13:37:32.237 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/33 file after the snapshot 2023-10-30T13:37:32.238 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/34 file after the snapshot 2023-10-30T13:37:32.238 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/35 file after the snapshot 2023-10-30T13:37:32.238 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/36 file after the snapshot 2023-10-30T13:37:32.238 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/37 file after the snapshot 2023-10-30T13:37:32.238 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/38 file after the snapshot 2023-10-30T13:37:32.239 INFO:tasks.workunit.client.0.smithi031.stdout: create 211/39 file after the snapshot 2023-10-30T13:37:32.239 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/.snap/snap-subdir-test 2023-10-30T13:37:32.275 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/30 file after the snapshot 2023-10-30T13:37:32.277 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/31 file after the snapshot 2023-10-30T13:37:32.277 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/32 file after the snapshot 2023-10-30T13:37:32.277 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/33 file after the snapshot 2023-10-30T13:37:32.278 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/34 file after the snapshot 2023-10-30T13:37:32.278 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/35 file after the snapshot 2023-10-30T13:37:32.278 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/36 file after the snapshot 2023-10-30T13:37:32.278 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/37 file after the snapshot 2023-10-30T13:37:32.278 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/38 file after the snapshot 2023-10-30T13:37:32.279 INFO:tasks.workunit.client.0.smithi031.stdout: create 212/39 file after the snapshot 2023-10-30T13:37:32.279 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/.snap/snap-subdir-test 2023-10-30T13:37:32.316 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/30 file after the snapshot 2023-10-30T13:37:32.318 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/31 file after the snapshot 2023-10-30T13:37:32.319 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/32 file after the snapshot 2023-10-30T13:37:32.319 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/33 file after the snapshot 2023-10-30T13:37:32.319 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/34 file after the snapshot 2023-10-30T13:37:32.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/35 file after the snapshot 2023-10-30T13:37:32.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/36 file after the snapshot 2023-10-30T13:37:32.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/37 file after the snapshot 2023-10-30T13:37:32.321 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/38 file after the snapshot 2023-10-30T13:37:32.321 INFO:tasks.workunit.client.0.smithi031.stdout: create 213/39 file after the snapshot 2023-10-30T13:37:32.321 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/.snap/snap-subdir-test 2023-10-30T13:37:32.727 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/30 file after the snapshot 2023-10-30T13:37:32.730 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/31 file after the snapshot 2023-10-30T13:37:32.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/32 file after the snapshot 2023-10-30T13:37:32.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/33 file after the snapshot 2023-10-30T13:37:32.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/34 file after the snapshot 2023-10-30T13:37:32.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/35 file after the snapshot 2023-10-30T13:37:32.731 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/36 file after the snapshot 2023-10-30T13:37:32.732 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/37 file after the snapshot 2023-10-30T13:37:32.732 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/38 file after the snapshot 2023-10-30T13:37:32.732 INFO:tasks.workunit.client.0.smithi031.stdout: create 214/39 file after the snapshot 2023-10-30T13:37:32.732 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/.snap/snap-subdir-test 2023-10-30T13:37:32.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/30 file after the snapshot 2023-10-30T13:37:32.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/31 file after the snapshot 2023-10-30T13:37:32.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/32 file after the snapshot 2023-10-30T13:37:32.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/33 file after the snapshot 2023-10-30T13:37:32.782 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/34 file after the snapshot 2023-10-30T13:37:32.782 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/35 file after the snapshot 2023-10-30T13:37:32.782 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/36 file after the snapshot 2023-10-30T13:37:32.782 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/37 file after the snapshot 2023-10-30T13:37:32.782 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/38 file after the snapshot 2023-10-30T13:37:32.783 INFO:tasks.workunit.client.0.smithi031.stdout: create 215/39 file after the snapshot 2023-10-30T13:37:32.783 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/.snap/snap-subdir-test 2023-10-30T13:37:32.812 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/30 file after the snapshot 2023-10-30T13:37:32.815 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/31 file after the snapshot 2023-10-30T13:37:32.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/32 file after the snapshot 2023-10-30T13:37:32.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/33 file after the snapshot 2023-10-30T13:37:32.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/34 file after the snapshot 2023-10-30T13:37:32.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/35 file after the snapshot 2023-10-30T13:37:32.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/36 file after the snapshot 2023-10-30T13:37:32.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/37 file after the snapshot 2023-10-30T13:37:32.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/38 file after the snapshot 2023-10-30T13:37:32.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 216/39 file after the snapshot 2023-10-30T13:37:32.817 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/.snap/snap-subdir-test 2023-10-30T13:37:32.865 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/30 file after the snapshot 2023-10-30T13:37:32.870 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/31 file after the snapshot 2023-10-30T13:37:32.870 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/32 file after the snapshot 2023-10-30T13:37:32.870 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/33 file after the snapshot 2023-10-30T13:37:32.871 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/34 file after the snapshot 2023-10-30T13:37:32.871 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/35 file after the snapshot 2023-10-30T13:37:32.871 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/36 file after the snapshot 2023-10-30T13:37:32.871 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/37 file after the snapshot 2023-10-30T13:37:32.871 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/38 file after the snapshot 2023-10-30T13:37:32.872 INFO:tasks.workunit.client.0.smithi031.stdout: create 217/39 file after the snapshot 2023-10-30T13:37:32.872 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/.snap/snap-subdir-test 2023-10-30T13:37:32.933 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/30 file after the snapshot 2023-10-30T13:37:32.936 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/31 file after the snapshot 2023-10-30T13:37:32.936 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/32 file after the snapshot 2023-10-30T13:37:32.936 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/33 file after the snapshot 2023-10-30T13:37:32.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/34 file after the snapshot 2023-10-30T13:37:32.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/35 file after the snapshot 2023-10-30T13:37:32.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/36 file after the snapshot 2023-10-30T13:37:32.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/37 file after the snapshot 2023-10-30T13:37:32.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/38 file after the snapshot 2023-10-30T13:37:32.938 INFO:tasks.workunit.client.0.smithi031.stdout: create 218/39 file after the snapshot 2023-10-30T13:37:32.938 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/.snap/snap-subdir-test 2023-10-30T13:37:33.537 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/30 file after the snapshot 2023-10-30T13:37:33.538 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/31 file after the snapshot 2023-10-30T13:37:33.538 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/32 file after the snapshot 2023-10-30T13:37:33.538 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/33 file after the snapshot 2023-10-30T13:37:33.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/34 file after the snapshot 2023-10-30T13:37:33.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/35 file after the snapshot 2023-10-30T13:37:33.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/36 file after the snapshot 2023-10-30T13:37:33.540 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/37 file after the snapshot 2023-10-30T13:37:33.540 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/38 file after the snapshot 2023-10-30T13:37:33.540 INFO:tasks.workunit.client.0.smithi031.stdout: create 219/39 file after the snapshot 2023-10-30T13:37:33.540 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/.snap/snap-subdir-test 2023-10-30T13:37:33.565 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/30 file after the snapshot 2023-10-30T13:37:33.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/31 file after the snapshot 2023-10-30T13:37:33.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/32 file after the snapshot 2023-10-30T13:37:33.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/33 file after the snapshot 2023-10-30T13:37:33.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/34 file after the snapshot 2023-10-30T13:37:33.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/35 file after the snapshot 2023-10-30T13:37:33.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/36 file after the snapshot 2023-10-30T13:37:33.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/37 file after the snapshot 2023-10-30T13:37:33.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/38 file after the snapshot 2023-10-30T13:37:33.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 220/39 file after the snapshot 2023-10-30T13:37:33.569 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/.snap/snap-subdir-test 2023-10-30T13:37:33.590 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/30 file after the snapshot 2023-10-30T13:37:33.593 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/31 file after the snapshot 2023-10-30T13:37:33.594 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/32 file after the snapshot 2023-10-30T13:37:33.594 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/33 file after the snapshot 2023-10-30T13:37:33.594 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/34 file after the snapshot 2023-10-30T13:37:33.594 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/35 file after the snapshot 2023-10-30T13:37:33.594 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/36 file after the snapshot 2023-10-30T13:37:33.595 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/37 file after the snapshot 2023-10-30T13:37:33.595 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/38 file after the snapshot 2023-10-30T13:37:33.595 INFO:tasks.workunit.client.0.smithi031.stdout: create 221/39 file after the snapshot 2023-10-30T13:37:33.595 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/.snap/snap-subdir-test 2023-10-30T13:37:33.630 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/30 file after the snapshot 2023-10-30T13:37:33.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/31 file after the snapshot 2023-10-30T13:37:33.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/32 file after the snapshot 2023-10-30T13:37:33.632 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/33 file after the snapshot 2023-10-30T13:37:33.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/34 file after the snapshot 2023-10-30T13:37:33.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/35 file after the snapshot 2023-10-30T13:37:33.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/36 file after the snapshot 2023-10-30T13:37:33.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/37 file after the snapshot 2023-10-30T13:37:33.633 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/38 file after the snapshot 2023-10-30T13:37:33.634 INFO:tasks.workunit.client.0.smithi031.stdout: create 222/39 file after the snapshot 2023-10-30T13:37:33.634 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/.snap/snap-subdir-test 2023-10-30T13:37:33.669 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/30 file after the snapshot 2023-10-30T13:37:33.670 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/31 file after the snapshot 2023-10-30T13:37:33.670 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/32 file after the snapshot 2023-10-30T13:37:33.671 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/33 file after the snapshot 2023-10-30T13:37:33.671 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/34 file after the snapshot 2023-10-30T13:37:33.671 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/35 file after the snapshot 2023-10-30T13:37:33.671 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/36 file after the snapshot 2023-10-30T13:37:33.671 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/37 file after the snapshot 2023-10-30T13:37:33.672 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/38 file after the snapshot 2023-10-30T13:37:33.672 INFO:tasks.workunit.client.0.smithi031.stdout: create 223/39 file after the snapshot 2023-10-30T13:37:33.672 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/.snap/snap-subdir-test 2023-10-30T13:37:33.691 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/30 file after the snapshot 2023-10-30T13:37:33.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/31 file after the snapshot 2023-10-30T13:37:33.694 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/32 file after the snapshot 2023-10-30T13:37:33.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/33 file after the snapshot 2023-10-30T13:37:33.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/34 file after the snapshot 2023-10-30T13:37:33.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/35 file after the snapshot 2023-10-30T13:37:33.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/36 file after the snapshot 2023-10-30T13:37:33.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/37 file after the snapshot 2023-10-30T13:37:33.695 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/38 file after the snapshot 2023-10-30T13:37:33.696 INFO:tasks.workunit.client.0.smithi031.stdout: create 224/39 file after the snapshot 2023-10-30T13:37:33.696 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/.snap/snap-subdir-test 2023-10-30T13:37:33.740 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/30 file after the snapshot 2023-10-30T13:37:33.743 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/31 file after the snapshot 2023-10-30T13:37:33.743 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/32 file after the snapshot 2023-10-30T13:37:33.744 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/33 file after the snapshot 2023-10-30T13:37:33.744 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/34 file after the snapshot 2023-10-30T13:37:33.744 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/35 file after the snapshot 2023-10-30T13:37:33.744 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/36 file after the snapshot 2023-10-30T13:37:33.744 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/37 file after the snapshot 2023-10-30T13:37:33.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/38 file after the snapshot 2023-10-30T13:37:33.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 225/39 file after the snapshot 2023-10-30T13:37:33.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/.snap/snap-subdir-test 2023-10-30T13:37:33.774 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/30 file after the snapshot 2023-10-30T13:37:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/31 file after the snapshot 2023-10-30T13:37:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/32 file after the snapshot 2023-10-30T13:37:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/33 file after the snapshot 2023-10-30T13:37:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/34 file after the snapshot 2023-10-30T13:37:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/35 file after the snapshot 2023-10-30T13:37:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/36 file after the snapshot 2023-10-30T13:37:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/37 file after the snapshot 2023-10-30T13:37:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/38 file after the snapshot 2023-10-30T13:37:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: create 226/39 file after the snapshot 2023-10-30T13:37:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/.snap/snap-subdir-test 2023-10-30T13:37:33.825 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/30 file after the snapshot 2023-10-30T13:37:33.828 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/31 file after the snapshot 2023-10-30T13:37:33.828 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/32 file after the snapshot 2023-10-30T13:37:33.828 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/33 file after the snapshot 2023-10-30T13:37:33.828 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/34 file after the snapshot 2023-10-30T13:37:33.828 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/35 file after the snapshot 2023-10-30T13:37:33.829 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/36 file after the snapshot 2023-10-30T13:37:33.829 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/37 file after the snapshot 2023-10-30T13:37:33.829 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/38 file after the snapshot 2023-10-30T13:37:33.829 INFO:tasks.workunit.client.0.smithi031.stdout: create 227/39 file after the snapshot 2023-10-30T13:37:33.829 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/.snap/snap-subdir-test 2023-10-30T13:37:33.860 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/30 file after the snapshot 2023-10-30T13:37:33.862 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/31 file after the snapshot 2023-10-30T13:37:33.862 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/32 file after the snapshot 2023-10-30T13:37:33.862 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/33 file after the snapshot 2023-10-30T13:37:33.862 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/34 file after the snapshot 2023-10-30T13:37:33.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/35 file after the snapshot 2023-10-30T13:37:33.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/36 file after the snapshot 2023-10-30T13:37:33.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/37 file after the snapshot 2023-10-30T13:37:33.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/38 file after the snapshot 2023-10-30T13:37:33.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 228/39 file after the snapshot 2023-10-30T13:37:33.863 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/.snap/snap-subdir-test 2023-10-30T13:37:33.888 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/30 file after the snapshot 2023-10-30T13:37:33.891 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/31 file after the snapshot 2023-10-30T13:37:33.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/32 file after the snapshot 2023-10-30T13:37:33.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/33 file after the snapshot 2023-10-30T13:37:33.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/34 file after the snapshot 2023-10-30T13:37:33.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/35 file after the snapshot 2023-10-30T13:37:33.892 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/36 file after the snapshot 2023-10-30T13:37:33.893 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/37 file after the snapshot 2023-10-30T13:37:33.893 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/38 file after the snapshot 2023-10-30T13:37:33.893 INFO:tasks.workunit.client.0.smithi031.stdout: create 229/39 file after the snapshot 2023-10-30T13:37:33.893 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/.snap/snap-subdir-test 2023-10-30T13:37:33.916 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/30 file after the snapshot 2023-10-30T13:37:33.917 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/31 file after the snapshot 2023-10-30T13:37:33.918 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/32 file after the snapshot 2023-10-30T13:37:33.918 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/33 file after the snapshot 2023-10-30T13:37:33.918 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/34 file after the snapshot 2023-10-30T13:37:33.918 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/35 file after the snapshot 2023-10-30T13:37:33.918 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/36 file after the snapshot 2023-10-30T13:37:33.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/37 file after the snapshot 2023-10-30T13:37:33.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/38 file after the snapshot 2023-10-30T13:37:33.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 230/39 file after the snapshot 2023-10-30T13:37:33.919 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/.snap/snap-subdir-test 2023-10-30T13:37:33.938 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/30 file after the snapshot 2023-10-30T13:37:33.941 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/31 file after the snapshot 2023-10-30T13:37:33.941 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/32 file after the snapshot 2023-10-30T13:37:33.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/33 file after the snapshot 2023-10-30T13:37:33.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/34 file after the snapshot 2023-10-30T13:37:33.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/35 file after the snapshot 2023-10-30T13:37:33.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/36 file after the snapshot 2023-10-30T13:37:33.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/37 file after the snapshot 2023-10-30T13:37:33.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/38 file after the snapshot 2023-10-30T13:37:33.943 INFO:tasks.workunit.client.0.smithi031.stdout: create 231/39 file after the snapshot 2023-10-30T13:37:33.943 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/.snap/snap-subdir-test 2023-10-30T13:37:33.975 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/30 file after the snapshot 2023-10-30T13:37:33.977 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/31 file after the snapshot 2023-10-30T13:37:33.977 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/32 file after the snapshot 2023-10-30T13:37:33.978 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/33 file after the snapshot 2023-10-30T13:37:33.978 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/34 file after the snapshot 2023-10-30T13:37:33.978 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/35 file after the snapshot 2023-10-30T13:37:33.978 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/36 file after the snapshot 2023-10-30T13:37:33.978 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/37 file after the snapshot 2023-10-30T13:37:33.978 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/38 file after the snapshot 2023-10-30T13:37:33.979 INFO:tasks.workunit.client.0.smithi031.stdout: create 232/39 file after the snapshot 2023-10-30T13:37:33.979 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/.snap/snap-subdir-test 2023-10-30T13:37:34.016 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/30 file after the snapshot 2023-10-30T13:37:34.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/31 file after the snapshot 2023-10-30T13:37:34.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/32 file after the snapshot 2023-10-30T13:37:34.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/33 file after the snapshot 2023-10-30T13:37:34.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/34 file after the snapshot 2023-10-30T13:37:34.019 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/35 file after the snapshot 2023-10-30T13:37:34.020 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/36 file after the snapshot 2023-10-30T13:37:34.020 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/37 file after the snapshot 2023-10-30T13:37:34.020 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/38 file after the snapshot 2023-10-30T13:37:34.020 INFO:tasks.workunit.client.0.smithi031.stdout: create 233/39 file after the snapshot 2023-10-30T13:37:34.020 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/.snap/snap-subdir-test 2023-10-30T13:37:34.048 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/30 file after the snapshot 2023-10-30T13:37:34.051 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/31 file after the snapshot 2023-10-30T13:37:34.052 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/32 file after the snapshot 2023-10-30T13:37:34.052 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/33 file after the snapshot 2023-10-30T13:37:34.052 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/34 file after the snapshot 2023-10-30T13:37:34.052 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/35 file after the snapshot 2023-10-30T13:37:34.052 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/36 file after the snapshot 2023-10-30T13:37:34.053 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/37 file after the snapshot 2023-10-30T13:37:34.053 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/38 file after the snapshot 2023-10-30T13:37:34.053 INFO:tasks.workunit.client.0.smithi031.stdout: create 234/39 file after the snapshot 2023-10-30T13:37:34.053 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/.snap/snap-subdir-test 2023-10-30T13:37:34.076 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/30 file after the snapshot 2023-10-30T13:37:34.077 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/31 file after the snapshot 2023-10-30T13:37:34.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/32 file after the snapshot 2023-10-30T13:37:34.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/33 file after the snapshot 2023-10-30T13:37:34.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/34 file after the snapshot 2023-10-30T13:37:34.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/35 file after the snapshot 2023-10-30T13:37:34.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/36 file after the snapshot 2023-10-30T13:37:34.079 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/37 file after the snapshot 2023-10-30T13:37:34.079 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/38 file after the snapshot 2023-10-30T13:37:34.079 INFO:tasks.workunit.client.0.smithi031.stdout: create 235/39 file after the snapshot 2023-10-30T13:37:34.079 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/.snap/snap-subdir-test 2023-10-30T13:37:34.098 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/30 file after the snapshot 2023-10-30T13:37:34.101 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/31 file after the snapshot 2023-10-30T13:37:34.101 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/32 file after the snapshot 2023-10-30T13:37:34.101 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/33 file after the snapshot 2023-10-30T13:37:34.101 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/34 file after the snapshot 2023-10-30T13:37:34.101 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/35 file after the snapshot 2023-10-30T13:37:34.102 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/36 file after the snapshot 2023-10-30T13:37:34.102 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/37 file after the snapshot 2023-10-30T13:37:34.103 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/38 file after the snapshot 2023-10-30T13:37:34.103 INFO:tasks.workunit.client.0.smithi031.stdout: create 236/39 file after the snapshot 2023-10-30T13:37:34.103 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/.snap/snap-subdir-test 2023-10-30T13:37:34.460 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/30 file after the snapshot 2023-10-30T13:37:34.463 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/31 file after the snapshot 2023-10-30T13:37:34.463 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/32 file after the snapshot 2023-10-30T13:37:34.463 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/33 file after the snapshot 2023-10-30T13:37:34.463 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/34 file after the snapshot 2023-10-30T13:37:34.463 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/35 file after the snapshot 2023-10-30T13:37:34.463 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/36 file after the snapshot 2023-10-30T13:37:34.464 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/37 file after the snapshot 2023-10-30T13:37:34.464 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/38 file after the snapshot 2023-10-30T13:37:34.464 INFO:tasks.workunit.client.0.smithi031.stdout: create 237/39 file after the snapshot 2023-10-30T13:37:34.464 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/.snap/snap-subdir-test 2023-10-30T13:37:34.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/30 file after the snapshot 2023-10-30T13:37:34.496 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/31 file after the snapshot 2023-10-30T13:37:34.496 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/32 file after the snapshot 2023-10-30T13:37:34.497 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/33 file after the snapshot 2023-10-30T13:37:34.497 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/34 file after the snapshot 2023-10-30T13:37:34.497 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/35 file after the snapshot 2023-10-30T13:37:34.497 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/36 file after the snapshot 2023-10-30T13:37:34.497 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/37 file after the snapshot 2023-10-30T13:37:34.498 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/38 file after the snapshot 2023-10-30T13:37:34.498 INFO:tasks.workunit.client.0.smithi031.stdout: create 238/39 file after the snapshot 2023-10-30T13:37:34.498 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/.snap/snap-subdir-test 2023-10-30T13:37:34.520 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/30 file after the snapshot 2023-10-30T13:37:34.523 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/31 file after the snapshot 2023-10-30T13:37:34.524 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/32 file after the snapshot 2023-10-30T13:37:34.524 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/33 file after the snapshot 2023-10-30T13:37:34.524 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/34 file after the snapshot 2023-10-30T13:37:34.524 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/35 file after the snapshot 2023-10-30T13:37:34.524 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/36 file after the snapshot 2023-10-30T13:37:34.525 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/37 file after the snapshot 2023-10-30T13:37:34.525 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/38 file after the snapshot 2023-10-30T13:37:34.525 INFO:tasks.workunit.client.0.smithi031.stdout: create 239/39 file after the snapshot 2023-10-30T13:37:34.525 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/.snap/snap-subdir-test 2023-10-30T13:37:34.570 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/30 file after the snapshot 2023-10-30T13:37:34.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/31 file after the snapshot 2023-10-30T13:37:34.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/32 file after the snapshot 2023-10-30T13:37:34.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/33 file after the snapshot 2023-10-30T13:37:34.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/34 file after the snapshot 2023-10-30T13:37:34.573 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/35 file after the snapshot 2023-10-30T13:37:34.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/36 file after the snapshot 2023-10-30T13:37:34.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/37 file after the snapshot 2023-10-30T13:37:34.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/38 file after the snapshot 2023-10-30T13:37:34.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 240/39 file after the snapshot 2023-10-30T13:37:34.574 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/.snap/snap-subdir-test 2023-10-30T13:37:35.307 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/30 file after the snapshot 2023-10-30T13:37:35.310 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/31 file after the snapshot 2023-10-30T13:37:35.310 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/32 file after the snapshot 2023-10-30T13:37:35.311 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/33 file after the snapshot 2023-10-30T13:37:35.311 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/34 file after the snapshot 2023-10-30T13:37:35.311 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/35 file after the snapshot 2023-10-30T13:37:35.311 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/36 file after the snapshot 2023-10-30T13:37:35.311 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/37 file after the snapshot 2023-10-30T13:37:35.312 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/38 file after the snapshot 2023-10-30T13:37:35.312 INFO:tasks.workunit.client.0.smithi031.stdout: create 241/39 file after the snapshot 2023-10-30T13:37:35.312 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/.snap/snap-subdir-test 2023-10-30T13:37:35.608 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/30 file after the snapshot 2023-10-30T13:37:35.611 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/31 file after the snapshot 2023-10-30T13:37:35.611 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/32 file after the snapshot 2023-10-30T13:37:35.611 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/33 file after the snapshot 2023-10-30T13:37:35.611 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/34 file after the snapshot 2023-10-30T13:37:35.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/35 file after the snapshot 2023-10-30T13:37:35.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/36 file after the snapshot 2023-10-30T13:37:35.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/37 file after the snapshot 2023-10-30T13:37:35.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/38 file after the snapshot 2023-10-30T13:37:35.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 242/39 file after the snapshot 2023-10-30T13:37:35.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/.snap/snap-subdir-test 2023-10-30T13:37:36.487 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/30 file after the snapshot 2023-10-30T13:37:36.492 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/31 file after the snapshot 2023-10-30T13:37:36.492 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/32 file after the snapshot 2023-10-30T13:37:36.492 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/33 file after the snapshot 2023-10-30T13:37:36.492 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/34 file after the snapshot 2023-10-30T13:37:36.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/35 file after the snapshot 2023-10-30T13:37:36.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/36 file after the snapshot 2023-10-30T13:37:36.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/37 file after the snapshot 2023-10-30T13:37:36.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/38 file after the snapshot 2023-10-30T13:37:36.493 INFO:tasks.workunit.client.0.smithi031.stdout: create 243/39 file after the snapshot 2023-10-30T13:37:36.494 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/.snap/snap-subdir-test 2023-10-30T13:37:36.522 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/30 file after the snapshot 2023-10-30T13:37:36.525 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/31 file after the snapshot 2023-10-30T13:37:36.526 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/32 file after the snapshot 2023-10-30T13:37:36.526 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/33 file after the snapshot 2023-10-30T13:37:36.526 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/34 file after the snapshot 2023-10-30T13:37:36.526 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/35 file after the snapshot 2023-10-30T13:37:36.526 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/36 file after the snapshot 2023-10-30T13:37:36.527 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/37 file after the snapshot 2023-10-30T13:37:36.527 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/38 file after the snapshot 2023-10-30T13:37:36.527 INFO:tasks.workunit.client.0.smithi031.stdout: create 244/39 file after the snapshot 2023-10-30T13:37:36.527 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/.snap/snap-subdir-test 2023-10-30T13:37:36.565 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/30 file after the snapshot 2023-10-30T13:37:36.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/31 file after the snapshot 2023-10-30T13:37:36.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/32 file after the snapshot 2023-10-30T13:37:36.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/33 file after the snapshot 2023-10-30T13:37:36.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/34 file after the snapshot 2023-10-30T13:37:36.567 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/35 file after the snapshot 2023-10-30T13:37:36.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/36 file after the snapshot 2023-10-30T13:37:36.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/37 file after the snapshot 2023-10-30T13:37:36.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/38 file after the snapshot 2023-10-30T13:37:36.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 245/39 file after the snapshot 2023-10-30T13:37:36.568 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/.snap/snap-subdir-test 2023-10-30T13:37:36.605 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/30 file after the snapshot 2023-10-30T13:37:36.608 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/31 file after the snapshot 2023-10-30T13:37:36.608 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/32 file after the snapshot 2023-10-30T13:37:36.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/33 file after the snapshot 2023-10-30T13:37:36.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/34 file after the snapshot 2023-10-30T13:37:36.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/35 file after the snapshot 2023-10-30T13:37:36.610 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/36 file after the snapshot 2023-10-30T13:37:36.610 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/37 file after the snapshot 2023-10-30T13:37:36.611 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/38 file after the snapshot 2023-10-30T13:37:36.611 INFO:tasks.workunit.client.0.smithi031.stdout: create 246/39 file after the snapshot 2023-10-30T13:37:36.611 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/.snap/snap-subdir-test 2023-10-30T13:37:36.788 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/30 file after the snapshot 2023-10-30T13:37:37.040 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/31 file after the snapshot 2023-10-30T13:37:37.041 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/32 file after the snapshot 2023-10-30T13:37:37.041 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/33 file after the snapshot 2023-10-30T13:37:37.041 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/34 file after the snapshot 2023-10-30T13:37:37.041 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/35 file after the snapshot 2023-10-30T13:37:37.041 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/36 file after the snapshot 2023-10-30T13:37:37.042 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/37 file after the snapshot 2023-10-30T13:37:37.042 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/38 file after the snapshot 2023-10-30T13:37:37.042 INFO:tasks.workunit.client.0.smithi031.stdout: create 247/39 file after the snapshot 2023-10-30T13:37:37.042 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/.snap/snap-subdir-test 2023-10-30T13:37:37.087 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/30 file after the snapshot 2023-10-30T13:37:37.092 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/31 file after the snapshot 2023-10-30T13:37:37.092 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/32 file after the snapshot 2023-10-30T13:37:37.092 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/33 file after the snapshot 2023-10-30T13:37:37.092 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/34 file after the snapshot 2023-10-30T13:37:37.092 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/35 file after the snapshot 2023-10-30T13:37:37.093 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/36 file after the snapshot 2023-10-30T13:37:37.093 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/37 file after the snapshot 2023-10-30T13:37:37.093 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/38 file after the snapshot 2023-10-30T13:37:37.093 INFO:tasks.workunit.client.0.smithi031.stdout: create 248/39 file after the snapshot 2023-10-30T13:37:37.093 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/.snap/snap-subdir-test 2023-10-30T13:37:37.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/30 file after the snapshot 2023-10-30T13:37:37.126 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/31 file after the snapshot 2023-10-30T13:37:37.126 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/32 file after the snapshot 2023-10-30T13:37:37.127 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/33 file after the snapshot 2023-10-30T13:37:37.127 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/34 file after the snapshot 2023-10-30T13:37:37.127 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/35 file after the snapshot 2023-10-30T13:37:37.127 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/36 file after the snapshot 2023-10-30T13:37:37.127 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/37 file after the snapshot 2023-10-30T13:37:37.127 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/38 file after the snapshot 2023-10-30T13:37:37.128 INFO:tasks.workunit.client.0.smithi031.stdout: create 249/39 file after the snapshot 2023-10-30T13:37:37.128 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/.snap/snap-subdir-test 2023-10-30T13:37:37.193 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/30 file after the snapshot 2023-10-30T13:37:37.198 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/31 file after the snapshot 2023-10-30T13:37:37.198 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/32 file after the snapshot 2023-10-30T13:37:37.198 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/33 file after the snapshot 2023-10-30T13:37:37.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/34 file after the snapshot 2023-10-30T13:37:37.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/35 file after the snapshot 2023-10-30T13:37:37.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/36 file after the snapshot 2023-10-30T13:37:37.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/37 file after the snapshot 2023-10-30T13:37:37.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/38 file after the snapshot 2023-10-30T13:37:37.199 INFO:tasks.workunit.client.0.smithi031.stdout: create 250/39 file after the snapshot 2023-10-30T13:37:37.200 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/.snap/snap-subdir-test 2023-10-30T13:37:37.249 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/30 file after the snapshot 2023-10-30T13:37:37.251 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/31 file after the snapshot 2023-10-30T13:37:37.251 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/32 file after the snapshot 2023-10-30T13:37:37.251 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/33 file after the snapshot 2023-10-30T13:37:37.251 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/34 file after the snapshot 2023-10-30T13:37:37.252 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/35 file after the snapshot 2023-10-30T13:37:37.252 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/36 file after the snapshot 2023-10-30T13:37:37.252 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/37 file after the snapshot 2023-10-30T13:37:37.252 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/38 file after the snapshot 2023-10-30T13:37:37.252 INFO:tasks.workunit.client.0.smithi031.stdout: create 251/39 file after the snapshot 2023-10-30T13:37:37.252 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/.snap/snap-subdir-test 2023-10-30T13:37:37.541 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/30 file after the snapshot 2023-10-30T13:37:37.545 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/31 file after the snapshot 2023-10-30T13:37:37.546 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/32 file after the snapshot 2023-10-30T13:37:37.546 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/33 file after the snapshot 2023-10-30T13:37:37.546 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/34 file after the snapshot 2023-10-30T13:37:37.546 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/35 file after the snapshot 2023-10-30T13:37:37.547 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/36 file after the snapshot 2023-10-30T13:37:37.547 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/37 file after the snapshot 2023-10-30T13:37:37.547 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/38 file after the snapshot 2023-10-30T13:37:37.547 INFO:tasks.workunit.client.0.smithi031.stdout: create 252/39 file after the snapshot 2023-10-30T13:37:37.547 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/.snap/snap-subdir-test 2023-10-30T13:37:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/30 file after the snapshot 2023-10-30T13:37:37.617 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/31 file after the snapshot 2023-10-30T13:37:37.617 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/32 file after the snapshot 2023-10-30T13:37:37.617 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/33 file after the snapshot 2023-10-30T13:37:37.618 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/34 file after the snapshot 2023-10-30T13:37:37.618 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/35 file after the snapshot 2023-10-30T13:37:37.618 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/36 file after the snapshot 2023-10-30T13:37:37.618 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/37 file after the snapshot 2023-10-30T13:37:37.618 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/38 file after the snapshot 2023-10-30T13:37:37.618 INFO:tasks.workunit.client.0.smithi031.stdout: create 253/39 file after the snapshot 2023-10-30T13:37:37.619 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/.snap/snap-subdir-test 2023-10-30T13:37:37.683 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/30 file after the snapshot 2023-10-30T13:37:37.686 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/31 file after the snapshot 2023-10-30T13:37:37.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/32 file after the snapshot 2023-10-30T13:37:37.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/33 file after the snapshot 2023-10-30T13:37:37.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/34 file after the snapshot 2023-10-30T13:37:37.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/35 file after the snapshot 2023-10-30T13:37:37.687 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/36 file after the snapshot 2023-10-30T13:37:37.688 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/37 file after the snapshot 2023-10-30T13:37:37.688 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/38 file after the snapshot 2023-10-30T13:37:37.688 INFO:tasks.workunit.client.0.smithi031.stdout: create 254/39 file after the snapshot 2023-10-30T13:37:37.688 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/.snap/snap-subdir-test 2023-10-30T13:37:37.741 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/30 file after the snapshot 2023-10-30T13:37:37.744 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/31 file after the snapshot 2023-10-30T13:37:37.744 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/32 file after the snapshot 2023-10-30T13:37:37.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/33 file after the snapshot 2023-10-30T13:37:37.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/34 file after the snapshot 2023-10-30T13:37:37.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/35 file after the snapshot 2023-10-30T13:37:37.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/36 file after the snapshot 2023-10-30T13:37:37.745 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/37 file after the snapshot 2023-10-30T13:37:37.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/38 file after the snapshot 2023-10-30T13:37:37.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 255/39 file after the snapshot 2023-10-30T13:37:37.746 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/.snap/snap-subdir-test 2023-10-30T13:37:37.811 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/30 file after the snapshot 2023-10-30T13:37:37.815 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/31 file after the snapshot 2023-10-30T13:37:37.815 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/32 file after the snapshot 2023-10-30T13:37:37.815 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/33 file after the snapshot 2023-10-30T13:37:37.815 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/34 file after the snapshot 2023-10-30T13:37:37.815 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/35 file after the snapshot 2023-10-30T13:37:37.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/36 file after the snapshot 2023-10-30T13:37:37.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/37 file after the snapshot 2023-10-30T13:37:37.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/38 file after the snapshot 2023-10-30T13:37:37.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 256/39 file after the snapshot 2023-10-30T13:37:37.816 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/.snap/snap-subdir-test 2023-10-30T13:37:37.843 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/30 file after the snapshot 2023-10-30T13:37:37.846 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/31 file after the snapshot 2023-10-30T13:37:37.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/32 file after the snapshot 2023-10-30T13:37:37.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/33 file after the snapshot 2023-10-30T13:37:37.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/34 file after the snapshot 2023-10-30T13:37:37.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/35 file after the snapshot 2023-10-30T13:37:37.847 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/36 file after the snapshot 2023-10-30T13:37:37.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/37 file after the snapshot 2023-10-30T13:37:37.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/38 file after the snapshot 2023-10-30T13:37:37.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 257/39 file after the snapshot 2023-10-30T13:37:37.848 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/.snap/snap-subdir-test 2023-10-30T13:37:38.539 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/30 file after the snapshot 2023-10-30T13:37:38.542 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/31 file after the snapshot 2023-10-30T13:37:38.542 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/32 file after the snapshot 2023-10-30T13:37:38.542 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/33 file after the snapshot 2023-10-30T13:37:38.542 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/34 file after the snapshot 2023-10-30T13:37:38.543 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/35 file after the snapshot 2023-10-30T13:37:38.543 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/36 file after the snapshot 2023-10-30T13:37:38.543 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/37 file after the snapshot 2023-10-30T13:37:38.543 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/38 file after the snapshot 2023-10-30T13:37:38.543 INFO:tasks.workunit.client.0.smithi031.stdout: create 258/39 file after the snapshot 2023-10-30T13:37:38.544 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/.snap/snap-subdir-test 2023-10-30T13:37:38.605 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/30 file after the snapshot 2023-10-30T13:37:38.608 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/31 file after the snapshot 2023-10-30T13:37:38.608 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/32 file after the snapshot 2023-10-30T13:37:38.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/33 file after the snapshot 2023-10-30T13:37:38.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/34 file after the snapshot 2023-10-30T13:37:38.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/35 file after the snapshot 2023-10-30T13:37:38.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/36 file after the snapshot 2023-10-30T13:37:38.609 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/37 file after the snapshot 2023-10-30T13:37:38.610 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/38 file after the snapshot 2023-10-30T13:37:38.610 INFO:tasks.workunit.client.0.smithi031.stdout: create 259/39 file after the snapshot 2023-10-30T13:37:38.610 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/.snap/snap-subdir-test 2023-10-30T13:37:38.754 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/30 file after the snapshot 2023-10-30T13:37:38.755 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/31 file after the snapshot 2023-10-30T13:37:38.756 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/32 file after the snapshot 2023-10-30T13:37:38.756 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/33 file after the snapshot 2023-10-30T13:37:38.756 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/34 file after the snapshot 2023-10-30T13:37:38.756 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/35 file after the snapshot 2023-10-30T13:37:38.757 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/36 file after the snapshot 2023-10-30T13:37:38.757 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/37 file after the snapshot 2023-10-30T13:37:38.757 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/38 file after the snapshot 2023-10-30T13:37:38.757 INFO:tasks.workunit.client.0.smithi031.stdout: create 260/39 file after the snapshot 2023-10-30T13:37:38.758 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/.snap/snap-subdir-test 2023-10-30T13:37:39.637 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/30 file after the snapshot 2023-10-30T13:37:39.641 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/31 file after the snapshot 2023-10-30T13:37:39.641 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/32 file after the snapshot 2023-10-30T13:37:39.641 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/33 file after the snapshot 2023-10-30T13:37:39.641 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/34 file after the snapshot 2023-10-30T13:37:39.642 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/35 file after the snapshot 2023-10-30T13:37:39.642 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/36 file after the snapshot 2023-10-30T13:37:39.642 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/37 file after the snapshot 2023-10-30T13:37:39.642 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/38 file after the snapshot 2023-10-30T13:37:39.643 INFO:tasks.workunit.client.0.smithi031.stdout: create 261/39 file after the snapshot 2023-10-30T13:37:39.643 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/.snap/snap-subdir-test 2023-10-30T13:37:39.931 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/30 file after the snapshot 2023-10-30T13:37:39.934 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/31 file after the snapshot 2023-10-30T13:37:39.935 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/32 file after the snapshot 2023-10-30T13:37:39.935 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/33 file after the snapshot 2023-10-30T13:37:39.935 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/34 file after the snapshot 2023-10-30T13:37:39.936 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/35 file after the snapshot 2023-10-30T13:37:39.936 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/36 file after the snapshot 2023-10-30T13:37:39.936 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/37 file after the snapshot 2023-10-30T13:37:39.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/38 file after the snapshot 2023-10-30T13:37:39.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 262/39 file after the snapshot 2023-10-30T13:37:39.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/.snap/snap-subdir-test 2023-10-30T13:37:39.970 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/30 file after the snapshot 2023-10-30T13:37:39.972 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/31 file after the snapshot 2023-10-30T13:37:39.973 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/32 file after the snapshot 2023-10-30T13:37:39.973 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/33 file after the snapshot 2023-10-30T13:37:39.973 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/34 file after the snapshot 2023-10-30T13:37:39.974 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/35 file after the snapshot 2023-10-30T13:37:39.974 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/36 file after the snapshot 2023-10-30T13:37:39.974 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/37 file after the snapshot 2023-10-30T13:37:39.975 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/38 file after the snapshot 2023-10-30T13:37:39.975 INFO:tasks.workunit.client.0.smithi031.stdout: create 263/39 file after the snapshot 2023-10-30T13:37:39.975 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/.snap/snap-subdir-test 2023-10-30T13:37:40.002 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/30 file after the snapshot 2023-10-30T13:37:40.007 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/31 file after the snapshot 2023-10-30T13:37:40.008 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/32 file after the snapshot 2023-10-30T13:37:40.008 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/33 file after the snapshot 2023-10-30T13:37:40.008 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/34 file after the snapshot 2023-10-30T13:37:40.008 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/35 file after the snapshot 2023-10-30T13:37:40.009 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/36 file after the snapshot 2023-10-30T13:37:40.009 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/37 file after the snapshot 2023-10-30T13:37:40.009 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/38 file after the snapshot 2023-10-30T13:37:40.010 INFO:tasks.workunit.client.0.smithi031.stdout: create 264/39 file after the snapshot 2023-10-30T13:37:40.010 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/.snap/snap-subdir-test 2023-10-30T13:37:40.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/30 file after the snapshot 2023-10-30T13:37:40.047 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/31 file after the snapshot 2023-10-30T13:37:40.047 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/32 file after the snapshot 2023-10-30T13:37:40.048 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/33 file after the snapshot 2023-10-30T13:37:40.048 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/34 file after the snapshot 2023-10-30T13:37:40.048 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/35 file after the snapshot 2023-10-30T13:37:40.049 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/36 file after the snapshot 2023-10-30T13:37:40.049 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/37 file after the snapshot 2023-10-30T13:37:40.049 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/38 file after the snapshot 2023-10-30T13:37:40.049 INFO:tasks.workunit.client.0.smithi031.stdout: create 265/39 file after the snapshot 2023-10-30T13:37:40.050 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/.snap/snap-subdir-test 2023-10-30T13:37:40.076 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/30 file after the snapshot 2023-10-30T13:37:40.079 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/31 file after the snapshot 2023-10-30T13:37:40.079 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/32 file after the snapshot 2023-10-30T13:37:40.080 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/33 file after the snapshot 2023-10-30T13:37:40.080 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/34 file after the snapshot 2023-10-30T13:37:40.080 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/35 file after the snapshot 2023-10-30T13:37:40.081 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/36 file after the snapshot 2023-10-30T13:37:40.081 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/37 file after the snapshot 2023-10-30T13:37:40.081 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/38 file after the snapshot 2023-10-30T13:37:40.082 INFO:tasks.workunit.client.0.smithi031.stdout: create 266/39 file after the snapshot 2023-10-30T13:37:40.082 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/.snap/snap-subdir-test 2023-10-30T13:37:43.874 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/30 file after the snapshot 2023-10-30T13:37:43.876 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/31 file after the snapshot 2023-10-30T13:37:43.876 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/32 file after the snapshot 2023-10-30T13:37:43.876 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/33 file after the snapshot 2023-10-30T13:37:43.876 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/34 file after the snapshot 2023-10-30T13:37:43.877 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/35 file after the snapshot 2023-10-30T13:37:43.877 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/36 file after the snapshot 2023-10-30T13:37:43.877 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/37 file after the snapshot 2023-10-30T13:37:43.877 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/38 file after the snapshot 2023-10-30T13:37:43.877 INFO:tasks.workunit.client.0.smithi031.stdout: create 267/39 file after the snapshot 2023-10-30T13:37:43.878 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/.snap/snap-subdir-test 2023-10-30T13:37:43.907 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/30 file after the snapshot 2023-10-30T13:37:43.910 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/31 file after the snapshot 2023-10-30T13:37:43.910 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/32 file after the snapshot 2023-10-30T13:37:43.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/33 file after the snapshot 2023-10-30T13:37:43.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/34 file after the snapshot 2023-10-30T13:37:43.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/35 file after the snapshot 2023-10-30T13:37:43.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/36 file after the snapshot 2023-10-30T13:37:43.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/37 file after the snapshot 2023-10-30T13:37:43.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/38 file after the snapshot 2023-10-30T13:37:43.912 INFO:tasks.workunit.client.0.smithi031.stdout: create 268/39 file after the snapshot 2023-10-30T13:37:43.912 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/.snap/snap-subdir-test 2023-10-30T13:37:43.960 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/30 file after the snapshot 2023-10-30T13:37:43.963 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/31 file after the snapshot 2023-10-30T13:37:43.963 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/32 file after the snapshot 2023-10-30T13:37:43.963 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/33 file after the snapshot 2023-10-30T13:37:43.964 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/34 file after the snapshot 2023-10-30T13:37:43.964 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/35 file after the snapshot 2023-10-30T13:37:43.964 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/36 file after the snapshot 2023-10-30T13:37:43.964 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/37 file after the snapshot 2023-10-30T13:37:43.964 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/38 file after the snapshot 2023-10-30T13:37:43.964 INFO:tasks.workunit.client.0.smithi031.stdout: create 269/39 file after the snapshot 2023-10-30T13:37:43.965 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/.snap/snap-subdir-test 2023-10-30T13:37:44.155 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/30 file after the snapshot 2023-10-30T13:37:44.156 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/31 file after the snapshot 2023-10-30T13:37:44.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/32 file after the snapshot 2023-10-30T13:37:44.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/33 file after the snapshot 2023-10-30T13:37:44.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/34 file after the snapshot 2023-10-30T13:37:44.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/35 file after the snapshot 2023-10-30T13:37:44.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/36 file after the snapshot 2023-10-30T13:37:44.158 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/37 file after the snapshot 2023-10-30T13:37:44.158 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/38 file after the snapshot 2023-10-30T13:37:44.158 INFO:tasks.workunit.client.0.smithi031.stdout: create 270/39 file after the snapshot 2023-10-30T13:37:44.158 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/.snap/snap-subdir-test 2023-10-30T13:37:44.181 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/30 file after the snapshot 2023-10-30T13:37:44.183 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/31 file after the snapshot 2023-10-30T13:37:44.183 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/32 file after the snapshot 2023-10-30T13:37:44.183 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/33 file after the snapshot 2023-10-30T13:37:44.184 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/34 file after the snapshot 2023-10-30T13:37:44.184 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/35 file after the snapshot 2023-10-30T13:37:44.184 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/36 file after the snapshot 2023-10-30T13:37:44.184 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/37 file after the snapshot 2023-10-30T13:37:44.184 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/38 file after the snapshot 2023-10-30T13:37:44.184 INFO:tasks.workunit.client.0.smithi031.stdout: create 271/39 file after the snapshot 2023-10-30T13:37:44.185 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/.snap/snap-subdir-test 2023-10-30T13:37:44.386 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/30 file after the snapshot 2023-10-30T13:37:44.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/31 file after the snapshot 2023-10-30T13:37:44.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/32 file after the snapshot 2023-10-30T13:37:44.387 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/33 file after the snapshot 2023-10-30T13:37:44.388 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/34 file after the snapshot 2023-10-30T13:37:44.388 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/35 file after the snapshot 2023-10-30T13:37:44.388 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/36 file after the snapshot 2023-10-30T13:37:44.388 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/37 file after the snapshot 2023-10-30T13:37:44.388 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/38 file after the snapshot 2023-10-30T13:37:44.389 INFO:tasks.workunit.client.0.smithi031.stdout: create 272/39 file after the snapshot 2023-10-30T13:37:44.389 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/.snap/snap-subdir-test 2023-10-30T13:37:44.456 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/30 file after the snapshot 2023-10-30T13:37:44.459 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/31 file after the snapshot 2023-10-30T13:37:44.459 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/32 file after the snapshot 2023-10-30T13:37:44.459 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/33 file after the snapshot 2023-10-30T13:37:44.460 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/34 file after the snapshot 2023-10-30T13:37:44.460 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/35 file after the snapshot 2023-10-30T13:37:44.460 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/36 file after the snapshot 2023-10-30T13:37:44.460 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/37 file after the snapshot 2023-10-30T13:37:44.460 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/38 file after the snapshot 2023-10-30T13:37:44.461 INFO:tasks.workunit.client.0.smithi031.stdout: create 273/39 file after the snapshot 2023-10-30T13:37:44.461 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/.snap/snap-subdir-test 2023-10-30T13:37:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/30 file after the snapshot 2023-10-30T13:37:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/31 file after the snapshot 2023-10-30T13:37:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/32 file after the snapshot 2023-10-30T13:37:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/33 file after the snapshot 2023-10-30T13:37:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/34 file after the snapshot 2023-10-30T13:37:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/35 file after the snapshot 2023-10-30T13:37:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/36 file after the snapshot 2023-10-30T13:37:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/37 file after the snapshot 2023-10-30T13:37:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/38 file after the snapshot 2023-10-30T13:37:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 274/39 file after the snapshot 2023-10-30T13:37:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/.snap/snap-subdir-test 2023-10-30T13:37:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/30 file after the snapshot 2023-10-30T13:37:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/31 file after the snapshot 2023-10-30T13:37:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/32 file after the snapshot 2023-10-30T13:37:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/33 file after the snapshot 2023-10-30T13:37:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/34 file after the snapshot 2023-10-30T13:37:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/35 file after the snapshot 2023-10-30T13:37:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/36 file after the snapshot 2023-10-30T13:37:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/37 file after the snapshot 2023-10-30T13:37:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/38 file after the snapshot 2023-10-30T13:37:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 275/39 file after the snapshot 2023-10-30T13:37:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/.snap/snap-subdir-test 2023-10-30T13:37:45.118 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/30 file after the snapshot 2023-10-30T13:37:45.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/31 file after the snapshot 2023-10-30T13:37:45.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/32 file after the snapshot 2023-10-30T13:37:45.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/33 file after the snapshot 2023-10-30T13:37:45.120 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/34 file after the snapshot 2023-10-30T13:37:45.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/35 file after the snapshot 2023-10-30T13:37:45.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/36 file after the snapshot 2023-10-30T13:37:45.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/37 file after the snapshot 2023-10-30T13:37:45.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/38 file after the snapshot 2023-10-30T13:37:45.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 276/39 file after the snapshot 2023-10-30T13:37:45.121 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/.snap/snap-subdir-test 2023-10-30T13:37:45.154 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/30 file after the snapshot 2023-10-30T13:37:45.155 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/31 file after the snapshot 2023-10-30T13:37:45.156 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/32 file after the snapshot 2023-10-30T13:37:45.156 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/33 file after the snapshot 2023-10-30T13:37:45.156 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/34 file after the snapshot 2023-10-30T13:37:45.156 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/35 file after the snapshot 2023-10-30T13:37:45.156 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/36 file after the snapshot 2023-10-30T13:37:45.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/37 file after the snapshot 2023-10-30T13:37:45.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/38 file after the snapshot 2023-10-30T13:37:45.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 277/39 file after the snapshot 2023-10-30T13:37:45.157 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/.snap/snap-subdir-test 2023-10-30T13:37:45.187 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/30 file after the snapshot 2023-10-30T13:37:45.190 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/31 file after the snapshot 2023-10-30T13:37:45.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/32 file after the snapshot 2023-10-30T13:37:45.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/33 file after the snapshot 2023-10-30T13:37:45.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/34 file after the snapshot 2023-10-30T13:37:45.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/35 file after the snapshot 2023-10-30T13:37:45.191 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/36 file after the snapshot 2023-10-30T13:37:45.192 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/37 file after the snapshot 2023-10-30T13:37:45.192 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/38 file after the snapshot 2023-10-30T13:37:45.192 INFO:tasks.workunit.client.0.smithi031.stdout: create 278/39 file after the snapshot 2023-10-30T13:37:45.192 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/.snap/snap-subdir-test 2023-10-30T13:37:45.221 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/30 file after the snapshot 2023-10-30T13:37:45.224 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/31 file after the snapshot 2023-10-30T13:37:45.224 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/32 file after the snapshot 2023-10-30T13:37:45.225 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/33 file after the snapshot 2023-10-30T13:37:45.225 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/34 file after the snapshot 2023-10-30T13:37:45.225 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/35 file after the snapshot 2023-10-30T13:37:45.225 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/36 file after the snapshot 2023-10-30T13:37:45.225 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/37 file after the snapshot 2023-10-30T13:37:45.225 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/38 file after the snapshot 2023-10-30T13:37:45.226 INFO:tasks.workunit.client.0.smithi031.stdout: create 279/39 file after the snapshot 2023-10-30T13:37:45.226 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/.snap/snap-subdir-test 2023-10-30T13:37:45.263 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/30 file after the snapshot 2023-10-30T13:37:45.265 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/31 file after the snapshot 2023-10-30T13:37:45.265 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/32 file after the snapshot 2023-10-30T13:37:45.265 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/33 file after the snapshot 2023-10-30T13:37:45.265 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/34 file after the snapshot 2023-10-30T13:37:45.265 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/35 file after the snapshot 2023-10-30T13:37:45.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/36 file after the snapshot 2023-10-30T13:37:45.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/37 file after the snapshot 2023-10-30T13:37:45.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/38 file after the snapshot 2023-10-30T13:37:45.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 280/39 file after the snapshot 2023-10-30T13:37:45.266 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/.snap/snap-subdir-test 2023-10-30T13:37:45.303 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/30 file after the snapshot 2023-10-30T13:37:45.306 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/31 file after the snapshot 2023-10-30T13:37:45.306 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/32 file after the snapshot 2023-10-30T13:37:45.306 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/33 file after the snapshot 2023-10-30T13:37:45.307 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/34 file after the snapshot 2023-10-30T13:37:45.307 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/35 file after the snapshot 2023-10-30T13:37:45.307 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/36 file after the snapshot 2023-10-30T13:37:45.307 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/37 file after the snapshot 2023-10-30T13:37:45.307 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/38 file after the snapshot 2023-10-30T13:37:45.308 INFO:tasks.workunit.client.0.smithi031.stdout: create 281/39 file after the snapshot 2023-10-30T13:37:45.308 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/.snap/snap-subdir-test 2023-10-30T13:37:45.366 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/30 file after the snapshot 2023-10-30T13:37:45.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/31 file after the snapshot 2023-10-30T13:37:45.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/32 file after the snapshot 2023-10-30T13:37:45.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/33 file after the snapshot 2023-10-30T13:37:45.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/34 file after the snapshot 2023-10-30T13:37:45.370 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/35 file after the snapshot 2023-10-30T13:37:45.371 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/36 file after the snapshot 2023-10-30T13:37:45.371 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/37 file after the snapshot 2023-10-30T13:37:45.371 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/38 file after the snapshot 2023-10-30T13:37:45.371 INFO:tasks.workunit.client.0.smithi031.stdout: create 282/39 file after the snapshot 2023-10-30T13:37:45.371 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/.snap/snap-subdir-test 2023-10-30T13:37:45.404 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/30 file after the snapshot 2023-10-30T13:37:45.406 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/31 file after the snapshot 2023-10-30T13:37:45.406 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/32 file after the snapshot 2023-10-30T13:37:45.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/33 file after the snapshot 2023-10-30T13:37:45.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/34 file after the snapshot 2023-10-30T13:37:45.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/35 file after the snapshot 2023-10-30T13:37:45.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/36 file after the snapshot 2023-10-30T13:37:45.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/37 file after the snapshot 2023-10-30T13:37:45.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/38 file after the snapshot 2023-10-30T13:37:45.408 INFO:tasks.workunit.client.0.smithi031.stdout: create 283/39 file after the snapshot 2023-10-30T13:37:45.408 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/.snap/snap-subdir-test 2023-10-30T13:37:45.438 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/30 file after the snapshot 2023-10-30T13:37:45.439 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/31 file after the snapshot 2023-10-30T13:37:45.439 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/32 file after the snapshot 2023-10-30T13:37:45.439 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/33 file after the snapshot 2023-10-30T13:37:45.440 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/34 file after the snapshot 2023-10-30T13:37:45.440 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/35 file after the snapshot 2023-10-30T13:37:45.440 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/36 file after the snapshot 2023-10-30T13:37:45.440 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/37 file after the snapshot 2023-10-30T13:37:45.440 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/38 file after the snapshot 2023-10-30T13:37:45.440 INFO:tasks.workunit.client.0.smithi031.stdout: create 284/39 file after the snapshot 2023-10-30T13:37:45.441 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/.snap/snap-subdir-test 2023-10-30T13:37:45.462 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/30 file after the snapshot 2023-10-30T13:37:45.465 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/31 file after the snapshot 2023-10-30T13:37:45.465 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/32 file after the snapshot 2023-10-30T13:37:45.465 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/33 file after the snapshot 2023-10-30T13:37:45.465 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/34 file after the snapshot 2023-10-30T13:37:45.465 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/35 file after the snapshot 2023-10-30T13:37:45.466 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/36 file after the snapshot 2023-10-30T13:37:45.466 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/37 file after the snapshot 2023-10-30T13:37:45.466 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/38 file after the snapshot 2023-10-30T13:37:45.466 INFO:tasks.workunit.client.0.smithi031.stdout: create 285/39 file after the snapshot 2023-10-30T13:37:45.466 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/.snap/snap-subdir-test 2023-10-30T13:37:45.500 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/30 file after the snapshot 2023-10-30T13:37:45.502 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/31 file after the snapshot 2023-10-30T13:37:45.502 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/32 file after the snapshot 2023-10-30T13:37:45.502 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/33 file after the snapshot 2023-10-30T13:37:45.502 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/34 file after the snapshot 2023-10-30T13:37:45.503 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/35 file after the snapshot 2023-10-30T13:37:45.503 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/36 file after the snapshot 2023-10-30T13:37:45.503 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/37 file after the snapshot 2023-10-30T13:37:45.503 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/38 file after the snapshot 2023-10-30T13:37:45.503 INFO:tasks.workunit.client.0.smithi031.stdout: create 286/39 file after the snapshot 2023-10-30T13:37:45.504 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/.snap/snap-subdir-test 2023-10-30T13:37:46.781 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/30 file after the snapshot 2023-10-30T13:37:46.784 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/31 file after the snapshot 2023-10-30T13:37:46.784 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/32 file after the snapshot 2023-10-30T13:37:46.785 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/33 file after the snapshot 2023-10-30T13:37:46.785 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/34 file after the snapshot 2023-10-30T13:37:46.785 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/35 file after the snapshot 2023-10-30T13:37:46.785 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/36 file after the snapshot 2023-10-30T13:37:46.785 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/37 file after the snapshot 2023-10-30T13:37:46.786 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/38 file after the snapshot 2023-10-30T13:37:46.786 INFO:tasks.workunit.client.0.smithi031.stdout: create 287/39 file after the snapshot 2023-10-30T13:37:46.786 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/.snap/snap-subdir-test 2023-10-30T13:37:46.835 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/30 file after the snapshot 2023-10-30T13:37:46.839 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/31 file after the snapshot 2023-10-30T13:37:46.839 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/32 file after the snapshot 2023-10-30T13:37:46.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/33 file after the snapshot 2023-10-30T13:37:46.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/34 file after the snapshot 2023-10-30T13:37:46.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/35 file after the snapshot 2023-10-30T13:37:46.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/36 file after the snapshot 2023-10-30T13:37:46.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/37 file after the snapshot 2023-10-30T13:37:46.840 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/38 file after the snapshot 2023-10-30T13:37:46.841 INFO:tasks.workunit.client.0.smithi031.stdout: create 288/39 file after the snapshot 2023-10-30T13:37:46.841 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/.snap/snap-subdir-test 2023-10-30T13:37:46.901 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/30 file after the snapshot 2023-10-30T13:37:46.904 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/31 file after the snapshot 2023-10-30T13:37:46.904 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/32 file after the snapshot 2023-10-30T13:37:46.904 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/33 file after the snapshot 2023-10-30T13:37:46.904 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/34 file after the snapshot 2023-10-30T13:37:46.905 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/35 file after the snapshot 2023-10-30T13:37:46.905 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/36 file after the snapshot 2023-10-30T13:37:46.905 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/37 file after the snapshot 2023-10-30T13:37:46.905 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/38 file after the snapshot 2023-10-30T13:37:46.905 INFO:tasks.workunit.client.0.smithi031.stdout: create 289/39 file after the snapshot 2023-10-30T13:37:46.906 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/.snap/snap-subdir-test 2023-10-30T13:37:46.944 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/30 file after the snapshot 2023-10-30T13:37:46.946 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/31 file after the snapshot 2023-10-30T13:37:46.946 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/32 file after the snapshot 2023-10-30T13:37:46.946 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/33 file after the snapshot 2023-10-30T13:37:46.947 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/34 file after the snapshot 2023-10-30T13:37:46.947 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/35 file after the snapshot 2023-10-30T13:37:46.947 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/36 file after the snapshot 2023-10-30T13:37:46.947 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/37 file after the snapshot 2023-10-30T13:37:46.947 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/38 file after the snapshot 2023-10-30T13:37:46.947 INFO:tasks.workunit.client.0.smithi031.stdout: create 290/39 file after the snapshot 2023-10-30T13:37:46.948 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/.snap/snap-subdir-test 2023-10-30T13:37:46.977 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/30 file after the snapshot 2023-10-30T13:37:46.980 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/31 file after the snapshot 2023-10-30T13:37:46.980 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/32 file after the snapshot 2023-10-30T13:37:46.980 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/33 file after the snapshot 2023-10-30T13:37:46.980 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/34 file after the snapshot 2023-10-30T13:37:46.981 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/35 file after the snapshot 2023-10-30T13:37:46.981 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/36 file after the snapshot 2023-10-30T13:37:46.981 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/37 file after the snapshot 2023-10-30T13:37:46.981 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/38 file after the snapshot 2023-10-30T13:37:46.981 INFO:tasks.workunit.client.0.smithi031.stdout: create 291/39 file after the snapshot 2023-10-30T13:37:46.981 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/.snap/snap-subdir-test 2023-10-30T13:37:47.733 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/30 file after the snapshot 2023-10-30T13:37:47.736 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/31 file after the snapshot 2023-10-30T13:37:47.736 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/32 file after the snapshot 2023-10-30T13:37:47.736 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/33 file after the snapshot 2023-10-30T13:37:47.737 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/34 file after the snapshot 2023-10-30T13:37:47.737 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/35 file after the snapshot 2023-10-30T13:37:47.737 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/36 file after the snapshot 2023-10-30T13:37:47.737 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/37 file after the snapshot 2023-10-30T13:37:47.737 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/38 file after the snapshot 2023-10-30T13:37:47.738 INFO:tasks.workunit.client.0.smithi031.stdout: create 292/39 file after the snapshot 2023-10-30T13:37:47.738 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/.snap/snap-subdir-test 2023-10-30T13:37:47.757 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/30 file after the snapshot 2023-10-30T13:37:48.319 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/31 file after the snapshot 2023-10-30T13:37:48.319 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/32 file after the snapshot 2023-10-30T13:37:48.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/33 file after the snapshot 2023-10-30T13:37:48.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/34 file after the snapshot 2023-10-30T13:37:48.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/35 file after the snapshot 2023-10-30T13:37:48.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/36 file after the snapshot 2023-10-30T13:37:48.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/37 file after the snapshot 2023-10-30T13:37:48.320 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/38 file after the snapshot 2023-10-30T13:37:48.321 INFO:tasks.workunit.client.0.smithi031.stdout: create 293/39 file after the snapshot 2023-10-30T13:37:48.321 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/.snap/snap-subdir-test 2023-10-30T13:37:48.345 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/30 file after the snapshot 2023-10-30T13:37:48.347 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/31 file after the snapshot 2023-10-30T13:37:48.347 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/32 file after the snapshot 2023-10-30T13:37:48.347 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/33 file after the snapshot 2023-10-30T13:37:48.347 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/34 file after the snapshot 2023-10-30T13:37:48.347 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/35 file after the snapshot 2023-10-30T13:37:48.348 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/36 file after the snapshot 2023-10-30T13:37:48.348 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/37 file after the snapshot 2023-10-30T13:37:48.348 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/38 file after the snapshot 2023-10-30T13:37:48.348 INFO:tasks.workunit.client.0.smithi031.stdout: create 294/39 file after the snapshot 2023-10-30T13:37:48.348 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/.snap/snap-subdir-test 2023-10-30T13:37:48.373 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/30 file after the snapshot 2023-10-30T13:37:48.375 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/31 file after the snapshot 2023-10-30T13:37:48.375 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/32 file after the snapshot 2023-10-30T13:37:48.375 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/33 file after the snapshot 2023-10-30T13:37:48.375 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/34 file after the snapshot 2023-10-30T13:37:48.376 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/35 file after the snapshot 2023-10-30T13:37:48.376 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/36 file after the snapshot 2023-10-30T13:37:48.376 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/37 file after the snapshot 2023-10-30T13:37:48.376 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/38 file after the snapshot 2023-10-30T13:37:48.376 INFO:tasks.workunit.client.0.smithi031.stdout: create 295/39 file after the snapshot 2023-10-30T13:37:48.377 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/.snap/snap-subdir-test 2023-10-30T13:37:48.402 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/30 file after the snapshot 2023-10-30T13:37:48.405 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/31 file after the snapshot 2023-10-30T13:37:48.405 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/32 file after the snapshot 2023-10-30T13:37:48.406 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/33 file after the snapshot 2023-10-30T13:37:48.406 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/34 file after the snapshot 2023-10-30T13:37:48.406 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/35 file after the snapshot 2023-10-30T13:37:48.406 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/36 file after the snapshot 2023-10-30T13:37:48.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/37 file after the snapshot 2023-10-30T13:37:48.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/38 file after the snapshot 2023-10-30T13:37:48.407 INFO:tasks.workunit.client.0.smithi031.stdout: create 296/39 file after the snapshot 2023-10-30T13:37:48.408 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/.snap/snap-subdir-test 2023-10-30T13:37:48.660 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/30 file after the snapshot 2023-10-30T13:37:48.911 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/31 file after the snapshot 2023-10-30T13:37:48.912 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/32 file after the snapshot 2023-10-30T13:37:48.912 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/33 file after the snapshot 2023-10-30T13:37:48.912 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/34 file after the snapshot 2023-10-30T13:37:48.912 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/35 file after the snapshot 2023-10-30T13:37:48.913 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/36 file after the snapshot 2023-10-30T13:37:48.913 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/37 file after the snapshot 2023-10-30T13:37:48.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/38 file after the snapshot 2023-10-30T13:37:48.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 297/39 file after the snapshot 2023-10-30T13:37:48.914 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/.snap/snap-subdir-test 2023-10-30T13:37:48.937 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/30 file after the snapshot 2023-10-30T13:37:48.939 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/31 file after the snapshot 2023-10-30T13:37:48.940 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/32 file after the snapshot 2023-10-30T13:37:48.940 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/33 file after the snapshot 2023-10-30T13:37:48.940 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/34 file after the snapshot 2023-10-30T13:37:48.940 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/35 file after the snapshot 2023-10-30T13:37:48.940 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/36 file after the snapshot 2023-10-30T13:37:48.941 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/37 file after the snapshot 2023-10-30T13:37:48.941 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/38 file after the snapshot 2023-10-30T13:37:48.941 INFO:tasks.workunit.client.0.smithi031.stdout: create 298/39 file after the snapshot 2023-10-30T13:37:48.942 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/.snap/snap-subdir-test 2023-10-30T13:37:48.962 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/30 file after the snapshot 2023-10-30T13:37:48.966 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/31 file after the snapshot 2023-10-30T13:37:48.967 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/32 file after the snapshot 2023-10-30T13:37:48.967 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/33 file after the snapshot 2023-10-30T13:37:48.967 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/34 file after the snapshot 2023-10-30T13:37:48.967 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/35 file after the snapshot 2023-10-30T13:37:48.967 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/36 file after the snapshot 2023-10-30T13:37:48.968 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/37 file after the snapshot 2023-10-30T13:37:48.968 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/38 file after the snapshot 2023-10-30T13:37:48.968 INFO:tasks.workunit.client.0.smithi031.stdout: create 299/39 file after the snapshot 2023-10-30T13:37:48.968 INFO:tasks.workunit.client.0.smithi031.stdout:Take second snapshot .snap/test2 2023-10-30T13:37:49.744 INFO:tasks.workunit.client.0.smithi031.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-30T13:37:54.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-30T13:37:54.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-30T13:37:54.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-30T13:37:54.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-30T13:37:54.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-30T13:37:54.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-30T13:37:54.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-30T13:37:54.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-30T13:37:54.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-30T13:37:54.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-30T13:37:54.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-30T13:37:54.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-30T13:37:54.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-30T13:37:54.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-30T13:37:54.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-30T13:37:54.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-30T13:37:54.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-30T13:37:54.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-30T13:37:54.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-30T13:37:54.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-30T13:37:54.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-30T13:37:54.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-30T13:37:54.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-30T13:37:54.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-30T13:37:54.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-30T13:37:54.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-30T13:37:54.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-30T13:37:54.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-30T13:37:54.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-30T13:37:54.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-30T13:37:54.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-30T13:37:54.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-30T13:37:54.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-30T13:37:54.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-30T13:37:54.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-30T13:37:54.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-30T13:37:54.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-30T13:37:54.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-30T13:37:54.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-30T13:37:54.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-30T13:37:54.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-30T13:37:54.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-30T13:37:54.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-30T13:37:54.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-30T13:37:54.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-30T13:37:54.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-30T13:37:54.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-30T13:37:54.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-30T13:37:54.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-30T13:37:54.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-30T13:37:54.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-30T13:37:54.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-30T13:37:54.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-30T13:37:54.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-30T13:37:54.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-30T13:37:54.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-30T13:37:54.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-30T13:37:54.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-30T13:37:54.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-30T13:37:54.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-30T13:37:54.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-30T13:37:54.801 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-30T13:37:54.801 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-30T13:37:54.801 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-30T13:37:54.801 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-30T13:37:54.801 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-30T13:37:54.802 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-30T13:37:54.802 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-30T13:37:54.802 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-30T13:37:54.802 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-30T13:37:54.802 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-30T13:37:54.803 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-30T13:37:54.803 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-30T13:37:54.803 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-30T13:37:54.803 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-30T13:37:54.803 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-30T13:37:54.803 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-30T13:37:54.804 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-30T13:37:54.804 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-30T13:37:54.804 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-30T13:37:54.805 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-30T13:37:55.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:55.248 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:02.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-30T13:38:02.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-30T13:38:02.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-30T13:38:02.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-30T13:38:02.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-30T13:38:02.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-30T13:38:02.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-30T13:38:02.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-30T13:38:02.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-30T13:38:02.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-30T13:38:02.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-30T13:38:02.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-30T13:38:02.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-30T13:38:02.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-30T13:38:02.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-30T13:38:02.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-30T13:38:02.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-30T13:38:02.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-30T13:38:02.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-30T13:38:02.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-30T13:38:02.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-30T13:38:02.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-30T13:38:02.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-30T13:38:02.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-30T13:38:02.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-30T13:38:02.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-30T13:38:02.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-30T13:38:02.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-30T13:38:02.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-30T13:38:02.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-30T13:38:02.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-30T13:38:02.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-30T13:38:02.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-30T13:38:02.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-30T13:38:02.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-30T13:38:02.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-30T13:38:02.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-30T13:38:02.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-30T13:38:02.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-30T13:38:02.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-30T13:38:02.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-30T13:38:02.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-30T13:38:02.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-30T13:38:02.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-30T13:38:02.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-30T13:38:02.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-30T13:38:02.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-30T13:38:02.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-30T13:38:02.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-30T13:38:02.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-30T13:38:02.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-30T13:38:02.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-30T13:38:02.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-30T13:38:02.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-30T13:38:02.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-30T13:38:02.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-30T13:38:02.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-30T13:38:02.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-30T13:38:02.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-30T13:38:02.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-30T13:38:02.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-30T13:38:02.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-30T13:38:02.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-30T13:38:02.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-30T13:38:02.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-30T13:38:02.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-30T13:38:02.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-30T13:38:02.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-30T13:38:02.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-30T13:38:02.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-30T13:38:02.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-30T13:38:02.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-30T13:38:02.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-30T13:38:02.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-30T13:38:02.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-30T13:38:02.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-30T13:38:02.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-30T13:38:02.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-30T13:38:02.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-30T13:38:02.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-30T13:38:02.145 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-30T13:38:02.878 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-30T13:38:02.878 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-30T13:38:02.878 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-30T13:38:02.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-30T13:38:02.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-30T13:38:02.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-30T13:38:02.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-30T13:38:02.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-30T13:38:02.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-30T13:38:02.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-30T13:38:02.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-30T13:38:02.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-30T13:38:02.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-30T13:38:02.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-30T13:38:02.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-30T13:38:02.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-30T13:38:02.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-30T13:38:02.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-30T13:38:02.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-30T13:38:02.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-30T13:38:02.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-30T13:38:02.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-30T13:38:02.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-30T13:38:02.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-30T13:38:02.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-30T13:38:02.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-30T13:38:02.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-30T13:38:02.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-30T13:38:02.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-30T13:38:02.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-30T13:38:02.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-30T13:38:02.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-30T13:38:02.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-30T13:38:02.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-30T13:38:02.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-30T13:38:02.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-30T13:38:02.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-30T13:38:02.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-30T13:38:02.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-30T13:38:02.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-30T13:38:02.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-30T13:38:02.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-30T13:38:02.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-30T13:38:02.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-30T13:38:02.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-30T13:38:02.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-30T13:38:02.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-30T13:38:02.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-30T13:38:02.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-30T13:38:02.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-30T13:38:02.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-30T13:38:02.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-30T13:38:02.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-30T13:38:02.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-30T13:38:02.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-30T13:38:02.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-30T13:38:02.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-30T13:38:02.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-30T13:38:02.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-30T13:38:02.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-30T13:38:02.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-30T13:38:02.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-30T13:38:02.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-30T13:38:02.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-30T13:38:02.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-30T13:38:02.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-30T13:38:02.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-30T13:38:02.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-30T13:38:02.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-30T13:38:02.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-30T13:38:02.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-30T13:38:02.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-30T13:38:02.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-30T13:38:02.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-30T13:38:02.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-30T13:38:02.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-30T13:38:02.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-30T13:38:02.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-30T13:38:02.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-30T13:38:02.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-30T13:38:02.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-30T13:38:03.784 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-30T13:38:03.784 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-30T13:38:03.784 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-30T13:38:03.785 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-30T13:38:03.785 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-30T13:38:03.785 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-30T13:38:03.785 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-30T13:38:03.785 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-30T13:38:03.786 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-30T13:38:03.786 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-30T13:38:03.786 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-30T13:38:03.786 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-30T13:38:03.786 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-30T13:38:03.787 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-30T13:38:03.787 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-30T13:38:03.787 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-30T13:38:03.787 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-30T13:38:03.787 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-30T13:38:03.788 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-30T13:38:03.788 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-30T13:38:03.788 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-30T13:38:03.788 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-30T13:38:03.788 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-30T13:38:03.789 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-30T13:38:03.789 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-30T13:38:03.789 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-30T13:38:03.789 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-30T13:38:03.789 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-30T13:38:03.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-30T13:38:03.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-30T13:38:03.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-30T13:38:03.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-30T13:38:03.790 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-30T13:38:03.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-30T13:38:03.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-30T13:38:03.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-30T13:38:03.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-30T13:38:03.791 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-30T13:38:03.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-30T13:38:03.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-30T13:38:03.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-30T13:38:03.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-30T13:38:03.792 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-30T13:38:03.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-30T13:38:03.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-30T13:38:03.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-30T13:38:03.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-30T13:38:03.793 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-30T13:38:03.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-30T13:38:03.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-30T13:38:03.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-30T13:38:03.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-30T13:38:03.794 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-30T13:38:03.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-30T13:38:03.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-30T13:38:03.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-30T13:38:03.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-30T13:38:03.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-30T13:38:03.795 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-30T13:38:03.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-30T13:38:03.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-30T13:38:03.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-30T13:38:03.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-30T13:38:03.796 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-30T13:38:03.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-30T13:38:03.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-30T13:38:03.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-30T13:38:03.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-30T13:38:03.797 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-30T13:38:03.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-30T13:38:03.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-30T13:38:03.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-30T13:38:03.798 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-30T13:38:03.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-30T13:38:03.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-30T13:38:03.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-30T13:38:03.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-30T13:38:03.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-30T13:38:03.799 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-30T13:38:03.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-30T13:38:03.800 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-30T13:38:14.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-30T13:38:14.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-30T13:38:14.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-30T13:38:14.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-30T13:38:14.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-30T13:38:14.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-30T13:38:14.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-30T13:38:14.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-30T13:38:14.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-30T13:38:14.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-30T13:38:14.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-30T13:38:14.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-30T13:38:14.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-30T13:38:14.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-30T13:38:14.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-30T13:38:14.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-30T13:38:14.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-30T13:38:14.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-30T13:38:14.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-30T13:38:14.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-30T13:38:14.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-30T13:38:14.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-30T13:38:14.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-30T13:38:14.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-30T13:38:14.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-30T13:38:14.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-30T13:38:14.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-30T13:38:14.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-30T13:38:14.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-30T13:38:14.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-30T13:38:14.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-30T13:38:14.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-30T13:38:14.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-30T13:38:14.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-30T13:38:14.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-30T13:38:14.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-30T13:38:14.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-30T13:38:14.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-30T13:38:14.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-30T13:38:14.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-30T13:38:14.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-30T13:38:14.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-30T13:38:14.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-30T13:38:14.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-30T13:38:14.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-30T13:38:14.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-30T13:38:14.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-30T13:38:14.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-30T13:38:14.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-30T13:38:14.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-30T13:38:14.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-30T13:38:14.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-30T13:38:14.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-30T13:38:14.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-30T13:38:14.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-30T13:38:14.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-30T13:38:14.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-30T13:38:14.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-30T13:38:14.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-30T13:38:14.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-30T13:38:14.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-30T13:38:14.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-30T13:38:14.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-30T13:38:14.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-30T13:38:14.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-30T13:38:14.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-30T13:38:14.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-30T13:38:14.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-30T13:38:14.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-30T13:38:14.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-30T13:38:14.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-30T13:38:14.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-30T13:38:14.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-30T13:38:14.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-30T13:38:14.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-30T13:38:14.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-30T13:38:14.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-30T13:38:14.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-30T13:38:14.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-30T13:38:14.895 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-30T13:38:14.895 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-30T13:38:25.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:25.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:51.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:51.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cedb8640 1 Processor -- start 2023-10-30T13:38:51.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cedb8640 1 -- start start 2023-10-30T13:38:51.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cedb8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151c20 0x7fb4c80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cedb8640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80c5030 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cedb8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c8157830 con 0x7fb4c8151840 2023-10-30T13:38:51.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cedb8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8157b10 con 0x7fb4c8151c20 2023-10-30T13:38:51.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cd355640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb4c80cb8b0 0x7fb4c806b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:50136/0 (socket says 172.21.15.31:50136) 2023-10-30T13:38:51.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.150+0000 7fb4cd355640 1 -- 172.21.15.31:0/4212523026 learned_addr learned my addr 172.21.15.31:0/4212523026 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:38:51.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c7fff640 1 --2- 172.21.15.31:0/4212523026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151c20 0x7fb4c80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415971108 0 0) 0x7fb4c80c5030 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b8003530 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222249236 0 0) 0x7fb4c8157830 con 0x7fb4c8151840 2023-10-30T13:38:51.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c80c5030 con 0x7fb4c8151840 2023-10-30T13:38:51.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3325117333 0 0) 0x7fb4b8003530 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c7fff640 1 -- 172.21.15.31:0/4212523026 >> v1:172.21.15.33:6789/0 conn(0x7fb4c8151840 legacy=0x7fb4c8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c7fff640 1 -- 172.21.15.31:0/4212523026 >> v1:172.21.15.31:6789/0 conn(0x7fb4c80cb8b0 legacy=0x7fb4c806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c7fff640 1 -- 172.21.15.31:0/4212523026 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c8157df0 con 0x7fb4c8151c20 2023-10-30T13:38:51.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.154+0000 7fb4c7fff640 1 --2- 172.21.15.31:0/4212523026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151c20 0x7fb4c80c49a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40097f0 tx=0x7fb4b401ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f36736d62cfce6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:51.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0001a00 con 0x7fb4c8151c20 2023-10-30T13:38:51.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b40a8650 con 0x7fb4c8151c20 2023-10-30T13:38:51.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4c77fe640 1 -- 172.21.15.31:0/4212523026 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b40a89c0 con 0x7fb4c8151c20 2023-10-30T13:38:51.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- 172.21.15.31:0/4212523026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151c20 msgr2=0x7fb4c80c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/4212523026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151c20 0x7fb4c80c49a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40097f0 tx=0x7fb4b401ebd0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- 172.21.15.31:0/4212523026 shutdown_connections 2023-10-30T13:38:51.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/4212523026 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151c20 0x7fb4c80c49a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- 172.21.15.31:0/4212523026 >> 172.21.15.31:0/4212523026 conn(0x7fb4c80cac70 msgr2=0x7fb4c80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:51.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- 172.21.15.31:0/4212523026 shutdown_connections 2023-10-30T13:38:51.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- 172.21.15.31:0/4212523026 wait complete. 2023-10-30T13:38:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 Processor -- start 2023-10-30T13:38:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- start start 2023-10-30T13:38:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4c80cb8b0 0x7fb4c8145990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151840 0x7fb4c81464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4c8151c20 0x7fb4c81469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c81e1940 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c81e1030 con 0x7fb4c8151c20 2023-10-30T13:38:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4cedb8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8157110 con 0x7fb4c8151840 2023-10-30T13:38:51.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4ccb54640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4c80cb8b0 0x7fb4c8145990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4ccb54640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4c80cb8b0 0x7fb4c8145990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53626/0 (socket says 172.21.15.31:53626) 2023-10-30T13:38:51.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.158+0000 7fb4ccb54640 1 -- 172.21.15.31:0/483272262 learned_addr learned my addr 172.21.15.31:0/483272262 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:38:51.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4c7fff640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151840 0x7fb4c81464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4cd355640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4c8151c20 0x7fb4c81469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4ccb54640 1 -- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151840 msgr2=0x7fb4c81464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4ccb54640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151840 0x7fb4c81464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4ccb54640 1 -- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4c8151c20 msgr2=0x7fb4c81469e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4ccb54640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4c8151c20 0x7fb4c81469e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4ccb54640 1 -- 172.21.15.31:0/483272262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4009420 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4c7fff640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151840 0x7fb4c81464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:38:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4ccb54640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4c80cb8b0 0x7fb4c8145990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00c490 tx=0x7fb4bc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4c57fa640 1 -- 172.21.15.31:0/483272262 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0052d0 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4c57fa640 1 -- 172.21.15.31:0/483272262 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc005430 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4c57fa640 1 -- 172.21.15.31:0/483272262 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc003840 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c81573f0 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c81e1560 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c81e1a80 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4c57fa640 1 -- 172.21.15.31:0/483272262 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4bc0039a0 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.162+0000 7fb4c57fa640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4ac0664d0 0x7fb4ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.166+0000 7fb4c57fa640 1 -- 172.21.15.31:0/483272262 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fb4bc083630 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.166+0000 7fb4c7fff640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4ac0664d0 0x7fb4ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.166+0000 7fb4c7fff640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4ac0664d0 0x7fb4ac068990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40b2000 tx=0x7fb4b40aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:51.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.170+0000 7fb4c57fa640 1 -- 172.21.15.31:0/483272262 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4c81e1a80 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4c81e1a80 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4c57fa640 1 -- 172.21.15.31:0/483272262 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7fb4c81e1a80 con 0x7fb4c80cb8b0 2023-10-30T13:38:51.366 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:38:51.366 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:51.367 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:38:51.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4ac0664d0 msgr2=0x7fb4ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4ac0664d0 0x7fb4ac068990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40b2000 tx=0x7fb4b40aa070 comp rx=0 tx=0).stop 2023-10-30T13:38:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4c80cb8b0 msgr2=0x7fb4c8145990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4c80cb8b0 0x7fb4c8145990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00c490 tx=0x7fb4bc00c950 comp rx=0 tx=0).stop 2023-10-30T13:38:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 shutdown_connections 2023-10-30T13:38:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4ac0664d0 0x7fb4ac068990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4c8151c20 0x7fb4c81469e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4c8151840 0x7fb4c81464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 --2- 172.21.15.31:0/483272262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4c80cb8b0 0x7fb4c8145990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 >> 172.21.15.31:0/483272262 conn(0x7fb4c80cac70 msgr2=0x7fb4c8072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 shutdown_connections 2023-10-30T13:38:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.362+0000 7fb4cedb8640 1 -- 172.21.15.31:0/483272262 wait complete. 2023-10-30T13:38:51.387 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=2) 2023-10-30T13:38:51.388 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:38:51.388 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:38:51.405 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:38:51.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:51.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.586+0000 7f25f5446640 1 Processor -- start 2023-10-30T13:38:51.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.586+0000 7f25f5446640 1 -- start start 2023-10-30T13:38:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.586+0000 7f25f5446640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f25f006c420 0x7f25f00c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.586+0000 7f25f5446640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 0x7f25f0159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.586+0000 7f25f5446640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f00c5f80 con 0x7f25f0153bb0 2023-10-30T13:38:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.586+0000 7f25f5446640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f00c5690 con 0x7f25f006c420 2023-10-30T13:38:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.586+0000 7f25f5446640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f015a580 con 0x7f25f00c7170 2023-10-30T13:38:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f25f0153bb0 0x7f25f00c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:50144/0 (socket says 172.21.15.31:50144) 2023-10-30T13:38:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 -- 172.21.15.31:0/2679431872 learned_addr learned my addr 172.21.15.31:0/2679431872 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:38:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25e7fff640 1 --2- 172.21.15.31:0/2679431872 >> v2:172.21.15.33:3300/0 conn(0x7f25f006c420 0x7f25f00c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ef7fe640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 0x7f25f0159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162418007 0 0) 0x7f25f00c5f80 con 0x7f25f0153bb0 2023-10-30T13:38:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc003530 con 0x7f25f0153bb0 2023-10-30T13:38:51.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229753200 0 0) 0x7f25dc003530 con 0x7f25f0153bb0 2023-10-30T13:38:51.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 msgr2=0x7f25f0159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:51.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 0x7f25f0159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 >> v2:172.21.15.33:3300/0 conn(0x7f25f006c420 msgr2=0x7f25f00c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 --2- 172.21.15.31:0/2679431872 >> v2:172.21.15.33:3300/0 conn(0x7f25f006c420 0x7f25f00c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f00c5930 con 0x7f25f0153bb0 2023-10-30T13:38:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f25e0001680 con 0x7f25f0153bb0 2023-10-30T13:38:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 >> v1:172.21.15.31:6789/0 conn(0x7f25f0153bb0 legacy=0x7f25f00c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25dc003980 0x7f25dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc0047d0 0x7f25dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 0x7f25dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00c5690 con 0x7f25dc003980 2023-10-30T13:38:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f015a580 con 0x7f25dc0047d0 2023-10-30T13:38:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc00b990 con 0x7f25f00c7170 2023-10-30T13:38:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25dc003980 0x7f25dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25e7fff640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc0047d0 0x7f25dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25ef7fe640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 0x7f25dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 -- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 msgr2=0x7f25dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f00c7170 0x7f25dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 -- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc0047d0 msgr2=0x7f25dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc0047d0 0x7f25dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25eeffd640 1 -- 172.21.15.31:0/2679431872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc003530 con 0x7f25dc003980 2023-10-30T13:38:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.590+0000 7f25e7fff640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc0047d0 0x7f25dc004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:38:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25eeffd640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25dc003980 0x7f25dc003da0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c290 tx=0x7f25e001f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0dde98caeaae92 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e00a8cd0 con 0x7f25dc003980 2023-10-30T13:38:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e00a8e30 con 0x7f25dc003980 2023-10-30T13:38:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25ee7fc640 1 -- 172.21.15.31:0/2679431872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0005b20 con 0x7f25dc003980 2023-10-30T13:38:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 -- 172.21.15.31:0/2679431872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25dc003980 msgr2=0x7f25dc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 --2- 172.21.15.31:0/2679431872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25dc003980 0x7f25dc003da0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c290 tx=0x7f25e001f430 comp rx=0 tx=0).stop 2023-10-30T13:38:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25ef7fe640 1 -- 172.21.15.31:0/2679431872 reap_dead start 2023-10-30T13:38:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 -- 172.21.15.31:0/2679431872 shutdown_connections 2023-10-30T13:38:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 -- 172.21.15.31:0/2679431872 >> 172.21.15.31:0/2679431872 conn(0x7f25f00cac70 msgr2=0x7f25f00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 -- 172.21.15.31:0/2679431872 shutdown_connections 2023-10-30T13:38:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 -- 172.21.15.31:0/2679431872 wait complete. 2023-10-30T13:38:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 Processor -- start 2023-10-30T13:38:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.594+0000 7f25f5446640 1 -- start start 2023-10-30T13:38:51.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc003980 0x7f25f01f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f01577f0 0x7f25f00c3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f0155660 0x7f25f014d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f01f3160 con 0x7f25f01577f0 2023-10-30T13:38:51.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f01f21d0 con 0x7f25dc003980 2023-10-30T13:38:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f01f2790 con 0x7f25f0155660 2023-10-30T13:38:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f01577f0 0x7f25f00c3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f01577f0 0x7f25f00c3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53646/0 (socket says 172.21.15.31:53646) 2023-10-30T13:38:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 -- 172.21.15.31:0/1927062610 learned_addr learned my addr 172.21.15.31:0/1927062610 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:38:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25eeffd640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc003980 0x7f25f01f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25ef7fe640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f0155660 0x7f25f014d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 -- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f0155660 msgr2=0x7f25f014d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f0155660 0x7f25f014d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 -- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc003980 msgr2=0x7f25f01f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc003980 0x7f25f01f2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 -- 172.21.15.31:0/1927062610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f01592d0 con 0x7f25f01577f0 2023-10-30T13:38:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25eeffd640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc003980 0x7f25f01f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:38:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e7fff640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f01577f0 0x7f25f00c3030 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009fd0 tx=0x7f25d8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e77fe640 1 -- 172.21.15.31:0/1927062610 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8004220 con 0x7f25f01577f0 2023-10-30T13:38:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e77fe640 1 -- 172.21.15.31:0/1927062610 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d8004380 con 0x7f25f01577f0 2023-10-30T13:38:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e77fe640 1 -- 172.21.15.31:0/1927062610 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8015890 con 0x7f25f01577f0 2023-10-30T13:38:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f0159590 con 0x7f25f01577f0 2023-10-30T13:38:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f01f2420 con 0x7f25f01577f0 2023-10-30T13:38:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f006bcd0 con 0x7f25f01577f0 2023-10-30T13:38:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.598+0000 7f25e77fe640 1 -- 172.21.15.31:0/1927062610 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25d80159f0 con 0x7f25f01577f0 2023-10-30T13:38:51.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.602+0000 7f25e77fe640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25d00664a0 0x7f25d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:51.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.602+0000 7f25e77fe640 1 -- 172.21.15.31:0/1927062610 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f25d8084370 con 0x7f25f01577f0 2023-10-30T13:38:51.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.602+0000 7f25eeffd640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25d00664a0 0x7f25d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:51.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.602+0000 7f25eeffd640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25d00664a0 0x7f25d0068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f25e00ad000 tx=0x7f25e0001a10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:51.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.606+0000 7f25e77fe640 1 -- 172.21.15.31:0/1927062610 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25d8051670 con 0x7f25f01577f0 2023-10-30T13:38:51.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25f004e170 con 0x7f25f01577f0 2023-10-30T13:38:51.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25e77fe640 1 -- 172.21.15.31:0/1927062610 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f25d8054c90 con 0x7f25f01577f0 2023-10-30T13:38:51.803 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:38:51.803 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:51.804 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:38:51.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25d00664a0 msgr2=0x7f25d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25d00664a0 0x7f25d0068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f25e00ad000 tx=0x7f25e0001a10 comp rx=0 tx=0).stop 2023-10-30T13:38:51.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f01577f0 msgr2=0x7f25f00c3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:51.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f01577f0 0x7f25f00c3030 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009fd0 tx=0x7f25d8011a60 comp rx=0 tx=0).stop 2023-10-30T13:38:51.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 shutdown_connections 2023-10-30T13:38:51.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25d00664a0 0x7f25d0068960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f25f0155660 0x7f25f014d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f01577f0 0x7f25f00c3030 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 --2- 172.21.15.31:0/1927062610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25dc003980 0x7f25f01f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:51.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 >> 172.21.15.31:0/1927062610 conn(0x7f25f00cac70 msgr2=0x7f25f00c6410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:51.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 shutdown_connections 2023-10-30T13:38:51.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:51.798+0000 7f25f5446640 1 -- 172.21.15.31:0/1927062610 wait complete. 2023-10-30T13:38:51.824 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:38:53.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:53.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3b3b5640 1 Processor -- start 2023-10-30T13:38:53.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3b3b5640 1 -- start start 2023-10-30T13:38:53.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3b3b5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae340c8eb0 0x7fae3415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3b3b5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fae3406aeb0 0x7fae34069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3b3b5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3415dcc0 con 0x7fae3415ac70 2023-10-30T13:38:53.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3b3b5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae34159800 con 0x7fae3406aeb0 2023-10-30T13:38:53.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3b3b5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae34159d80 con 0x7fae340c8eb0 2023-10-30T13:38:53.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae39bb2640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fae3415ac70 0x7fae3415a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:50150/0 (socket says 172.21.15.31:50150) 2023-10-30T13:38:53.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae39bb2640 1 -- 172.21.15.31:0/1555977823 learned_addr learned my addr 172.21.15.31:0/1555977823 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:38:53.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3a3b3640 1 --2- 172.21.15.31:0/1555977823 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae340c8eb0 0x7fae3415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.974+0000 7fae3abb4640 1 --2- 172.21.15.31:0/1555977823 >> v2:172.21.15.33:3300/0 conn(0x7fae3406aeb0 0x7fae34069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae393b1640 1 -- 172.21.15.31:0/1555977823 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174630539 0 0) 0x7fae24001120 con 0x7fae3415ac70 2023-10-30T13:38:53.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae393b1640 1 -- 172.21.15.31:0/1555977823 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae20003530 con 0x7fae3415ac70 2023-10-30T13:38:53.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae3a3b3640 1 -- 172.21.15.31:0/1555977823 >> v2:172.21.15.33:3300/0 conn(0x7fae3406aeb0 msgr2=0x7fae34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:53.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae393b1640 1 -- 172.21.15.31:0/1555977823 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61702651 0 0) 0x7fae20003530 con 0x7fae3415ac70 2023-10-30T13:38:53.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae3a3b3640 1 --2- 172.21.15.31:0/1555977823 >> v2:172.21.15.33:3300/0 conn(0x7fae3406aeb0 0x7fae34069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae3a3b3640 1 -- 172.21.15.31:0/1555977823 >> v1:172.21.15.31:6789/0 conn(0x7fae3415ac70 legacy=0x7fae3415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae3a3b3640 1 -- 172.21.15.31:0/1555977823 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3415d420 con 0x7fae340c8eb0 2023-10-30T13:38:53.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae3abb4640 1 --2- 172.21.15.31:0/1555977823 >> v2:172.21.15.33:3300/0 conn(0x7fae3406aeb0 0x7fae34069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:38:53.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae3a3b3640 1 --2- 172.21.15.31:0/1555977823 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae340c8eb0 0x7fae3415cd70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae300097c0 tx=0x7fae3001eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c360511e435d2591 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.978+0000 7fae393b1640 1 -- 172.21.15.31:0/1555977823 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24001810 con 0x7fae340c8eb0 2023-10-30T13:38:53.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae393b1640 1 -- 172.21.15.31:0/1555977823 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae300a85d0 con 0x7fae340c8eb0 2023-10-30T13:38:53.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae393b1640 1 -- 172.21.15.31:0/1555977823 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae300a8940 con 0x7fae340c8eb0 2023-10-30T13:38:53.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1555977823 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae340c8eb0 msgr2=0x7fae3415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1555977823 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae340c8eb0 0x7fae3415cd70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae300097c0 tx=0x7fae3001eb90 comp rx=0 tx=0).stop 2023-10-30T13:38:53.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1555977823 shutdown_connections 2023-10-30T13:38:53.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1555977823 >> v2:172.21.15.33:3300/0 conn(0x7fae3406aeb0 0x7fae34069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1555977823 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae340c8eb0 0x7fae3415cd70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1555977823 >> 172.21.15.31:0/1555977823 conn(0x7fae340cef00 msgr2=0x7fae340cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:53.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1555977823 shutdown_connections 2023-10-30T13:38:53.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1555977823 wait complete. 2023-10-30T13:38:53.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 Processor -- start 2023-10-30T13:38:53.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- start start 2023-10-30T13:38:53.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae3406aeb0 0x7fae341f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae340c8eb0 0x7fae34150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae3415ac70 0x7fae34152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae341e5bf0 con 0x7fae3406aeb0 2023-10-30T13:38:53.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae341e5070 con 0x7fae340c8eb0 2023-10-30T13:38:53.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3b3b5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae341f3100 con 0x7fae3415ac70 2023-10-30T13:38:53.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3a3b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae3406aeb0 0x7fae341f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3a3b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae3406aeb0 0x7fae341f2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53656/0 (socket says 172.21.15.31:53656) 2023-10-30T13:38:53.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3a3b3640 1 -- 172.21.15.31:0/1084351601 learned_addr learned my addr 172.21.15.31:0/1084351601 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:38:53.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae39bb2640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae340c8eb0 0x7fae34150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.982+0000 7fae3abb4640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae3415ac70 0x7fae34152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3a3b3640 1 -- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae3415ac70 msgr2=0x7fae34152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3a3b3640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae3415ac70 0x7fae34152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3a3b3640 1 -- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae340c8eb0 msgr2=0x7fae34150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3a3b3640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae340c8eb0 0x7fae34150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3a3b3640 1 -- 172.21.15.31:0/1084351601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c009a70 con 0x7fae3406aeb0 2023-10-30T13:38:53.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3a3b3640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae3406aeb0 0x7fae341f2bc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae30000c10 tx=0x7fae3001b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae2affd640 1 -- 172.21.15.31:0/1084351601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae300a9c60 con 0x7fae3406aeb0 2023-10-30T13:38:53.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae2affd640 1 -- 172.21.15.31:0/1084351601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae300a9dc0 con 0x7fae3406aeb0 2023-10-30T13:38:53.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae30009420 con 0x7fae3406aeb0 2023-10-30T13:38:53.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae2affd640 1 -- 172.21.15.31:0/1084351601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae30003820 con 0x7fae3406aeb0 2023-10-30T13:38:53.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae34150250 con 0x7fae3406aeb0 2023-10-30T13:38:53.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3404e170 con 0x7fae3406aeb0 2023-10-30T13:38:53.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae2affd640 1 -- 172.21.15.31:0/1084351601 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae300b7040 con 0x7fae3406aeb0 2023-10-30T13:38:53.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae2affd640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae180664d0 0x7fae18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae2affd640 1 -- 172.21.15.31:0/1084351601 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fae301216c0 con 0x7fae3406aeb0 2023-10-30T13:38:53.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.986+0000 7fae39bb2640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae180664d0 0x7fae18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.990+0000 7fae39bb2640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae180664d0 0x7fae18068990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fae341584e0 tx=0x7fae24009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:53.990+0000 7fae2affd640 1 -- 172.21.15.31:0/1084351601 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae300eea50 con 0x7fae3406aeb0 2023-10-30T13:38:54.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.178+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae340ca960 con 0x7fae3406aeb0 2023-10-30T13:38:54.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.178+0000 7fae2affd640 1 -- 172.21.15.31:0/1084351601 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7fae300b5090 con 0x7fae3406aeb0 2023-10-30T13:38:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:38:54.185 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:54.186 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:38:54.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae180664d0 msgr2=0x7fae18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:54.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae180664d0 0x7fae18068990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fae341584e0 tx=0x7fae24009040 comp rx=0 tx=0).stop 2023-10-30T13:38:54.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae3406aeb0 msgr2=0x7fae341f2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:54.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae3406aeb0 0x7fae341f2bc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae30000c10 tx=0x7fae3001b930 comp rx=0 tx=0).stop 2023-10-30T13:38:54.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 shutdown_connections 2023-10-30T13:38:54.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae180664d0 0x7fae18068990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:54.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae3415ac70 0x7fae34152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:54.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae340c8eb0 0x7fae34150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:54.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 --2- 172.21.15.31:0/1084351601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae3406aeb0 0x7fae341f2bc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:54.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 >> 172.21.15.31:0/1084351601 conn(0x7fae340cef00 msgr2=0x7fae340c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:54.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 shutdown_connections 2023-10-30T13:38:54.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:54.182+0000 7fae3b3b5640 1 -- 172.21.15.31:0/1084351601 wait complete. 2023-10-30T13:38:54.207 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:38:55.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:55.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:56.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca77f3e640 1 Processor -- start 2023-10-30T13:38:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca77f3e640 1 -- start start 2023-10-30T13:38:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca77f3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fca700c4a10 0x7fca7015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca77f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca77f3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca70159f20 con 0x7fca70153c80 2023-10-30T13:38:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca77f3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7015dec0 con 0x7fca700c4a10 2023-10-30T13:38:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca77f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca701599a0 con 0x7fca700c42a0 2023-10-30T13:38:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca754d9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fca700c4a10 0x7fca7015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca754d9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fca700c4a10 0x7fca7015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46130/0 (socket says 172.21.15.31:46130) 2023-10-30T13:38:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca754d9640 1 -- 172.21.15.31:0/3932387986 learned_addr learned my addr 172.21.15.31:0/3932387986 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:38:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca764db640 1 --2- 172.21.15.31:0/3932387986 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca754d9640 1 -- 172.21.15.31:0/3932387986 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 msgr2=0x7fca700ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca754d9640 1 --2- 172.21.15.31:0/3932387986 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca754d9640 1 -- 172.21.15.31:0/3932387986 >> v1:172.21.15.33:6789/0 conn(0x7fca70153c80 legacy=0x7fca7015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.358+0000 7fca754d9640 1 -- 172.21.15.31:0/3932387986 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7015d620 con 0x7fca700c4a10 2023-10-30T13:38:56.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca754d9640 1 --2- 172.21.15.31:0/3932387986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca700c4a10 0x7fca7015cf90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fca60009870 tx=0x7fca6001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fbebcde4800ff1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:56.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca74cd8640 1 -- 172.21.15.31:0/3932387986 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca600a8a30 con 0x7fca700c4a10 2023-10-30T13:38:56.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca74cd8640 1 -- 172.21.15.31:0/3932387986 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca600a8b90 con 0x7fca700c4a10 2023-10-30T13:38:56.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca74cd8640 1 -- 172.21.15.31:0/3932387986 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca600b0440 con 0x7fca700c4a10 2023-10-30T13:38:56.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 -- 172.21.15.31:0/3932387986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca700c4a10 msgr2=0x7fca7015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 --2- 172.21.15.31:0/3932387986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca700c4a10 0x7fca7015cf90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fca60009870 tx=0x7fca6001ed30 comp rx=0 tx=0).stop 2023-10-30T13:38:56.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 -- 172.21.15.31:0/3932387986 shutdown_connections 2023-10-30T13:38:56.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 --2- 172.21.15.31:0/3932387986 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 --2- 172.21.15.31:0/3932387986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca700c4a10 0x7fca7015cf90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 -- 172.21.15.31:0/3932387986 >> 172.21.15.31:0/3932387986 conn(0x7fca700715c0 msgr2=0x7fca700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 -- 172.21.15.31:0/3932387986 shutdown_connections 2023-10-30T13:38:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.362+0000 7fca77f3e640 1 -- 172.21.15.31:0/3932387986 wait complete. 2023-10-30T13:38:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 Processor -- start 2023-10-30T13:38:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 -- start start 2023-10-30T13:38:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca700c4a10 0x7fca70149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca70153c80 0x7fca7014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca701f1e30 con 0x7fca70153c80 2023-10-30T13:38:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca701f2110 con 0x7fca700c4a10 2023-10-30T13:38:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca701f23f0 con 0x7fca700c42a0 2023-10-30T13:38:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca70153c80 0x7fca7014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca70153c80 0x7fca7014a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46138/0 (socket says 172.21.15.31:46138) 2023-10-30T13:38:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 -- 172.21.15.31:0/2341066297 learned_addr learned my addr 172.21.15.31:0/2341066297 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:38:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca754d9640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca700c4a10 0x7fca70149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca75cda640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 -- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 msgr2=0x7fca700c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 -- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca700c4a10 msgr2=0x7fca70149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca700c4a10 0x7fca70149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 -- 172.21.15.31:0/2341066297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca600094d0 con 0x7fca70153c80 2023-10-30T13:38:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca764db640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca70153c80 0x7fca7014a070 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fca68007e40 tx=0x7fca6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca667fc640 1 -- 172.21.15.31:0/2341066297 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68014620 con 0x7fca70153c80 2023-10-30T13:38:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca667fc640 1 -- 172.21.15.31:0/2341066297 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6800ee90 con 0x7fca70153c80 2023-10-30T13:38:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca667fc640 1 -- 172.21.15.31:0/2341066297 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680149e0 con 0x7fca70153c80 2023-10-30T13:38:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca701f26d0 con 0x7fca70153c80 2023-10-30T13:38:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.366+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca701592b0 con 0x7fca70153c80 2023-10-30T13:38:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.370+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7004e170 con 0x7fca70153c80 2023-10-30T13:38:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.370+0000 7fca667fc640 1 -- 172.21.15.31:0/2341066297 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca68014b40 con 0x7fca70153c80 2023-10-30T13:38:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.370+0000 7fca667fc640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca540664a0 0x7fca54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.370+0000 7fca667fc640 1 -- 172.21.15.31:0/2341066297 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fca68084810 con 0x7fca70153c80 2023-10-30T13:38:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.370+0000 7fca75cda640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca540664a0 0x7fca54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.370+0000 7fca75cda640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca540664a0 0x7fca54068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca6c000f70 tx=0x7fca6c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.374+0000 7fca667fc640 1 -- 172.21.15.31:0/2341066297 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca68051c20 con 0x7fca70153c80 2023-10-30T13:38:56.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca7014dad0 con 0x7fca70153c80 2023-10-30T13:38:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca667fc640 1 -- 172.21.15.31:0/2341066297 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7fca68055240 con 0x7fca70153c80 2023-10-30T13:38:56.571 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:38:56.572 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:56.572 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:38:56.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca540664a0 msgr2=0x7fca54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca540664a0 0x7fca54068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca6c000f70 tx=0x7fca6c009040 comp rx=0 tx=0).stop 2023-10-30T13:38:56.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca70153c80 msgr2=0x7fca7014a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca70153c80 0x7fca7014a070 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fca68007e40 tx=0x7fca6800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 shutdown_connections 2023-10-30T13:38:56.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca540664a0 0x7fca54068960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca70153c80 0x7fca7014a070 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca700c4a10 0x7fca70149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 --2- 172.21.15.31:0/2341066297 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca700c42a0 0x7fca700c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.566+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 >> 172.21.15.31:0/2341066297 conn(0x7fca700715c0 msgr2=0x7fca701582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.570+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 shutdown_connections 2023-10-30T13:38:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:56.570+0000 7fca77f3e640 1 -- 172.21.15.31:0/2341066297 wait complete. 2023-10-30T13:38:56.593 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:38:58.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:58.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.742+0000 7f344a980640 1 Processor -- start 2023-10-30T13:38:58.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.742+0000 7f344a980640 1 -- start start 2023-10-30T13:38:58.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f344a980640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3444151c20 0x7f34440c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f344a980640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440c5030 con 0x7f34440cb8b0 2023-10-30T13:38:58.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f344a980640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444157830 con 0x7f3444151840 2023-10-30T13:38:58.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f344a980640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3444157b10 con 0x7f3444151c20 2023-10-30T13:38:58.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f34437fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3444151c20 0x7f34440c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f34437fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3444151c20 0x7f34440c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46148/0 (socket says 172.21.15.31:46148) 2023-10-30T13:38:58.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f34437fe640 1 -- 172.21.15.31:0/3863197429 learned_addr learned my addr 172.21.15.31:0/3863197429 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:38:58.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f3448f1d640 1 --1- 172.21.15.31:0/3863197429 >> v1:172.21.15.31:6789/0 conn(0x7f34440cb8b0 0x7f344406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46884/0 (socket says 172.21.15.31:46884) 2023-10-30T13:38:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f34437fe640 1 -- 172.21.15.31:0/3863197429 >> v1:172.21.15.33:6789/0 conn(0x7f3444151840 legacy=0x7f3444072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f34437fe640 1 -- 172.21.15.31:0/3863197429 >> v1:172.21.15.31:6789/0 conn(0x7f34440cb8b0 legacy=0x7f344406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f34437fe640 1 -- 172.21.15.31:0/3863197429 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444157df0 con 0x7f3444151c20 2023-10-30T13:38:58.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.746+0000 7f3442ffd640 1 -- 172.21.15.31:0/3863197429 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848988414 0 0) 0x7f34440c5030 con 0x7f34440cb8b0 2023-10-30T13:38:58.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f34437fe640 1 --2- 172.21.15.31:0/3863197429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151c20 0x7f34440c49a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3438009870 tx=0x7f343801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90e39974598fde6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:58.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f3442ffd640 1 -- 172.21.15.31:0/3863197429 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34380a8a30 con 0x7f3444151c20 2023-10-30T13:38:58.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f3442ffd640 1 -- 172.21.15.31:0/3863197429 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34380a8b90 con 0x7f3444151c20 2023-10-30T13:38:58.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f3442ffd640 1 -- 172.21.15.31:0/3863197429 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34380b0440 con 0x7f3444151c20 2023-10-30T13:38:58.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 -- 172.21.15.31:0/3863197429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151c20 msgr2=0x7f34440c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 --2- 172.21.15.31:0/3863197429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151c20 0x7f34440c49a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3438009870 tx=0x7f343801ed30 comp rx=0 tx=0).stop 2023-10-30T13:38:58.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 -- 172.21.15.31:0/3863197429 shutdown_connections 2023-10-30T13:38:58.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 --2- 172.21.15.31:0/3863197429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151c20 0x7f34440c49a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 -- 172.21.15.31:0/3863197429 >> 172.21.15.31:0/3863197429 conn(0x7f34440cac70 msgr2=0x7f34440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:58.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 -- 172.21.15.31:0/3863197429 shutdown_connections 2023-10-30T13:38:58.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 -- 172.21.15.31:0/3863197429 wait complete. 2023-10-30T13:38:58.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 Processor -- start 2023-10-30T13:38:58.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.750+0000 7f344a980640 1 -- start start 2023-10-30T13:38:58.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34440cb8b0 0x7f344414b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151840 0x7f344414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444151c20 0x7f3444152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34441eec20 con 0x7f3444151840 2023-10-30T13:38:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34441efbf0 con 0x7f3444151c20 2023-10-30T13:38:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34441f2f00 con 0x7f34440cb8b0 2023-10-30T13:38:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151840 0x7f344414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151840 0x7f344414daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46150/0 (socket says 172.21.15.31:46150) 2023-10-30T13:38:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 -- 172.21.15.31:0/1521468914 learned_addr learned my addr 172.21.15.31:0/1521468914 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:38:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f3443fff640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34440cb8b0 0x7f344414b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f3448f1d640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444151c20 0x7f3444152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 -- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34440cb8b0 msgr2=0x7f344414b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34440cb8b0 0x7f344414b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 -- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444151c20 msgr2=0x7f3444152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444151c20 0x7f3444152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 -- 172.21.15.31:0/1521468914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380094d0 con 0x7f3444151840 2023-10-30T13:38:58.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f3448f1d640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444151c20 0x7f3444152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:38:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f34437fe640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151840 0x7f344414daf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f343801b940 tx=0x7f343801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f3440ff9640 1 -- 172.21.15.31:0/1521468914 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34380ba600 con 0x7f3444151840 2023-10-30T13:38:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f3440ff9640 1 -- 172.21.15.31:0/1521468914 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34380ba780 con 0x7f3444151840 2023-10-30T13:38:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34441eee30 con 0x7f3444151840 2023-10-30T13:38:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f3440ff9640 1 -- 172.21.15.31:0/1521468914 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34380bab30 con 0x7f3444151840 2023-10-30T13:38:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34441ef3f0 con 0x7f3444151840 2023-10-30T13:38:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344404e170 con 0x7f3444151840 2023-10-30T13:38:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.754+0000 7f3440ff9640 1 -- 172.21.15.31:0/1521468914 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34380bac90 con 0x7f3444151840 2023-10-30T13:38:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.758+0000 7f3440ff9640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34300664a0 0x7f3430068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.758+0000 7f3440ff9640 1 -- 172.21.15.31:0/1521468914 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f34381224e0 con 0x7f3444151840 2023-10-30T13:38:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.758+0000 7f3443fff640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34300664a0 0x7f3430068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.758+0000 7f3443fff640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34300664a0 0x7f3430068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3434007fd0 tx=0x7f3434009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:58.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.762+0000 7f3440ff9640 1 -- 172.21.15.31:0/1521468914 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34380ef900 con 0x7f3444151840 2023-10-30T13:38:58.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.950+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3444066510 con 0x7f3444151840 2023-10-30T13:38:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.950+0000 7f3440ff9640 1 -- 172.21.15.31:0/1521468914 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f34380f2f20 con 0x7f3444151840 2023-10-30T13:38:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:38:58.956 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:58.957 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:38:58.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34300664a0 msgr2=0x7f3430068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34300664a0 0x7f3430068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3434007fd0 tx=0x7f3434009040 comp rx=0 tx=0).stop 2023-10-30T13:38:58.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151840 msgr2=0x7f344414daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151840 0x7f344414daf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f343801b940 tx=0x7f343801bad0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 shutdown_connections 2023-10-30T13:38:58.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34300664a0 0x7f3430068960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444151c20 0x7f3444152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444151840 0x7f344414daf0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 --2- 172.21.15.31:0/1521468914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34440cb8b0 0x7f344414b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 >> 172.21.15.31:0/1521468914 conn(0x7f34440cac70 msgr2=0x7f34440c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:58.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 shutdown_connections 2023-10-30T13:38:58.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:38:58.954+0000 7f344a980640 1 -- 172.21.15.31:0/1521468914 wait complete. 2023-10-30T13:38:58.977 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:00.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:01.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f82722d6640 1 Processor -- start 2023-10-30T13:39:01.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f82722d6640 1 -- start start 2023-10-30T13:39:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f82722d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f826c153c80 0x7f826c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f82722d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c42a0 0x7f826c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f82722d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c159f20 con 0x7f826c0c4a10 2023-10-30T13:39:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f82722d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c15dec0 con 0x7f826c153c80 2023-10-30T13:39:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f82722d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c1599a0 con 0x7f826c0c42a0 2023-10-30T13:39:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f826bfff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f826c153c80 0x7f826c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f8270873640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c42a0 0x7f826c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f826bfff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f826c153c80 0x7f826c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:37406/0 (socket says 172.21.15.31:37406) 2023-10-30T13:39:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f826bfff640 1 -- 172.21.15.31:0/3216233301 learned_addr learned my addr 172.21.15.31:0/3216233301 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f826bfff640 1 -- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c42a0 msgr2=0x7f826c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f826bfff640 1 --2- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c42a0 0x7f826c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f826bfff640 1 -- 172.21.15.31:0/3216233301 >> v1:172.21.15.33:6789/0 conn(0x7f826c0c4a10 legacy=0x7f826c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:01.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f826bfff640 1 -- 172.21.15.31:0/3216233301 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c15d620 con 0x7f826c153c80 2023-10-30T13:39:01.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.126+0000 7f8270873640 1 --2- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c42a0 0x7f826c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:39:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f826bfff640 1 --2- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 0x7f826c15cf90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f825c009870 tx=0x7f825c01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35f73d4b9dd614fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f826affd640 1 -- 172.21.15.31:0/3216233301 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0a8a30 con 0x7f826c153c80 2023-10-30T13:39:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f826affd640 1 -- 172.21.15.31:0/3216233301 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c0a8b90 con 0x7f826c153c80 2023-10-30T13:39:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f826affd640 1 -- 172.21.15.31:0/3216233301 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0b0440 con 0x7f826c153c80 2023-10-30T13:39:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 -- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 msgr2=0x7f826c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 --2- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 0x7f826c15cf90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f825c009870 tx=0x7f825c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:39:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 -- 172.21.15.31:0/3216233301 shutdown_connections 2023-10-30T13:39:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 --2- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c42a0 0x7f826c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 --2- 172.21.15.31:0/3216233301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 0x7f826c15cf90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 -- 172.21.15.31:0/3216233301 >> 172.21.15.31:0/3216233301 conn(0x7f826c0715c0 msgr2=0x7f826c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 -- 172.21.15.31:0/3216233301 shutdown_connections 2023-10-30T13:39:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.130+0000 7f82722d6640 1 -- 172.21.15.31:0/3216233301 wait complete. 2023-10-30T13:39:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 Processor -- start 2023-10-30T13:39:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 -- start start 2023-10-30T13:39:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0c42a0 0x7f826c156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c4a10 0x7f826c155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 0x7f826c155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:01.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c156d60 con 0x7f826c0c42a0 2023-10-30T13:39:01.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c154d80 con 0x7f826c153c80 2023-10-30T13:39:01.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c157380 con 0x7f826c0c4a10 2023-10-30T13:39:01.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0c42a0 0x7f826c156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:01.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0c42a0 0x7f826c156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46166/0 (socket says 172.21.15.31:46166) 2023-10-30T13:39:01.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 -- 172.21.15.31:0/215708428 learned_addr learned my addr 172.21.15.31:0/215708428 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:01.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826b7fe640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c4a10 0x7f826c155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:01.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f8270873640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 0x7f826c155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:01.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 -- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c4a10 msgr2=0x7f826c155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:01.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c4a10 0x7f826c155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 -- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 msgr2=0x7f826c155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:01.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 0x7f826c155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 -- 172.21.15.31:0/215708428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8264009a70 con 0x7f826c0c42a0 2023-10-30T13:39:01.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f826bfff640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0c42a0 0x7f826c156820 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f825c0b3040 tx=0x7f825c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:01.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f8268ff9640 1 -- 172.21.15.31:0/215708428 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0b0dd0 con 0x7f826c0c42a0 2023-10-30T13:39:01.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f8268ff9640 1 -- 172.21.15.31:0/215708428 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c0bb460 con 0x7f826c0c42a0 2023-10-30T13:39:01.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f8268ff9640 1 -- 172.21.15.31:0/215708428 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0bb810 con 0x7f826c0c42a0 2023-10-30T13:39:01.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.134+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c0094d0 con 0x7f826c0c42a0 2023-10-30T13:39:01.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.138+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c157e80 con 0x7f826c0c42a0 2023-10-30T13:39:01.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.138+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c1564f0 con 0x7f826c0c42a0 2023-10-30T13:39:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.138+0000 7f8268ff9640 1 -- 172.21.15.31:0/215708428 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f825c0bb970 con 0x7f826c0c42a0 2023-10-30T13:39:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.138+0000 7f8268ff9640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82580664a0 0x7f8258068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.138+0000 7f8268ff9640 1 -- 172.21.15.31:0/215708428 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f825c121650 con 0x7f826c0c42a0 2023-10-30T13:39:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.138+0000 7f826b7fe640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82580664a0 0x7f8258068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.138+0000 7f826b7fe640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82580664a0 0x7f8258068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f826c0ca7b0 tx=0x7f8260009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:01.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.142+0000 7f8268ff9640 1 -- 172.21.15.31:0/215708428 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f826c1564f0 con 0x7f826c0c42a0 2023-10-30T13:39:01.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.334+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f826c1564f0 con 0x7f826c0c42a0 2023-10-30T13:39:01.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f8268ff9640 1 -- 172.21.15.31:0/215708428 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f826c1564f0 con 0x7f826c0c42a0 2023-10-30T13:39:01.342 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:01.342 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:01.342 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:39:01.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82580664a0 msgr2=0x7f8258068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:01.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82580664a0 0x7f8258068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f826c0ca7b0 tx=0x7f8260009040 comp rx=0 tx=0).stop 2023-10-30T13:39:01.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0c42a0 msgr2=0x7f826c156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:01.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0c42a0 0x7f826c156820 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f825c0b3040 tx=0x7f825c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 shutdown_connections 2023-10-30T13:39:01.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82580664a0 0x7f8258068960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826c153c80 0x7f826c155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f826c0c4a10 0x7f826c155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 --2- 172.21.15.31:0/215708428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0c42a0 0x7f826c156820 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 >> 172.21.15.31:0/215708428 conn(0x7f826c0715c0 msgr2=0x7f826c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 shutdown_connections 2023-10-30T13:39:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:01.338+0000 7f82722d6640 1 -- 172.21.15.31:0/215708428 wait complete. 2023-10-30T13:39:01.363 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:03.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.514+0000 7f7a2e417640 1 Processor -- start 2023-10-30T13:39:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.514+0000 7f7a2e417640 1 -- start start 2023-10-30T13:39:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2e417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a280c7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2e417640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7a280c5fc0 0x7f7a2815cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2e417640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2815d320 con 0x7f7a2806a4d0 2023-10-30T13:39:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2e417640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2815a030 con 0x7f7a280c5fc0 2023-10-30T13:39:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2e417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2806f670 con 0x7f7a2806a0f0 2023-10-30T13:39:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2c9b4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7a280c5fc0 0x7f7a2815cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a280c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2c9b4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7a280c5fc0 0x7f7a2815cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:37422/0 (socket says 172.21.15.31:37422) 2023-10-30T13:39:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2c9b4640 1 -- 172.21.15.31:0/1732202217 learned_addr learned my addr 172.21.15.31:0/1732202217 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a277fe640 1 --1- 172.21.15.31:0/1732202217 >> v1:172.21.15.33:6789/0 conn(0x7f7a2806a4d0 0x7f7a280c58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:60650/0 (socket says 172.21.15.31:60650) 2023-10-30T13:39:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2c9b4640 1 -- 172.21.15.31:0/1732202217 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 msgr2=0x7f7a280c7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2c9b4640 1 --2- 172.21.15.31:0/1732202217 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a280c7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2c9b4640 1 -- 172.21.15.31:0/1732202217 >> v1:172.21.15.33:6789/0 conn(0x7f7a2806a4d0 legacy=0x7f7a280c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.518+0000 7f7a2c9b4640 1 -- 172.21.15.31:0/1732202217 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2806f0b0 con 0x7f7a280c5fc0 2023-10-30T13:39:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2c9b4640 1 --2- 172.21.15.31:0/1732202217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 0x7f7a2815cde0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009e50 tx=0x7f7a2001ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbb8aecc44d7fa5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a26ffd640 1 -- 172.21.15.31:0/1732202217 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a200a57d0 con 0x7f7a280c5fc0 2023-10-30T13:39:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a26ffd640 1 -- 172.21.15.31:0/1732202217 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a200a5930 con 0x7f7a280c5fc0 2023-10-30T13:39:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a26ffd640 1 -- 172.21.15.31:0/1732202217 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a200a5cc0 con 0x7f7a280c5fc0 2023-10-30T13:39:03.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 -- 172.21.15.31:0/1732202217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 msgr2=0x7f7a2815cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 --2- 172.21.15.31:0/1732202217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 0x7f7a2815cde0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009e50 tx=0x7f7a2001ca10 comp rx=0 tx=0).stop 2023-10-30T13:39:03.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 -- 172.21.15.31:0/1732202217 shutdown_connections 2023-10-30T13:39:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 --2- 172.21.15.31:0/1732202217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 0x7f7a2815cde0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 --2- 172.21.15.31:0/1732202217 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a280c7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 -- 172.21.15.31:0/1732202217 >> 172.21.15.31:0/1732202217 conn(0x7f7a2806afa0 msgr2=0x7f7a2806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 -- 172.21.15.31:0/1732202217 shutdown_connections 2023-10-30T13:39:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 -- 172.21.15.31:0/1732202217 wait complete. 2023-10-30T13:39:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 Processor -- start 2023-10-30T13:39:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.522+0000 7f7a2e417640 1 -- start start 2023-10-30T13:39:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a281f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a2806a4d0 0x7f7a2815c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 0x7f7a2815cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280ae1e0 con 0x7f7a2806a4d0 2023-10-30T13:39:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28070ee0 con 0x7f7a280c5fc0 2023-10-30T13:39:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2815bfe0 con 0x7f7a2806a0f0 2023-10-30T13:39:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a2806a4d0 0x7f7a2815c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a2806a4d0 0x7f7a2815c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46170/0 (socket says 172.21.15.31:46170) 2023-10-30T13:39:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 -- 172.21.15.31:0/4185780706 learned_addr learned my addr 172.21.15.31:0/4185780706 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a27fff640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a281f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2c9b4640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 0x7f7a2815cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 -- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 msgr2=0x7f7a281f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a281f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 -- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 msgr2=0x7f7a2815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 0x7f7a2815cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 -- 172.21.15.31:0/4185780706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a20009b30 con 0x7f7a2806a4d0 2023-10-30T13:39:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a277fe640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a2806a4d0 0x7f7a2815c710 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00c340 tx=0x7f7a1c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a24ff9640 1 -- 172.21.15.31:0/4185780706 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0074e0 con 0x7f7a2806a4d0 2023-10-30T13:39:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a24ff9640 1 -- 172.21.15.31:0/4185780706 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c007640 con 0x7f7a2806a4d0 2023-10-30T13:39:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a24ff9640 1 -- 172.21.15.31:0/4185780706 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c004a30 con 0x7f7a2806a4d0 2023-10-30T13:39:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2815ae50 con 0x7f7a2806a4d0 2023-10-30T13:39:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2815b0f0 con 0x7f7a2806a4d0 2023-10-30T13:39:03.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a280cbbf0 con 0x7f7a2806a4d0 2023-10-30T13:39:03.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.526+0000 7f7a24ff9640 1 -- 172.21.15.31:0/4185780706 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a1c004b90 con 0x7f7a2806a4d0 2023-10-30T13:39:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.530+0000 7f7a24ff9640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a140664d0 0x7f7a14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.530+0000 7f7a24ff9640 1 -- 172.21.15.31:0/4185780706 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f7a1c083820 con 0x7f7a2806a4d0 2023-10-30T13:39:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.530+0000 7f7a27fff640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a140664d0 0x7f7a14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.530+0000 7f7a27fff640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a140664d0 0x7f7a14068990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009ac0 tx=0x7f7a18009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.534+0000 7f7a24ff9640 1 -- 172.21.15.31:0/4185780706 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a1c050bd0 con 0x7f7a2806a4d0 2023-10-30T13:39:03.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.722+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a2804e170 con 0x7f7a2806a4d0 2023-10-30T13:39:03.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.722+0000 7f7a24ff9640 1 -- 172.21.15.31:0/4185780706 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f7a1c0541f0 con 0x7f7a2806a4d0 2023-10-30T13:39:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:03.729 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:39:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a140664d0 msgr2=0x7f7a14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a140664d0 0x7f7a14068990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009ac0 tx=0x7f7a18009190 comp rx=0 tx=0).stop 2023-10-30T13:39:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a2806a4d0 msgr2=0x7f7a2815c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a2806a4d0 0x7f7a2815c710 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00c340 tx=0x7f7a1c00c800 comp rx=0 tx=0).stop 2023-10-30T13:39:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 shutdown_connections 2023-10-30T13:39:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a140664d0 0x7f7a14068990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a280c5fc0 0x7f7a2815cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a2806a4d0 0x7f7a2815c710 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 --2- 172.21.15.31:0/4185780706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a2806a0f0 0x7f7a281f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 >> 172.21.15.31:0/4185780706 conn(0x7f7a2806afa0 msgr2=0x7f7a280c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:03.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 shutdown_connections 2023-10-30T13:39:03.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:03.726+0000 7f7a2e417640 1 -- 172.21.15.31:0/4185780706 wait complete. 2023-10-30T13:39:03.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:05.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:05.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 Processor -- start 2023-10-30T13:39:05.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 -- start start 2023-10-30T13:39:05.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f01cc0c4a10 0x7f01cc0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f01cc15a3b0 0x7f01cc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01cc15dec0 con 0x7f01cc15a3b0 2023-10-30T13:39:05.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01cc1599a0 con 0x7f01cc0c4a10 2023-10-30T13:39:05.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d409e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc159f20 con 0x7f01cc153c80 2023-10-30T13:39:05.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f01cc15a3b0 0x7f01cc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f01cc15a3b0 0x7f01cc0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59428/0 (socket says 172.21.15.31:59428) 2023-10-30T13:39:05.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 -- 172.21.15.31:0/474712329 learned_addr learned my addr 172.21.15.31:0/474712329 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:05.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d1639640 1 --2- 172.21.15.31:0/474712329 >> v2:172.21.15.33:3300/0 conn(0x7f01cc0c4a10 0x7f01cc0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d1e3a640 1 --2- 172.21.15.31:0/474712329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 -- 172.21.15.31:0/474712329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 msgr2=0x7f01cc15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:05.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 --2- 172.21.15.31:0/474712329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 -- 172.21.15.31:0/474712329 >> v2:172.21.15.33:3300/0 conn(0x7f01cc0c4a10 msgr2=0x7f01cc0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:05.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 --2- 172.21.15.31:0/474712329 >> v2:172.21.15.33:3300/0 conn(0x7f01cc0c4a10 0x7f01cc0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.902+0000 7f01d263b640 1 -- 172.21.15.31:0/474712329 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc0c47e0 con 0x7f01cc15a3b0 2023-10-30T13:39:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d263b640 1 --2- 172.21.15.31:0/474712329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15a3b0 0x7f01cc0ca1e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f01cc08cc20 tx=0x7f01c401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449eeeb7aa1a3487 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d0e38640 1 -- 172.21.15.31:0/474712329 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c40af400 con 0x7f01cc15a3b0 2023-10-30T13:39:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d0e38640 1 -- 172.21.15.31:0/474712329 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c40ab050 con 0x7f01cc15a3b0 2023-10-30T13:39:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d0e38640 1 -- 172.21.15.31:0/474712329 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c40af780 con 0x7f01cc15a3b0 2023-10-30T13:39:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 -- 172.21.15.31:0/474712329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15a3b0 msgr2=0x7f01cc0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 --2- 172.21.15.31:0/474712329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15a3b0 0x7f01cc0ca1e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f01cc08cc20 tx=0x7f01c401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 -- 172.21.15.31:0/474712329 shutdown_connections 2023-10-30T13:39:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 --2- 172.21.15.31:0/474712329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15a3b0 0x7f01cc0ca1e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 --2- 172.21.15.31:0/474712329 >> v2:172.21.15.33:3300/0 conn(0x7f01cc0c4a10 0x7f01cc0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 --2- 172.21.15.31:0/474712329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 -- 172.21.15.31:0/474712329 >> 172.21.15.31:0/474712329 conn(0x7f01cc0715c0 msgr2=0x7f01cc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 -- 172.21.15.31:0/474712329 shutdown_connections 2023-10-30T13:39:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.906+0000 7f01d409e640 1 -- 172.21.15.31:0/474712329 wait complete. 2023-10-30T13:39:05.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 Processor -- start 2023-10-30T13:39:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 -- start start 2023-10-30T13:39:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc0c4a10 0x7f01cc1f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc1ef420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01cc15a3b0 0x7f01cc14f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc14ad60 con 0x7f01cc0c4a10 2023-10-30T13:39:05.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1ef000 con 0x7f01cc15a3b0 2023-10-30T13:39:05.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0ce3b0 con 0x7f01cc153c80 2023-10-30T13:39:05.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc0c4a10 0x7f01cc1f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc0c4a10 0x7f01cc1f2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59434/0 (socket says 172.21.15.31:59434) 2023-10-30T13:39:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 -- 172.21.15.31:0/1558053312 learned_addr learned my addr 172.21.15.31:0/1558053312 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d263b640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01cc15a3b0 0x7f01cc14f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1639640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc1ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 -- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 msgr2=0x7f01cc1ef420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc1ef420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 -- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01cc15a3b0 msgr2=0x7f01cc14f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01cc15a3b0 0x7f01cc14f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 -- 172.21.15.31:0/1558053312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c4009b30 con 0x7f01cc0c4a10 2023-10-30T13:39:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d1e3a640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc0c4a10 0x7f01cc1f2030 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f01c800d800 tx=0x7f01c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01c27fc640 1 -- 172.21.15.31:0/1558053312 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800b6c0 con 0x7f01cc0c4a10 2023-10-30T13:39:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01c27fc640 1 -- 172.21.15.31:0/1558053312 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c800b820 con 0x7f01cc0c4a10 2023-10-30T13:39:05.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01c27fc640 1 -- 172.21.15.31:0/1558053312 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800bb90 con 0x7f01cc0c4a10 2023-10-30T13:39:05.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01cc1f2f20 con 0x7f01cc0c4a10 2023-10-30T13:39:05.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.910+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc1f2960 con 0x7f01cc0c4a10 2023-10-30T13:39:05.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.914+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc04e170 con 0x7f01cc0c4a10 2023-10-30T13:39:05.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.914+0000 7f01c27fc640 1 -- 172.21.15.31:0/1558053312 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01c800bcf0 con 0x7f01cc0c4a10 2023-10-30T13:39:05.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.914+0000 7f01c27fc640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.914+0000 7f01c27fc640 1 -- 172.21.15.31:0/1558053312 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f01c8084620 con 0x7f01cc0c4a10 2023-10-30T13:39:05.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.914+0000 7f01d1639640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.914+0000 7f01d1639640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01cc158b60 tx=0x7f01bc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:05.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:05.918+0000 7f01c27fc640 1 -- 172.21.15.31:0/1558053312 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01c8051890 con 0x7f01cc0c4a10 2023-10-30T13:39:06.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.110+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01cc0cad40 con 0x7f01cc0c4a10 2023-10-30T13:39:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01c27fc640 1 -- 172.21.15.31:0/1558053312 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f01c8054eb0 con 0x7f01cc0c4a10 2023-10-30T13:39:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:06.118 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:37:24.337560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":30,"addr":"172.21.15.31:6837/1956809670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1956809670},{"type":"v1","addr":"172.21.15.31:6837","nonce":1956809670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:06.118 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 21 2023-10-30T13:39:06.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 msgr2=0x7f01b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01cc158b60 tx=0x7f01bc008040 comp rx=0 tx=0).stop 2023-10-30T13:39:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc0c4a10 msgr2=0x7f01cc1f2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc0c4a10 0x7f01cc1f2030 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f01c800d800 tx=0x7f01c800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:39:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 shutdown_connections 2023-10-30T13:39:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:06.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01cc15a3b0 0x7f01cc14f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:06.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc153c80 0x7f01cc1ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:06.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 --2- 172.21.15.31:0/1558053312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc0c4a10 0x7f01cc1f2030 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:06.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 >> 172.21.15.31:0/1558053312 conn(0x7f01cc0715c0 msgr2=0x7f01cc0c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:06.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 shutdown_connections 2023-10-30T13:39:06.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:06.114+0000 7f01d409e640 1 -- 172.21.15.31:0/1558053312 wait complete. 2023-10-30T13:39:06.139 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:08.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:08.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.290+0000 7f79916ea640 1 Processor -- start 2023-10-30T13:39:08.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.290+0000 7f79916ea640 1 -- start start 2023-10-30T13:39:08.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f79916ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f79916ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f798c0701e0 0x7f798c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f79916ea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0c7b00 con 0x7f798c0cd420 2023-10-30T13:39:08.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f79916ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798c070a50 con 0x7f798c0701e0 2023-10-30T13:39:08.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f79916ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c15dd80 con 0x7f798c153aa0 2023-10-30T13:39:08.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f798c0cd420 0x7f798c06b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58794/0 (socket says 172.21.15.31:58794) 2023-10-30T13:39:08.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798b7fe640 1 -- 172.21.15.31:0/1292298892 learned_addr learned my addr 172.21.15.31:0/1292298892 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:08.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798affd640 1 --2- 172.21.15.31:0/1292298892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f7983fff640 1 --2- 172.21.15.31:0/1292298892 >> v2:172.21.15.33:3300/0 conn(0x7f798c0701e0 0x7f798c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798a7fc640 1 -- 172.21.15.31:0/1292298892 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803159755 0 0) 0x7f798c0c7b00 con 0x7f798c0cd420 2023-10-30T13:39:08.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798a7fc640 1 -- 172.21.15.31:0/1292298892 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7978003530 con 0x7f798c0cd420 2023-10-30T13:39:08.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798affd640 1 -- 172.21.15.31:0/1292298892 >> v2:172.21.15.33:3300/0 conn(0x7f798c0701e0 msgr2=0x7f798c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:08.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798affd640 1 --2- 172.21.15.31:0/1292298892 >> v2:172.21.15.33:3300/0 conn(0x7f798c0701e0 0x7f798c0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798affd640 1 -- 172.21.15.31:0/1292298892 >> v1:172.21.15.31:6789/0 conn(0x7f798c0cd420 legacy=0x7f798c06b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f798affd640 1 -- 172.21.15.31:0/1292298892 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0c2050 con 0x7f798c153aa0 2023-10-30T13:39:08.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.294+0000 7f7983fff640 1 --2- 172.21.15.31:0/1292298892 >> v2:172.21.15.33:3300/0 conn(0x7f798c0701e0 0x7f798c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:39:08.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f798affd640 1 --2- 172.21.15.31:0/1292298892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c15d840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f797c0097c0 tx=0x7f797c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7c5a21141129741 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:08.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f798a7fc640 1 -- 172.21.15.31:0/1292298892 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c0a8970 con 0x7f798c153aa0 2023-10-30T13:39:08.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f798a7fc640 1 -- 172.21.15.31:0/1292298892 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c0a8ad0 con 0x7f798c153aa0 2023-10-30T13:39:08.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f798a7fc640 1 -- 172.21.15.31:0/1292298892 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c0a8e60 con 0x7f798c153aa0 2023-10-30T13:39:08.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 -- 172.21.15.31:0/1292298892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 msgr2=0x7f798c15d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 --2- 172.21.15.31:0/1292298892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c15d840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f797c0097c0 tx=0x7f797c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:39:08.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 -- 172.21.15.31:0/1292298892 shutdown_connections 2023-10-30T13:39:08.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 --2- 172.21.15.31:0/1292298892 >> v2:172.21.15.33:3300/0 conn(0x7f798c0701e0 0x7f798c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 --2- 172.21.15.31:0/1292298892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c15d840 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 -- 172.21.15.31:0/1292298892 >> 172.21.15.31:0/1292298892 conn(0x7f798c071fb0 msgr2=0x7f798c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:08.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 -- 172.21.15.31:0/1292298892 shutdown_connections 2023-10-30T13:39:08.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 -- 172.21.15.31:0/1292298892 wait complete. 2023-10-30T13:39:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 Processor -- start 2023-10-30T13:39:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.298+0000 7f79916ea640 1 -- start start 2023-10-30T13:39:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f798c0701e0 0x7f798c0ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0cd420 0x7f798c15d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c0c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0c6b90 con 0x7f798c0cd420 2023-10-30T13:39:08.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0c6e70 con 0x7f798c0701e0 2023-10-30T13:39:08.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0c7150 con 0x7f798c153aa0 2023-10-30T13:39:08.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0cd420 0x7f798c15d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0cd420 0x7f798c15d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59446/0 (socket says 172.21.15.31:59446) 2023-10-30T13:39:08.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 -- 172.21.15.31:0/2851567549 learned_addr learned my addr 172.21.15.31:0/2851567549 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:08.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f798b7fe640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c0c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f798affd640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f798c0701e0 0x7f798c0ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 -- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 msgr2=0x7f798c0c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c0c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 -- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f798c0701e0 msgr2=0x7f798c0ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f798c0701e0 0x7f798c0ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 -- 172.21.15.31:0/2851567549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7974009420 con 0x7f798c0cd420 2023-10-30T13:39:08.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f798affd640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f798c0701e0 0x7f798c0ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:08.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f7983fff640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0cd420 0x7f798c15d4d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f797400ebe0 tx=0x7f797400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:08.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79837fe640 1 -- 172.21.15.31:0/2851567549 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797400eee0 con 0x7f798c0cd420 2023-10-30T13:39:08.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79837fe640 1 -- 172.21.15.31:0/2851567549 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797400cc30 con 0x7f798c0cd420 2023-10-30T13:39:08.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79837fe640 1 -- 172.21.15.31:0/2851567549 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974013470 con 0x7f798c0cd420 2023-10-30T13:39:08.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c009420 con 0x7f798c0cd420 2023-10-30T13:39:08.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c0c4ce0 con 0x7f798c0cd420 2023-10-30T13:39:08.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798c0c4ea0 con 0x7f798c0cd420 2023-10-30T13:39:08.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.302+0000 7f79837fe640 1 -- 172.21.15.31:0/2851567549 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79740135d0 con 0x7f798c0cd420 2023-10-30T13:39:08.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.306+0000 7f79837fe640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f796c0664d0 0x7f796c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.306+0000 7f79837fe640 1 -- 172.21.15.31:0/2851567549 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f7974084350 con 0x7f798c0cd420 2023-10-30T13:39:08.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.306+0000 7f798affd640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f796c0664d0 0x7f796c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.306+0000 7f798affd640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f796c0664d0 0x7f796c068990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f797c0b2000 tx=0x7f797c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:08.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.310+0000 7f79837fe640 1 -- 172.21.15.31:0/2851567549 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7974051570 con 0x7f798c0cd420 2023-10-30T13:39:08.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.494+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f798c04e170 con 0x7f798c0cd420 2023-10-30T13:39:08.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79837fe640 1 -- 172.21.15.31:0/2851567549 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f7974054b90 con 0x7f798c0cd420 2023-10-30T13:39:08.501 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:08.502 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:06.794140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 22 2023-10-30T13:39:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f796c0664d0 msgr2=0x7f796c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f796c0664d0 0x7f796c068990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f797c0b2000 tx=0x7f797c008040 comp rx=0 tx=0).stop 2023-10-30T13:39:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0cd420 msgr2=0x7f798c15d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0cd420 0x7f798c15d4d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f797400ebe0 tx=0x7f797400c520 comp rx=0 tx=0).stop 2023-10-30T13:39:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 shutdown_connections 2023-10-30T13:39:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f796c0664d0 0x7f796c068990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f798c153aa0 0x7f798c0c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0cd420 0x7f798c15d4d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 --2- 172.21.15.31:0/2851567549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f798c0701e0 0x7f798c0ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 >> 172.21.15.31:0/2851567549 conn(0x7f798c071fb0 msgr2=0x7f798c150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:08.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 shutdown_connections 2023-10-30T13:39:08.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.498+0000 7f79916ea640 1 -- 172.21.15.31:0/2851567549 wait complete. 2023-10-30T13:39:08.523 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:39:08.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:39:08.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:08.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f704933e640 1 Processor -- start 2023-10-30T13:39:08.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f704933e640 1 -- start start 2023-10-30T13:39:08.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f704933e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f704406b3b0 0x7f704406b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f704933e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440c7b00 con 0x7f704415d840 2023-10-30T13:39:08.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f704933e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704406bd20 con 0x7f7044153aa0 2023-10-30T13:39:08.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f704933e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7044070a50 con 0x7f704406b3b0 2023-10-30T13:39:08.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f703bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f704415d840 0x7f70440ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58800/0 (socket says 172.21.15.31:58800) 2023-10-30T13:39:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f703bfff640 1 -- 172.21.15.31:0/828498329 learned_addr learned my addr 172.21.15.31:0/828498329 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f70437fe640 1 --2- 172.21.15.31:0/828498329 >> v2:172.21.15.33:3300/0 conn(0x7f704406b3b0 0x7f704406b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394726507 0 0) 0x7f70440c7b00 con 0x7f704415d840 2023-10-30T13:39:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.662+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7030003530 con 0x7f704415d840 2023-10-30T13:39:08.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3344477667 0 0) 0x7f7030003530 con 0x7f704415d840 2023-10-30T13:39:08.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 >> v2:172.21.15.33:3300/0 conn(0x7f704406b3b0 msgr2=0x7f704406b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:08.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 --2- 172.21.15.31:0/828498329 >> v2:172.21.15.33:3300/0 conn(0x7f704406b3b0 0x7f704406b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 >> v1:172.21.15.33:6789/0 conn(0x7f7044153aa0 legacy=0x7f70440701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70440c2050 con 0x7f704415d840 2023-10-30T13:39:08.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f702c001680 con 0x7f704415d840 2023-10-30T13:39:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 >> v1:172.21.15.31:6789/0 conn(0x7f704415d840 legacy=0x7f70440ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f7030003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70300047d0 0x7f7030004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7030005190 0x7f703000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7044070a50 con 0x7f7030003980 2023-10-30T13:39:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703000c040 con 0x7f7030005190 2023-10-30T13:39:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703000c200 con 0x7f70300047d0 2023-10-30T13:39:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f7042ffd640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f7030003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70437fe640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7030005190 0x7f703000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f703bfff640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70300047d0 0x7f7030004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f7042ffd640 1 -- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70300047d0 msgr2=0x7f7030004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f7042ffd640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70300047d0 0x7f7030004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f7042ffd640 1 -- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7030005190 msgr2=0x7f703000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f7042ffd640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7030005190 0x7f703000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f7042ffd640 1 -- 172.21.15.31:0/828498329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030003530 con 0x7f7030003980 2023-10-30T13:39:08.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70437fe640 1 -- 172.21.15.31:0/828498329 reap_dead start 2023-10-30T13:39:08.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f7042ffd640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f7030003da0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7034001780 tx=0x7f703401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1aad82107fec418 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70340a4ea0 con 0x7f7030003980 2023-10-30T13:39:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.666+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70340a7050 con 0x7f7030003980 2023-10-30T13:39:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f70427fc640 1 -- 172.21.15.31:0/828498329 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034003600 con 0x7f7030003980 2023-10-30T13:39:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- 172.21.15.31:0/828498329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 msgr2=0x7f7030003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f7030003da0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7034001780 tx=0x7f703401c3e0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- 172.21.15.31:0/828498329 shutdown_connections 2023-10-30T13:39:08.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 --2- 172.21.15.31:0/828498329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f7030003da0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- 172.21.15.31:0/828498329 >> 172.21.15.31:0/828498329 conn(0x7f7044071fb0 msgr2=0x7f70440723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:08.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- 172.21.15.31:0/828498329 shutdown_connections 2023-10-30T13:39:08.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- 172.21.15.31:0/828498329 wait complete. 2023-10-30T13:39:08.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 Processor -- start 2023-10-30T13:39:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- start start 2023-10-30T13:39:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f70441f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f704415a3b0 0x7f704406c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70441eee80 0x7f704406c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70441f29e0 con 0x7f7030003980 2023-10-30T13:39:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70441f3340 con 0x7f70441eee80 2023-10-30T13:39:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f704933e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70440c4630 con 0x7f704415a3b0 2023-10-30T13:39:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f7042ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f70441f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f7042ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f70441f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59464/0 (socket says 172.21.15.31:59464) 2023-10-30T13:39:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f7042ffd640 1 -- 172.21.15.31:0/3339929093 learned_addr learned my addr 172.21.15.31:0/3339929093 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f70437fe640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70441eee80 0x7f704406c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.670+0000 7f703bfff640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f704415a3b0 0x7f704406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f7042ffd640 1 -- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f704415a3b0 msgr2=0x7f704406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f7042ffd640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f704415a3b0 0x7f704406c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f7042ffd640 1 -- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70441eee80 msgr2=0x7f704406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f7042ffd640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70441eee80 0x7f704406c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f7042ffd640 1 -- 172.21.15.31:0/3339929093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70441f23a0 con 0x7f7030003980 2023-10-30T13:39:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f70437fe640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70441eee80 0x7f704406c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f703bfff640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f704415a3b0 0x7f704406c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f7042ffd640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f70441f2e00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f703401c210 tx=0x7f7030005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f703b7fe640 1 -- 172.21.15.31:0/3339929093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034003e40 con 0x7f7030003980 2023-10-30T13:39:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f703b7fe640 1 -- 172.21.15.31:0/3339929093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70340a4ea0 con 0x7f7030003980 2023-10-30T13:39:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7044157800 con 0x7f7030003980 2023-10-30T13:39:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f703b7fe640 1 -- 172.21.15.31:0/3339929093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034007270 con 0x7f7030003980 2023-10-30T13:39:08.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70441f2600 con 0x7f7030003980 2023-10-30T13:39:08.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70441f2c50 con 0x7f7030003980 2023-10-30T13:39:08.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f703b7fe640 1 -- 172.21.15.31:0/3339929093 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70340073d0 con 0x7f7030003980 2023-10-30T13:39:08.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f703b7fe640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70240664d0 0x7f7024068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:08.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.674+0000 7f703b7fe640 1 -- 172.21.15.31:0/3339929093 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f703411e540 con 0x7f7030003980 2023-10-30T13:39:08.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.678+0000 7f703bfff640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70240664d0 0x7f7024068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:08.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.678+0000 7f703bfff640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70240664d0 0x7f7024068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f704415a7f0 tx=0x7f702c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:08.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.678+0000 7f703b7fe640 1 -- 172.21.15.31:0/3339929093 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70340ef3c0 con 0x7f7030003980 2023-10-30T13:39:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7044074780 con 0x7f7030003980 2023-10-30T13:39:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f703b7fe640 1 -- 172.21.15.31:0/3339929093 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f7034007a40 con 0x7f7030003980 2023-10-30T13:39:08.879 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:08.880 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:06.794140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:08.880 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 22 2023-10-30T13:39:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70240664d0 msgr2=0x7f7024068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70240664d0 0x7f7024068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f704415a7f0 tx=0x7f702c009040 comp rx=0 tx=0).stop 2023-10-30T13:39:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 msgr2=0x7f70441f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f70441f2e00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f703401c210 tx=0x7f7030005800 comp rx=0 tx=0).stop 2023-10-30T13:39:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 shutdown_connections 2023-10-30T13:39:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70240664d0 0x7f7024068990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70441eee80 0x7f704406c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f704415a3b0 0x7f704406c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 --2- 172.21.15.31:0/3339929093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030003980 0x7f70441f2e00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.874+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 >> 172.21.15.31:0/3339929093 conn(0x7f7044071fb0 msgr2=0x7f70441527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.878+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 shutdown_connections 2023-10-30T13:39:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:08.878+0000 7f704933e640 1 -- 172.21.15.31:0/3339929093 wait complete. 2023-10-30T13:39:08.901 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4276": { "addr": "172.21.15.33:6837/3402459191", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 3402459191, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 3402459191, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4276, "incarnation": 10, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:39:06.794140+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4276, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:39:10.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d6510640 1 Processor -- start 2023-10-30T13:39:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d6510640 1 -- start start 2023-10-30T13:39:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d6510640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d015cd40 0x7f09d00c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d6510640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f09d00ca430 0x7f09d00b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d6510640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d015d1b0 con 0x7f09d0153a20 2023-10-30T13:39:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d6510640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d015dc90 con 0x7f09d00ca430 2023-10-30T13:39:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d6510640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00b3010 con 0x7f09d015cd40 2023-10-30T13:39:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d5d0f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f09d00ca430 0x7f09d00b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:11.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d550e640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f09d0153a20 0x7f09d00b1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58816/0 (socket says 172.21.15.31:58816) 2023-10-30T13:39:11.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d550e640 1 -- 172.21.15.31:0/749206001 learned_addr learned my addr 172.21.15.31:0/749206001 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:11.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d4d0d640 1 --2- 172.21.15.31:0/749206001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d015cd40 0x7f09d00c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:11.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d5d0f640 1 -- 172.21.15.31:0/749206001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d015cd40 msgr2=0x7f09d00c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:11.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d5d0f640 1 --2- 172.21.15.31:0/749206001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d015cd40 0x7f09d00c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d5d0f640 1 -- 172.21.15.31:0/749206001 >> v1:172.21.15.31:6789/0 conn(0x7f09d0153a20 legacy=0x7f09d00b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:11.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.054+0000 7f09d5d0f640 1 -- 172.21.15.31:0/749206001 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00b25a0 con 0x7f09d00ca430 2023-10-30T13:39:11.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d5d0f640 1 --2- 172.21.15.31:0/749206001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00ca430 0x7f09d00b34a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009d90 tx=0x7f09c801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d43f3a39b537435 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:11.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09c7fff640 1 -- 172.21.15.31:0/749206001 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80a58f0 con 0x7f09d00ca430 2023-10-30T13:39:11.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09c7fff640 1 -- 172.21.15.31:0/749206001 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c80a5a50 con 0x7f09d00ca430 2023-10-30T13:39:11.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09c7fff640 1 -- 172.21.15.31:0/749206001 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80a5de0 con 0x7f09d00ca430 2023-10-30T13:39:11.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 -- 172.21.15.31:0/749206001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00ca430 msgr2=0x7f09d00b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:11.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 --2- 172.21.15.31:0/749206001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00ca430 0x7f09d00b34a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009d90 tx=0x7f09c801c920 comp rx=0 tx=0).stop 2023-10-30T13:39:11.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 -- 172.21.15.31:0/749206001 shutdown_connections 2023-10-30T13:39:11.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 --2- 172.21.15.31:0/749206001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00ca430 0x7f09d00b34a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 --2- 172.21.15.31:0/749206001 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d015cd40 0x7f09d00c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 -- 172.21.15.31:0/749206001 >> 172.21.15.31:0/749206001 conn(0x7f09d006bf60 msgr2=0x7f09d006c370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:11.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 -- 172.21.15.31:0/749206001 shutdown_connections 2023-10-30T13:39:11.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 -- 172.21.15.31:0/749206001 wait complete. 2023-10-30T13:39:11.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.058+0000 7f09d6510640 1 Processor -- start 2023-10-30T13:39:11.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 -- start start 2023-10-30T13:39:11.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d00ca430 0x7f09d00c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:11.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d0153a20 0x7f09d00c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:11.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d015cd40 0x7f09d00c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:11.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00c41a0 con 0x7f09d015cd40 2023-10-30T13:39:11.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01f28f0 con 0x7f09d0153a20 2023-10-30T13:39:11.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00c3620 con 0x7f09d00ca430 2023-10-30T13:39:11.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d015cd40 0x7f09d00c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:11.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d015cd40 0x7f09d00c4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59484/0 (socket says 172.21.15.31:59484) 2023-10-30T13:39:11.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 -- 172.21.15.31:0/2032910180 learned_addr learned my addr 172.21.15.31:0/2032910180 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:11.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d4d0d640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d0153a20 0x7f09d00c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:11.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d550e640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d00ca430 0x7f09d00c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:11.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 -- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d00ca430 msgr2=0x7f09d00c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:11.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d00ca430 0x7f09d00c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 -- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d0153a20 msgr2=0x7f09d00c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:11.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d0153a20 0x7f09d00c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 -- 172.21.15.31:0/2032910180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8009a70 con 0x7f09d015cd40 2023-10-30T13:39:11.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d5d0f640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d015cd40 0x7f09d00c4d10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f09c80b0040 tx=0x7f09c80196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:11.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09c5ffb640 1 -- 172.21.15.31:0/2032910180 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80b9860 con 0x7f09d015cd40 2023-10-30T13:39:11.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09c5ffb640 1 -- 172.21.15.31:0/2032910180 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c80b99c0 con 0x7f09d015cd40 2023-10-30T13:39:11.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09c5ffb640 1 -- 172.21.15.31:0/2032910180 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80b9d70 con 0x7f09d015cd40 2023-10-30T13:39:11.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d00c3900 con 0x7f09d015cd40 2023-10-30T13:39:11.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d00c3ec0 con 0x7f09d015cd40 2023-10-30T13:39:11.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.062+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d0067cb0 con 0x7f09d015cd40 2023-10-30T13:39:11.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.066+0000 7f09c5ffb640 1 -- 172.21.15.31:0/2032910180 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09c80ad440 con 0x7f09d015cd40 2023-10-30T13:39:11.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.066+0000 7f09c5ffb640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09b40664a0 0x7f09b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:11.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.066+0000 7f09c5ffb640 1 -- 172.21.15.31:0/2032910180 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f09c811f460 con 0x7f09d015cd40 2023-10-30T13:39:11.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.066+0000 7f09d550e640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09b40664a0 0x7f09b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:11.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.066+0000 7f09d550e640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09b40664a0 0x7f09b4068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09cc005890 tx=0x7f09cc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:11.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.070+0000 7f09c5ffb640 1 -- 172.21.15.31:0/2032910180 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09c80ec840 con 0x7f09d015cd40 2023-10-30T13:39:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.262+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d0067540 con 0x7f09d015cd40 2023-10-30T13:39:11.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.262+0000 7f09c5ffb640 1 -- 172.21.15.31:0/2032910180 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f09c80efe60 con 0x7f09d015cd40 2023-10-30T13:39:11.268 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:11.268 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:06.794140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:11.269 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 22 2023-10-30T13:39:11.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09b40664a0 msgr2=0x7f09b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:11.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09b40664a0 0x7f09b4068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09cc005890 tx=0x7f09cc009040 comp rx=0 tx=0).stop 2023-10-30T13:39:11.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d015cd40 msgr2=0x7f09d00c4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d015cd40 0x7f09d00c4d10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f09c80b0040 tx=0x7f09c80196c0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 shutdown_connections 2023-10-30T13:39:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09b40664a0 0x7f09b4068960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d015cd40 0x7f09d00c4d10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d0153a20 0x7f09d00c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 --2- 172.21.15.31:0/2032910180 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09d00ca430 0x7f09d00c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 >> 172.21.15.31:0/2032910180 conn(0x7f09d006bf60 msgr2=0x7f09d015bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 shutdown_connections 2023-10-30T13:39:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:11.266+0000 7f09d6510640 1 -- 172.21.15.31:0/2032910180 wait complete. 2023-10-30T13:39:11.289 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:13.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:13.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2b03640 1 Processor -- start 2023-10-30T13:39:13.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2b03640 1 -- start start 2023-10-30T13:39:13.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2b03640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f70ec0c8eb0 0x7f70ec15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:13.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2b03640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f70ec06aeb0 0x7f70ec06b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:13.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2b03640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec15dcc0 con 0x7f70ec15ac70 2023-10-30T13:39:13.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2b03640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec159800 con 0x7f70ec0c8eb0 2023-10-30T13:39:13.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2b03640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec159d80 con 0x7f70ec06aeb0 2023-10-30T13:39:13.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f1b01640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f70ec0c8eb0 0x7f70ec15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:13.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f1b01640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f70ec0c8eb0 0x7f70ec15cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59492/0 (socket says 172.21.15.31:59492) 2023-10-30T13:39:13.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f1b01640 1 -- 172.21.15.31:0/571791107 learned_addr learned my addr 172.21.15.31:0/571791107 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:13.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f2302640 1 --2- 172.21.15.31:0/571791107 >> v2:172.21.15.33:3300/0 conn(0x7f70ec06aeb0 0x7f70ec06b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:13.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f1b01640 1 -- 172.21.15.31:0/571791107 >> v2:172.21.15.33:3300/0 conn(0x7f70ec06aeb0 msgr2=0x7f70ec06b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:13.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f1b01640 1 --2- 172.21.15.31:0/571791107 >> v2:172.21.15.33:3300/0 conn(0x7f70ec06aeb0 0x7f70ec06b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f1b01640 1 -- 172.21.15.31:0/571791107 >> v1:172.21.15.33:6789/0 conn(0x7f70ec15ac70 legacy=0x7f70ec15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:13.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.442+0000 7f70f1b01640 1 -- 172.21.15.31:0/571791107 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec15d420 con 0x7f70ec0c8eb0 2023-10-30T13:39:13.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f1b01640 1 --2- 172.21.15.31:0/571791107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec0c8eb0 0x7f70ec15cd70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f70ec08cce0 tx=0x7f70e801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a31fdc5c776d32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:13.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f0aff640 1 -- 172.21.15.31:0/571791107 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e80a89b0 con 0x7f70ec0c8eb0 2023-10-30T13:39:13.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f0aff640 1 -- 172.21.15.31:0/571791107 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e80a8b10 con 0x7f70ec0c8eb0 2023-10-30T13:39:13.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f0aff640 1 -- 172.21.15.31:0/571791107 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e80a8ea0 con 0x7f70ec0c8eb0 2023-10-30T13:39:13.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 -- 172.21.15.31:0/571791107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec0c8eb0 msgr2=0x7f70ec15cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:13.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 --2- 172.21.15.31:0/571791107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec0c8eb0 0x7f70ec15cd70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f70ec08cce0 tx=0x7f70e801ecb0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 -- 172.21.15.31:0/571791107 shutdown_connections 2023-10-30T13:39:13.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 --2- 172.21.15.31:0/571791107 >> v2:172.21.15.33:3300/0 conn(0x7f70ec06aeb0 0x7f70ec06b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 --2- 172.21.15.31:0/571791107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec0c8eb0 0x7f70ec15cd70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 -- 172.21.15.31:0/571791107 >> 172.21.15.31:0/571791107 conn(0x7f70ec0cef00 msgr2=0x7f70ec0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:13.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 -- 172.21.15.31:0/571791107 shutdown_connections 2023-10-30T13:39:13.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 -- 172.21.15.31:0/571791107 wait complete. 2023-10-30T13:39:13.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.446+0000 7f70f2b03640 1 Processor -- start 2023-10-30T13:39:13.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 -- start start 2023-10-30T13:39:13.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70ec06aeb0 0x7f70ec1495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:13.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70ec0c8eb0 0x7f70ec0c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:13.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec15ac70 0x7f70ec14e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:13.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec06a7e0 con 0x7f70ec15ac70 2023-10-30T13:39:13.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec15b730 con 0x7f70ec0c8eb0 2023-10-30T13:39:13.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec149af0 con 0x7f70ec06aeb0 2023-10-30T13:39:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec15ac70 0x7f70ec14e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec15ac70 0x7f70ec14e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59504/0 (socket says 172.21.15.31:59504) 2023-10-30T13:39:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 -- 172.21.15.31:0/261518580 learned_addr learned my addr 172.21.15.31:0/261518580 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f1300640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70ec0c8eb0 0x7f70ec0c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f1b01640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70ec06aeb0 0x7f70ec1495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 -- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70ec06aeb0 msgr2=0x7f70ec1495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70ec06aeb0 0x7f70ec1495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 -- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70ec0c8eb0 msgr2=0x7f70ec0c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70ec0c8eb0 0x7f70ec0c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 -- 172.21.15.31:0/261518580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e80094d0 con 0x7f70ec15ac70 2023-10-30T13:39:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2302640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec15ac70 0x7f70ec14e850 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f70e4007e70 tx=0x7f70e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:13.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70e27fc640 1 -- 172.21.15.31:0/261518580 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4014620 con 0x7f70ec15ac70 2023-10-30T13:39:13.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70e27fc640 1 -- 172.21.15.31:0/261518580 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e40147a0 con 0x7f70ec15ac70 2023-10-30T13:39:13.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70e27fc640 1 -- 172.21.15.31:0/261518580 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4014b50 con 0x7f70ec15ac70 2023-10-30T13:39:13.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec15b450 con 0x7f70ec15ac70 2023-10-30T13:39:13.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec0c2e70 con 0x7f70ec15ac70 2023-10-30T13:39:13.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.450+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ec04e170 con 0x7f70ec15ac70 2023-10-30T13:39:13.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.454+0000 7f70e27fc640 1 -- 172.21.15.31:0/261518580 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70e4014cb0 con 0x7f70ec15ac70 2023-10-30T13:39:13.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.454+0000 7f70e27fc640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70d00664a0 0x7f70d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.454+0000 7f70e27fc640 1 -- 172.21.15.31:0/261518580 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f70e40848f0 con 0x7f70ec15ac70 2023-10-30T13:39:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.454+0000 7f70f1b01640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70d00664a0 0x7f70d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.454+0000 7f70f1b01640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70d00664a0 0x7f70d0068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f70e80b2000 tx=0x7f70e8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:13.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.458+0000 7f70e27fc640 1 -- 172.21.15.31:0/261518580 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70e4051bb0 con 0x7f70ec15ac70 2023-10-30T13:39:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.646+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70ec0ca960 con 0x7f70ec15ac70 2023-10-30T13:39:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.646+0000 7f70e27fc640 1 -- 172.21.15.31:0/261518580 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5733 (secure 0 0 0) 0x7f70e40551d0 con 0x7f70ec15ac70 2023-10-30T13:39:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:13.653 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:12.009009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:resolve","state_seq":35,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-10-30T13:39:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70d00664a0 msgr2=0x7f70d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70d00664a0 0x7f70d0068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f70e80b2000 tx=0x7f70e8009040 comp rx=0 tx=0).stop 2023-10-30T13:39:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec15ac70 msgr2=0x7f70ec14e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec15ac70 0x7f70ec14e850 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f70e4007e70 tx=0x7f70e400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 shutdown_connections 2023-10-30T13:39:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70d00664a0 0x7f70d0068960 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70ec15ac70 0x7f70ec14e850 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70ec0c8eb0 0x7f70ec0c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 --2- 172.21.15.31:0/261518580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70ec06aeb0 0x7f70ec1495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 >> 172.21.15.31:0/261518580 conn(0x7f70ec0cef00 msgr2=0x7f70ec158a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 shutdown_connections 2023-10-30T13:39:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:13.650+0000 7f70f2b03640 1 -- 172.21.15.31:0/261518580 wait complete. 2023-10-30T13:39:13.674 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:15.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.826+0000 7fc4b471d640 1 Processor -- start 2023-10-30T13:39:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b471d640 1 -- start start 2023-10-30T13:39:15.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b471d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b471d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc4ac06b3b0 0x7fc4ac06b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b471d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac06bd80 con 0x7fc4ac0701e0 2023-10-30T13:39:15.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b471d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac070a50 con 0x7fc4ac06b3b0 2023-10-30T13:39:15.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b471d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac15dd80 con 0x7fc4ac153aa0 2023-10-30T13:39:15.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b2cba640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc4ac06b3b0 0x7fc4ac06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b2cba640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc4ac06b3b0 0x7fc4ac06b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51532/0 (socket says 172.21.15.31:51532) 2023-10-30T13:39:15.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b2cba640 1 -- 172.21.15.31:0/2623588916 learned_addr learned my addr 172.21.15.31:0/2623588916 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:15.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b1cb8640 1 --1- 172.21.15.31:0/2623588916 >> v1:172.21.15.31:6789/0 conn(0x7fc4ac0701e0 0x7fc4ac0ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:42994/0 (socket says 172.21.15.31:42994) 2023-10-30T13:39:15.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b24b9640 1 --2- 172.21.15.31:0/2623588916 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b2cba640 1 -- 172.21.15.31:0/2623588916 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 msgr2=0x7fc4ac15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:15.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b2cba640 1 --2- 172.21.15.31:0/2623588916 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b2cba640 1 -- 172.21.15.31:0/2623588916 >> v1:172.21.15.31:6789/0 conn(0x7fc4ac0701e0 legacy=0x7fc4ac0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b2cba640 1 -- 172.21.15.31:0/2623588916 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0c7b00 con 0x7fc4ac06b3b0 2023-10-30T13:39:15.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.830+0000 7fc4b24b9640 1 --2- 172.21.15.31:0/2623588916 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac15d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:15.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b2cba640 1 --2- 172.21.15.31:0/2623588916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac06b840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009e50 tx=0x7fc4a401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fb9af0a9063b47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:15.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b14b7640 1 -- 172.21.15.31:0/2623588916 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40af400 con 0x7fc4ac06b3b0 2023-10-30T13:39:15.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b14b7640 1 -- 172.21.15.31:0/2623588916 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a40ab050 con 0x7fc4ac06b3b0 2023-10-30T13:39:15.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b14b7640 1 -- 172.21.15.31:0/2623588916 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40af780 con 0x7fc4ac06b3b0 2023-10-30T13:39:15.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 -- 172.21.15.31:0/2623588916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac06b3b0 msgr2=0x7fc4ac06b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 --2- 172.21.15.31:0/2623588916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac06b840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009e50 tx=0x7fc4a401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 -- 172.21.15.31:0/2623588916 shutdown_connections 2023-10-30T13:39:15.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 --2- 172.21.15.31:0/2623588916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac06b840 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 --2- 172.21.15.31:0/2623588916 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac15d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 -- 172.21.15.31:0/2623588916 >> 172.21.15.31:0/2623588916 conn(0x7fc4ac071fb0 msgr2=0x7fc4ac0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:15.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 -- 172.21.15.31:0/2623588916 shutdown_connections 2023-10-30T13:39:15.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 -- 172.21.15.31:0/2623588916 wait complete. 2023-10-30T13:39:15.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.834+0000 7fc4b471d640 1 Processor -- start 2023-10-30T13:39:15.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 -- start start 2023-10-30T13:39:15.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac0701e0 0x7fc4ac14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac14ad00 con 0x7fc4ac0701e0 2023-10-30T13:39:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac1f3150 con 0x7fc4ac06b3b0 2023-10-30T13:39:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac151440 con 0x7fc4ac153aa0 2023-10-30T13:39:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac0701e0 0x7fc4ac14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac0701e0 0x7fc4ac14fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51548/0 (socket says 172.21.15.31:51548) 2023-10-30T13:39:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 -- 172.21.15.31:0/4063598933 learned_addr learned my addr 172.21.15.31:0/4063598933 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b2cba640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b24b9640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 -- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 msgr2=0x7fc4ac150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 -- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4ac06b3b0 msgr2=0x7fc4ac143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac143bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 -- 172.21.15.31:0/4063598933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4009b30 con 0x7fc4ac0701e0 2023-10-30T13:39:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b24b9640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:39:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b1cb8640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac0701e0 0x7fc4ac14fd00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c490 tx=0x7fc49c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4a2ffd640 1 -- 172.21.15.31:0/4063598933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c007480 con 0x7fc4ac0701e0 2023-10-30T13:39:15.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4a2ffd640 1 -- 172.21.15.31:0/4063598933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c0075e0 con 0x7fc4ac0701e0 2023-10-30T13:39:15.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4a2ffd640 1 -- 172.21.15.31:0/4063598933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c004a10 con 0x7fc4ac0701e0 2023-10-30T13:39:15.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac1f2bd0 con 0x7fc4ac0701e0 2023-10-30T13:39:15.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac1f2e70 con 0x7fc4ac0701e0 2023-10-30T13:39:15.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.838+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac04e170 con 0x7fc4ac0701e0 2023-10-30T13:39:15.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.842+0000 7fc4a2ffd640 1 -- 172.21.15.31:0/4063598933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc49c004b70 con 0x7fc4ac0701e0 2023-10-30T13:39:15.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.842+0000 7fc4a2ffd640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4900664a0 0x7fc490068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.842+0000 7fc4a2ffd640 1 -- 172.21.15.31:0/4063598933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fc49c0831c0 con 0x7fc4ac0701e0 2023-10-30T13:39:15.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.842+0000 7fc4b24b9640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4900664a0 0x7fc490068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.842+0000 7fc4b24b9640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4900664a0 0x7fc490068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80097a0 tx=0x7fc4a8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:15.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:15.846+0000 7fc4a2ffd640 1 -- 172.21.15.31:0/4063598933 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc49c0504f0 con 0x7fc4ac0701e0 2023-10-30T13:39:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4ac070600 con 0x7fc4ac0701e0 2023-10-30T13:39:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4a2ffd640 1 -- 172.21.15.31:0/4063598933 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5733 (secure 0 0 0) 0x7fc49c053b10 con 0x7fc4ac0701e0 2023-10-30T13:39:16.043 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:16.043 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:12.009009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:resolve","state_seq":35,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-10-30T13:39:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4900664a0 msgr2=0x7fc490068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4900664a0 0x7fc490068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80097a0 tx=0x7fc4a8008040 comp rx=0 tx=0).stop 2023-10-30T13:39:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac0701e0 msgr2=0x7fc4ac14fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac0701e0 0x7fc4ac14fd00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c490 tx=0x7fc49c00c950 comp rx=0 tx=0).stop 2023-10-30T13:39:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 shutdown_connections 2023-10-30T13:39:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4900664a0 0x7fc490068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc4ac153aa0 0x7fc4ac150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ac0701e0 0x7fc4ac14fd00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 --2- 172.21.15.31:0/4063598933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4ac06b3b0 0x7fc4ac143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 >> 172.21.15.31:0/4063598933 conn(0x7fc4ac071fb0 msgr2=0x7fc4ac0cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 shutdown_connections 2023-10-30T13:39:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:16.038+0000 7fc4b471d640 1 -- 172.21.15.31:0/4063598933 wait complete. 2023-10-30T13:39:16.064 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:18.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:18.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6f2ea640 1 Processor -- start 2023-10-30T13:39:18.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6f2ea640 1 -- start start 2023-10-30T13:39:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6f2ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f6f68153c10 0x7f6f68065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6f2ea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6806e390 con 0x7f6f68070c30 2023-10-30T13:39:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6f2ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f680660e0 con 0x7f6f68159970 2023-10-30T13:39:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6f2ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f68072390 con 0x7f6f68153c10 2023-10-30T13:39:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6c885640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f6f68070c30 0x7f6f68071070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43006/0 (socket says 172.21.15.31:43006) 2023-10-30T13:39:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6c885640 1 -- 172.21.15.31:0/949148657 learned_addr learned my addr 172.21.15.31:0/949148657 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:18.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.214+0000 7f6f6d086640 1 --2- 172.21.15.31:0/949148657 >> v2:172.21.15.33:3300/0 conn(0x7f6f68153c10 0x7f6f68065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686746734 0 0) 0x7f6f6806e390 con 0x7f6f68070c30 2023-10-30T13:39:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f54003530 con 0x7f6f68070c30 2023-10-30T13:39:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635565222 0 0) 0x7f6f54003530 con 0x7f6f68070c30 2023-10-30T13:39:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 >> v2:172.21.15.33:3300/0 conn(0x7f6f68153c10 msgr2=0x7f6f68065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 --2- 172.21.15.31:0/949148657 >> v2:172.21.15.33:3300/0 conn(0x7f6f68153c10 0x7f6f68065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 >> v1:172.21.15.33:6789/0 conn(0x7f6f68159970 legacy=0x7f6f68159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f680bec10 con 0x7f6f68070c30 2023-10-30T13:39:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f6f58001680 con 0x7f6f68070c30 2023-10-30T13:39:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 --2- 172.21.15.31:0/949148657 >> v2:172.21.15.33:3300/0 conn(0x7f6f68153c10 0x7f6f68065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:39:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 >> v1:172.21.15.31:6789/0 conn(0x7f6f68070c30 legacy=0x7f6f68071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:18.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f54003980 0x7f6f54003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f540047d0 0x7f6f54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f54005190 0x7f6f5400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f68072390 con 0x7f6f54003980 2023-10-30T13:39:18.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5400c130 con 0x7f6f540047d0 2023-10-30T13:39:18.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5400c2f0 con 0x7f6f54005190 2023-10-30T13:39:18.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f54003980 0x7f6f54003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d887640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f54005190 0x7f6f5400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6c885640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f540047d0 0x7f6f54004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 -- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f54005190 msgr2=0x7f6f5400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:18.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f54005190 0x7f6f5400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 -- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f540047d0 msgr2=0x7f6f54004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:18.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f540047d0 0x7f6f54004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 -- 172.21.15.31:0/949148657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54003530 con 0x7f6f54003980 2023-10-30T13:39:18.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d887640 1 -- 172.21.15.31:0/949148657 reap_dead start 2023-10-30T13:39:18.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.218+0000 7f6f6d086640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f54003980 0x7f6f54003da0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009cb0 tx=0x7f6f6401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4793b211c4d1bd93 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f640a84e0 con 0x7f6f54003980 2023-10-30T13:39:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f640a8640 con 0x7f6f54003980 2023-10-30T13:39:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f5ffff640 1 -- 172.21.15.31:0/949148657 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f640a8a30 con 0x7f6f54003980 2023-10-30T13:39:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/949148657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f54003980 msgr2=0x7f6f54003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f54003980 0x7f6f54003da0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009cb0 tx=0x7f6f6401cbf0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/949148657 shutdown_connections 2023-10-30T13:39:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/949148657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f54003980 0x7f6f54003da0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/949148657 >> 172.21.15.31:0/949148657 conn(0x7f6f680c7f60 msgr2=0x7f6f680c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/949148657 shutdown_connections 2023-10-30T13:39:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/949148657 wait complete. 2023-10-30T13:39:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 Processor -- start 2023-10-30T13:39:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- start start 2023-10-30T13:39:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f54003980 0x7f6f680c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f680c6570 0x7f6f6814bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f6814c230 0x7f6f6814a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f680c6ee0 con 0x7f6f680c6570 2023-10-30T13:39:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f680c2880 con 0x7f6f54003980 2023-10-30T13:39:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6f2ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f680c2b60 con 0x7f6f6814c230 2023-10-30T13:39:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6c885640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f680c6570 0x7f6f6814bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6c885640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f680c6570 0x7f6f6814bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51580/0 (socket says 172.21.15.31:51580) 2023-10-30T13:39:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6c885640 1 -- 172.21.15.31:0/4258503420 learned_addr learned my addr 172.21.15.31:0/4258503420 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.222+0000 7f6f6d086640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f54003980 0x7f6f680c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6d887640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f6814c230 0x7f6f6814a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6c885640 1 -- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f6814c230 msgr2=0x7f6f6814a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6c885640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f6814c230 0x7f6f6814a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6c885640 1 -- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f54003980 msgr2=0x7f6f680c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6c885640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f54003980 0x7f6f680c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6c885640 1 -- 172.21.15.31:0/4258503420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f680c2e40 con 0x7f6f680c6570 2023-10-30T13:39:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6d887640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f6814c230 0x7f6f6814a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6c885640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f680c6570 0x7f6f6814bcf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800e7b0 tx=0x7f6f5800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f5dffb640 1 -- 172.21.15.31:0/4258503420 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5800ee60 con 0x7f6f680c6570 2023-10-30T13:39:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f5dffb640 1 -- 172.21.15.31:0/4258503420 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5800c8d0 con 0x7f6f680c6570 2023-10-30T13:39:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f5dffb640 1 -- 172.21.15.31:0/4258503420 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5800cc80 con 0x7f6f680c6570 2023-10-30T13:39:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f680c3120 con 0x7f6f680c6570 2023-10-30T13:39:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f680c3920 con 0x7f6f680c6570 2023-10-30T13:39:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6804e170 con 0x7f6f680c6570 2023-10-30T13:39:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f5dffb640 1 -- 172.21.15.31:0/4258503420 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6f5801d460 con 0x7f6f680c6570 2023-10-30T13:39:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f5dffb640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f4c0664a0 0x7f6f4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.226+0000 7f6f5dffb640 1 -- 172.21.15.31:0/4258503420 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f6f58083e20 con 0x7f6f680c6570 2023-10-30T13:39:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.230+0000 7f6f6d086640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f4c0664a0 0x7f6f4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.230+0000 7f6f6d086640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f4c0664a0 0x7f6f4c068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f640af000 tx=0x7f6f640b2040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.234+0000 7f6f5dffb640 1 -- 172.21.15.31:0/4258503420 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6f58054bf0 con 0x7f6f680c6570 2023-10-30T13:39:18.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f68070150 con 0x7f6f680c6570 2023-10-30T13:39:18.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f5dffb640 1 -- 172.21.15.31:0/4258503420 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5733 (secure 0 0 0) 0x7f6f58054780 con 0x7f6f680c6570 2023-10-30T13:39:18.426 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:18.426 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:12.009009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:resolve","state_seq":35,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-10-30T13:39:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f4c0664a0 msgr2=0x7f6f4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f4c0664a0 0x7f6f4c068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f640af000 tx=0x7f6f640b2040 comp rx=0 tx=0).stop 2023-10-30T13:39:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f680c6570 msgr2=0x7f6f6814bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:18.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f680c6570 0x7f6f6814bcf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800e7b0 tx=0x7f6f5800ec70 comp rx=0 tx=0).stop 2023-10-30T13:39:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 shutdown_connections 2023-10-30T13:39:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6f4c0664a0 0x7f6f4c068960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6f6814c230 0x7f6f6814a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f680c6570 0x7f6f6814bcf0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 --2- 172.21.15.31:0/4258503420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f54003980 0x7f6f680c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 >> 172.21.15.31:0/4258503420 conn(0x7f6f680c7f60 msgr2=0x7f6f681586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 shutdown_connections 2023-10-30T13:39:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:18.422+0000 7f6f6f2ea640 1 -- 172.21.15.31:0/4258503420 wait complete. 2023-10-30T13:39:18.447 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:20.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:20.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d31b3e640 1 Processor -- start 2023-10-30T13:39:20.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d31b3e640 1 -- start start 2023-10-30T13:39:20.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d31b3e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8d2c153c80 0x7f8d2c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d31b3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c159f20 con 0x7f8d2c0c4a10 2023-10-30T13:39:20.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d31b3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c15d620 con 0x7f8d2c0c42a0 2023-10-30T13:39:20.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d31b3e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c15dec0 con 0x7f8d2c153c80 2023-10-30T13:39:20.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8d2c0c4a10 0x7f8d2c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43008/0 (socket says 172.21.15.31:43008) 2023-10-30T13:39:20.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2affd640 1 -- 172.21.15.31:0/3465712135 learned_addr learned my addr 172.21.15.31:0/3465712135 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:20.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2b7fe640 1 --2- 172.21.15.31:0/3465712135 >> v2:172.21.15.33:3300/0 conn(0x7f8d2c153c80 0x7f8d2c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908210788 0 0) 0x7f8d2c159f20 con 0x7f8d2c0c4a10 2023-10-30T13:39:20.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d14003530 con 0x7f8d2c0c4a10 2023-10-30T13:39:20.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643151977 0 0) 0x7f8d14003530 con 0x7f8d2c0c4a10 2023-10-30T13:39:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 >> v2:172.21.15.33:3300/0 conn(0x7f8d2c153c80 msgr2=0x7f8d2c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 --2- 172.21.15.31:0/3465712135 >> v2:172.21.15.33:3300/0 conn(0x7f8d2c153c80 0x7f8d2c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 >> v1:172.21.15.33:6789/0 conn(0x7f8d2c0c42a0 legacy=0x7f8d2c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c15d900 con 0x7f8d2c0c4a10 2023-10-30T13:39:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f8d20001680 con 0x7f8d2c0c4a10 2023-10-30T13:39:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.598+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 >> v1:172.21.15.31:6789/0 conn(0x7f8d2c0c4a10 legacy=0x7f8d2c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d14003980 0x7f8d14003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d140047d0 0x7f8d14004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d1400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c15dec0 con 0x7f8d14005190 2023-10-30T13:39:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400c040 con 0x7f8d14003980 2023-10-30T13:39:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400c200 con 0x7f8d140047d0 2023-10-30T13:39:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2b7fe640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d14003980 0x7f8d14003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d1400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2affd640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d140047d0 0x7f8d14004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 -- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d140047d0 msgr2=0x7f8d14004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d140047d0 0x7f8d14004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 -- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d14003980 msgr2=0x7f8d14003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d14003980 0x7f8d14003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 -- 172.21.15.31:0/3465712135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d14003530 con 0x7f8d14005190 2023-10-30T13:39:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2affd640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d140047d0 0x7f8d14004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 -- 172.21.15.31:0/3465712135 reap_dead start 2023-10-30T13:39:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2bfff640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d1400b990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8d24001e70 tx=0x7f8d2401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b7fb8ffae9ba6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24003b80 con 0x7f8d14005190 2023-10-30T13:39:20.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d24003d20 con 0x7f8d14005190 2023-10-30T13:39:20.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.602+0000 7f8d2a7fc640 1 -- 172.21.15.31:0/3465712135 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24005260 con 0x7f8d14005190 2023-10-30T13:39:20.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- 172.21.15.31:0/3465712135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 msgr2=0x7f8d1400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d1400b990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8d24001e70 tx=0x7f8d2401c760 comp rx=0 tx=0).stop 2023-10-30T13:39:20.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- 172.21.15.31:0/3465712135 shutdown_connections 2023-10-30T13:39:20.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/3465712135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d1400b990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- 172.21.15.31:0/3465712135 >> 172.21.15.31:0/3465712135 conn(0x7f8d2c0715c0 msgr2=0x7f8d2c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:20.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- 172.21.15.31:0/3465712135 shutdown_connections 2023-10-30T13:39:20.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- 172.21.15.31:0/3465712135 wait complete. 2023-10-30T13:39:20.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 Processor -- start 2023-10-30T13:39:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- start start 2023-10-30T13:39:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d2c1f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d2c0c8ba0 0x7f8d2c0cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2c0cdae0 0x7f8d2c146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0cdf60 con 0x7f8d14005190 2023-10-30T13:39:20.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c14e960 con 0x7f8d2c0cdae0 2023-10-30T13:39:20.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d31b3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c14ec40 con 0x7f8d2c0c8ba0 2023-10-30T13:39:20.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d2c1f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d2c1f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51598/0 (socket says 172.21.15.31:51598) 2023-10-30T13:39:20.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.606+0000 7f8d2b7fe640 1 -- 172.21.15.31:0/145667388 learned_addr learned my addr 172.21.15.31:0/145667388 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2bfff640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2c0cdae0 0x7f8d2c146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2affd640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d2c0c8ba0 0x7f8d2c0cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2b7fe640 1 -- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d2c0c8ba0 msgr2=0x7f8d2c0cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2b7fe640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d2c0c8ba0 0x7f8d2c0cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2b7fe640 1 -- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2c0cdae0 msgr2=0x7f8d2c146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2b7fe640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2c0cdae0 0x7f8d2c146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2b7fe640 1 -- 172.21.15.31:0/145667388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c14ef20 con 0x7f8d14005190 2023-10-30T13:39:20.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2bfff640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2c0cdae0 0x7f8d2c146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:39:20.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2affd640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d2c0c8ba0 0x7f8d2c0cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:39:20.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d2b7fe640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d2c1f2c00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011960 tx=0x7f8d1c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:20.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d13fff640 1 -- 172.21.15.31:0/145667388 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c00faf0 con 0x7f8d14005190 2023-10-30T13:39:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d13fff640 1 -- 172.21.15.31:0/145667388 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c00fc50 con 0x7f8d14005190 2023-10-30T13:39:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d13fff640 1 -- 172.21.15.31:0/145667388 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c014440 con 0x7f8d14005190 2023-10-30T13:39:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c14f200 con 0x7f8d14005190 2023-10-30T13:39:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c0cd080 con 0x7f8d14005190 2023-10-30T13:39:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c0cd210 con 0x7f8d14005190 2023-10-30T13:39:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d13fff640 1 -- 172.21.15.31:0/145667388 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d1c01d400 con 0x7f8d14005190 2023-10-30T13:39:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d13fff640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d180664f0 0x7f8d180689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.610+0000 7f8d13fff640 1 -- 172.21.15.31:0/145667388 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f8d1c084940 con 0x7f8d14005190 2023-10-30T13:39:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.614+0000 7f8d2affd640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d180664f0 0x7f8d180689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.614+0000 7f8d2affd640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d180664f0 0x7f8d180689b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8d20001ec0 tx=0x7f8d20000f20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.614+0000 7f8d13fff640 1 -- 172.21.15.31:0/145667388 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d1c051bd0 con 0x7f8d14005190 2023-10-30T13:39:20.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.806+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d2c04e170 con 0x7f8d14005190 2023-10-30T13:39:20.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d13fff640 1 -- 172.21.15.31:0/145667388 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f8d1c0551f0 con 0x7f8d14005190 2023-10-30T13:39:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:20.814 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:17.878036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:20.814 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2023-10-30T13:39:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d180664f0 msgr2=0x7f8d180689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d180664f0 0x7f8d180689b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8d20001ec0 tx=0x7f8d20000f20 comp rx=0 tx=0).stop 2023-10-30T13:39:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 msgr2=0x7f8d2c1f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d2c1f2c00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011960 tx=0x7f8d1c011e20 comp rx=0 tx=0).stop 2023-10-30T13:39:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 shutdown_connections 2023-10-30T13:39:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d180664f0 0x7f8d180689b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2c0cdae0 0x7f8d2c146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d2c0c8ba0 0x7f8d2c0cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 --2- 172.21.15.31:0/145667388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d14005190 0x7f8d2c1f2c00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 >> 172.21.15.31:0/145667388 conn(0x7f8d2c0715c0 msgr2=0x7f8d2c1526b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:20.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 shutdown_connections 2023-10-30T13:39:20.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:20.810+0000 7f8d31b3e640 1 -- 172.21.15.31:0/145667388 wait complete. 2023-10-30T13:39:20.836 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4276": { "addr": "172.21.15.33:6837/3402459191", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 3402459191, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 3402459191, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4276, "incarnation": 10, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:reconnect", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:39:17.878036+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4276, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:39:22.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:22.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab66e9c640 1 Processor -- start 2023-10-30T13:39:22.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab66e9c640 1 -- start start 2023-10-30T13:39:22.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab66e9c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fab600cb7d0 0x7fab600c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab66e9c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fab600c3e40 0x7fab60070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab66e9c640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab600c2310 con 0x7fab60150830 2023-10-30T13:39:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab66e9c640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6006bc10 con 0x7fab600cb7d0 2023-10-30T13:39:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab66e9c640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab60156710 con 0x7fab600c3e40 2023-10-30T13:39:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab5ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fab600cb7d0 0x7fab600c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab64c38640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fab60150830 0x7fab6006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43022/0 (socket says 172.21.15.31:43022) 2023-10-30T13:39:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.986+0000 7fab64c38640 1 -- 172.21.15.31:0/974954616 learned_addr learned my addr 172.21.15.31:0/974954616 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab65439640 1 --2- 172.21.15.31:0/974954616 >> v2:172.21.15.33:3300/0 conn(0x7fab600c3e40 0x7fab60070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5ffff640 1 -- 172.21.15.31:0/974954616 >> v2:172.21.15.33:3300/0 conn(0x7fab600c3e40 msgr2=0x7fab60070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5ffff640 1 --2- 172.21.15.31:0/974954616 >> v2:172.21.15.33:3300/0 conn(0x7fab600c3e40 0x7fab60070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5ffff640 1 -- 172.21.15.31:0/974954616 >> v1:172.21.15.31:6789/0 conn(0x7fab60150830 legacy=0x7fab6006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5ffff640 1 -- 172.21.15.31:0/974954616 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600c4360 con 0x7fab600cb7d0 2023-10-30T13:39:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5f7fe640 1 -- 172.21.15.31:0/974954616 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162356132 0 0) 0x7fab600c2310 con 0x7fab60150830 2023-10-30T13:39:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5ffff640 1 --2- 172.21.15.31:0/974954616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600cb7d0 0x7fab600c3900 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab60066e10 tx=0x7fab4c021320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871e44310ba71f4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5f7fe640 1 -- 172.21.15.31:0/974954616 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0b1600 con 0x7fab600cb7d0 2023-10-30T13:39:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5f7fe640 1 -- 172.21.15.31:0/974954616 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4c0ad050 con 0x7fab600cb7d0 2023-10-30T13:39:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.990+0000 7fab5f7fe640 1 -- 172.21.15.31:0/974954616 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0b1980 con 0x7fab600cb7d0 2023-10-30T13:39:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- 172.21.15.31:0/974954616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600cb7d0 msgr2=0x7fab600c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 --2- 172.21.15.31:0/974954616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600cb7d0 0x7fab600c3900 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab60066e10 tx=0x7fab4c021320 comp rx=0 tx=0).stop 2023-10-30T13:39:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- 172.21.15.31:0/974954616 shutdown_connections 2023-10-30T13:39:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 --2- 172.21.15.31:0/974954616 >> v2:172.21.15.33:3300/0 conn(0x7fab600c3e40 0x7fab60070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 --2- 172.21.15.31:0/974954616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600cb7d0 0x7fab600c3900 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- 172.21.15.31:0/974954616 >> 172.21.15.31:0/974954616 conn(0x7fab600cac70 msgr2=0x7fab600cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:22.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- 172.21.15.31:0/974954616 shutdown_connections 2023-10-30T13:39:22.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- 172.21.15.31:0/974954616 wait complete. 2023-10-30T13:39:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 Processor -- start 2023-10-30T13:39:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- start start 2023-10-30T13:39:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600c3e40 0x7fab60147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab600cb7d0 0x7fab601475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab60150830 0x7fab60148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006c180 con 0x7fab60150830 2023-10-30T13:39:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601ef8d0 con 0x7fab600c3e40 2023-10-30T13:39:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab66e9c640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab601eed50 con 0x7fab600cb7d0 2023-10-30T13:39:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab60150830 0x7fab60148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab60150830 0x7fab60148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51608/0 (socket says 172.21.15.31:51608) 2023-10-30T13:39:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 -- 172.21.15.31:0/2035314134 learned_addr learned my addr 172.21.15.31:0/2035314134 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab64c38640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600c3e40 0x7fab60147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab5ffff640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab600cb7d0 0x7fab601475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 -- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab600cb7d0 msgr2=0x7fab601475f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab600cb7d0 0x7fab601475f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 -- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600c3e40 msgr2=0x7fab60147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600c3e40 0x7fab60147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:23.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 -- 172.21.15.31:0/2035314134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c00b6c0 con 0x7fab60150830 2023-10-30T13:39:23.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab5ffff640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab600cb7d0 0x7fab601475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:23.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.994+0000 7fab65439640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab60150830 0x7fab60148830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab58009fd0 tx=0x7fab5800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:23.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab5d7fa640 1 -- 172.21.15.31:0/2035314134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58014420 con 0x7fab60150830 2023-10-30T13:39:23.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab5d7fa640 1 -- 172.21.15.31:0/2035314134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab58014580 con 0x7fab60150830 2023-10-30T13:39:23.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab5d7fa640 1 -- 172.21.15.31:0/2035314134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58014930 con 0x7fab60150830 2023-10-30T13:39:23.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab601ef030 con 0x7fab60150830 2023-10-30T13:39:23.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab601ef5f0 con 0x7fab60150830 2023-10-30T13:39:23.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6004e170 con 0x7fab60150830 2023-10-30T13:39:23.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab5d7fa640 1 -- 172.21.15.31:0/2035314134 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab58014a90 con 0x7fab60150830 2023-10-30T13:39:23.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab5d7fa640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab440664f0 0x7fab440689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:23.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab5d7fa640 1 -- 172.21.15.31:0/2035314134 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fab580849d0 con 0x7fab60150830 2023-10-30T13:39:23.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:22.998+0000 7fab64c38640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab440664f0 0x7fab440689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:23.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.002+0000 7fab64c38640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab440664f0 0x7fab440689b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab54000f70 tx=0x7fab54009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:23.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.002+0000 7fab5d7fa640 1 -- 172.21.15.31:0/2035314134 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab58051c60 con 0x7fab60150830 2023-10-30T13:39:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab600c2aa0 con 0x7fab60150830 2023-10-30T13:39:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab5d7fa640 1 -- 172.21.15.31:0/2035314134 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fab58055280 con 0x7fab60150830 2023-10-30T13:39:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:23.199 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:17.878036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2023-10-30T13:39:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab440664f0 msgr2=0x7fab440689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab440664f0 0x7fab440689b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab54000f70 tx=0x7fab54009040 comp rx=0 tx=0).stop 2023-10-30T13:39:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab60150830 msgr2=0x7fab60148830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab60150830 0x7fab60148830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab58009fd0 tx=0x7fab5800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:39:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 shutdown_connections 2023-10-30T13:39:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab440664f0 0x7fab440689b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab60150830 0x7fab60148830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab600cb7d0 0x7fab601475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 --2- 172.21.15.31:0/2035314134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600c3e40 0x7fab60147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 >> 172.21.15.31:0/2035314134 conn(0x7fab600cac70 msgr2=0x7fab6014e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 shutdown_connections 2023-10-30T13:39:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:23.194+0000 7fab66e9c640 1 -- 172.21.15.31:0/2035314134 wait complete. 2023-10-30T13:39:23.220 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:25.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a1506b640 1 Processor -- start 2023-10-30T13:39:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a1506b640 1 -- start start 2023-10-30T13:39:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a1506b640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f1a10069e90 0x7f1a1006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a1506b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1015cd50 con 0x7f1a100c5c10 2023-10-30T13:39:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a1506b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1006f600 con 0x7f1a10159760 2023-10-30T13:39:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a1506b640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1015d0a0 con 0x7f1a10069e90 2023-10-30T13:39:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a0ed9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f1a100c5c10 0x7f1a1006eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55318/0 (socket says 172.21.15.31:55318) 2023-10-30T13:39:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.370+0000 7f1a0ed9d640 1 -- 172.21.15.31:0/518836216 learned_addr learned my addr 172.21.15.31:0/518836216 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:25.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0f59e640 1 --2- 172.21.15.31:0/518836216 >> v2:172.21.15.33:3300/0 conn(0x7f1a10069e90 0x7f1a1006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974297646 0 0) 0x7f1a1015cd50 con 0x7f1a100c5c10 2023-10-30T13:39:25.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f8003530 con 0x7f1a100c5c10 2023-10-30T13:39:25.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106462003 0 0) 0x7f19f8003530 con 0x7f1a100c5c10 2023-10-30T13:39:25.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 >> v2:172.21.15.33:3300/0 conn(0x7f1a10069e90 msgr2=0x7f1a1006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:25.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 --2- 172.21.15.31:0/518836216 >> v2:172.21.15.33:3300/0 conn(0x7f1a10069e90 0x7f1a1006a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 >> v1:172.21.15.33:6789/0 conn(0x7f1a10159760 legacy=0x7f1a1015d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100ca000 con 0x7f1a100c5c10 2023-10-30T13:39:25.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f1a00001680 con 0x7f1a100c5c10 2023-10-30T13:39:25.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 >> v1:172.21.15.31:6789/0 conn(0x7f1a100c5c10 legacy=0x7f1a1006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f8003980 0x7f19f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80047d0 0x7f19f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19f8005190 0x7f19f800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1015d0a0 con 0x7f19f80047d0 2023-10-30T13:39:25.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f800c030 con 0x7f19f8005190 2023-10-30T13:39:25.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f800c1f0 con 0x7f19f8003980 2023-10-30T13:39:25.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0e59c640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80047d0 0x7f19f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0ed9d640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f8003980 0x7f19f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0f59e640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19f8005190 0x7f19f800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0e59c640 1 -- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f8003980 msgr2=0x7f19f8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0e59c640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f8003980 0x7f19f8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0e59c640 1 -- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19f8005190 msgr2=0x7f19f800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0e59c640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19f8005190 0x7f19f800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0e59c640 1 -- 172.21.15.31:0/518836216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f8003530 con 0x7f19f80047d0 2023-10-30T13:39:25.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.374+0000 7f1a0f59e640 1 -- 172.21.15.31:0/518836216 reap_dead start 2023-10-30T13:39:25.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a0e59c640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80047d0 0x7f19f8004c50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1a04001780 tx=0x7f1a0401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55707bf7422030e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:25.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a040052d0 con 0x7f19f80047d0 2023-10-30T13:39:25.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a040a7050 con 0x7f19f80047d0 2023-10-30T13:39:25.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a0dd9b640 1 -- 172.21.15.31:0/518836216 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04003600 con 0x7f19f80047d0 2023-10-30T13:39:25.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 -- 172.21.15.31:0/518836216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80047d0 msgr2=0x7f19f8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80047d0 0x7f19f8004c50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1a04001780 tx=0x7f1a0401c3e0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 -- 172.21.15.31:0/518836216 shutdown_connections 2023-10-30T13:39:25.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 --2- 172.21.15.31:0/518836216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80047d0 0x7f19f8004c50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 -- 172.21.15.31:0/518836216 >> 172.21.15.31:0/518836216 conn(0x7f1a1006afa0 msgr2=0x7f1a1006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 -- 172.21.15.31:0/518836216 shutdown_connections 2023-10-30T13:39:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 -- 172.21.15.31:0/518836216 wait complete. 2023-10-30T13:39:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 Processor -- start 2023-10-30T13:39:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.378+0000 7f1a1506b640 1 -- start start 2023-10-30T13:39:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f80047d0 0x7f1a101f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a100b1520 0x7f1a10153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a100c1ae0 0x7f1a100c1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100bf870 con 0x7f1a100c1ae0 2023-10-30T13:39:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a101f2f10 con 0x7f1a100b1520 2023-10-30T13:39:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100ae110 con 0x7f19f80047d0 2023-10-30T13:39:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a100c1ae0 0x7f1a100c1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a100c1ae0 0x7f1a100c1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54578/0 (socket says 172.21.15.31:54578) 2023-10-30T13:39:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 -- 172.21.15.31:0/1290806543 learned_addr learned my addr 172.21.15.31:0/1290806543 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0ed9d640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f80047d0 0x7f1a101f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0e59c640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a100b1520 0x7f1a10153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 -- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f80047d0 msgr2=0x7f1a101f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f80047d0 0x7f1a101f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 -- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a100b1520 msgr2=0x7f1a10153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a100b1520 0x7f1a10153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 -- 172.21.15.31:0/1290806543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a101e4960 con 0x7f1a100c1ae0 2023-10-30T13:39:25.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a0f59e640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a100c1ae0 0x7f1a100c1f60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1a100b21c0 tx=0x7f1a08011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:25.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f19f77fe640 1 -- 172.21.15.31:0/1290806543 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0800e040 con 0x7f1a100c1ae0 2023-10-30T13:39:25.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f19f77fe640 1 -- 172.21.15.31:0/1290806543 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0800f040 con 0x7f1a100c1ae0 2023-10-30T13:39:25.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a101e4c00 con 0x7f1a100c1ae0 2023-10-30T13:39:25.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f19f77fe640 1 -- 172.21.15.31:0/1290806543 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0800feb0 con 0x7f1a100c1ae0 2023-10-30T13:39:25.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a100c12e0 con 0x7f1a100c1ae0 2023-10-30T13:39:25.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1004e170 con 0x7f1a100c1ae0 2023-10-30T13:39:25.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.382+0000 7f19f77fe640 1 -- 172.21.15.31:0/1290806543 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a08015430 con 0x7f1a100c1ae0 2023-10-30T13:39:25.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.386+0000 7f19f77fe640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19fc0664a0 0x7f19fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:25.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.386+0000 7f19f77fe640 1 -- 172.21.15.31:0/1290806543 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f1a08084920 con 0x7f1a100c1ae0 2023-10-30T13:39:25.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.386+0000 7f1a0ed9d640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19fc0664a0 0x7f19fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:25.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.386+0000 7f1a0ed9d640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19fc0664a0 0x7f19fc068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a00007cc0 tx=0x7f1a00001460 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:25.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.390+0000 7f19f77fe640 1 -- 172.21.15.31:0/1290806543 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a08051d00 con 0x7f1a100c1ae0 2023-10-30T13:39:25.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:25.414 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.582+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a1015ccc0 con 0x7f1a100c1ae0 2023-10-30T13:39:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.582+0000 7f19f77fe640 1 -- 172.21.15.31:0/1290806543 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f1a08055320 con 0x7f1a100c1ae0 2023-10-30T13:39:25.588 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:25.588 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:23.871273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:25.588 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2023-10-30T13:39:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19fc0664a0 msgr2=0x7f19fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19fc0664a0 0x7f19fc068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a00007cc0 tx=0x7f1a00001460 comp rx=0 tx=0).stop 2023-10-30T13:39:25.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a100c1ae0 msgr2=0x7f1a100c1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a100c1ae0 0x7f1a100c1f60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1a100b21c0 tx=0x7f1a08011f20 comp rx=0 tx=0).stop 2023-10-30T13:39:25.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 shutdown_connections 2023-10-30T13:39:25.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19fc0664a0 0x7f19fc068960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a100c1ae0 0x7f1a100c1f60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a100b1520 0x7f1a10153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 --2- 172.21.15.31:0/1290806543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f19f80047d0 0x7f1a101f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 >> 172.21.15.31:0/1290806543 conn(0x7f1a1006afa0 msgr2=0x7f1a100726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:25.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 shutdown_connections 2023-10-30T13:39:25.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:25.586+0000 7f1a1506b640 1 -- 172.21.15.31:0/1290806543 wait complete. 2023-10-30T13:39:25.609 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:27.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d2d5c4640 1 Processor -- start 2023-10-30T13:39:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d2d5c4640 1 -- start start 2023-10-30T13:39:27.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d2d5c4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8d2815cf90 0x7f8d280c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d2d5c4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d28159f20 con 0x7f8d280c4a10 2023-10-30T13:39:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d2d5c4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d280c47e0 con 0x7f8d28153c80 2023-10-30T13:39:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d2d5c4640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2815dec0 con 0x7f8d2815cf90 2023-10-30T13:39:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d267fc640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8d280c4a10 0x7f8d280ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55326/0 (socket says 172.21.15.31:55326) 2023-10-30T13:39:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d277fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8d2815cf90 0x7f8d280c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d267fc640 1 -- 172.21.15.31:0/1133002173 learned_addr learned my addr 172.21.15.31:0/1133002173 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:27.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d277fe640 1 -- 172.21.15.31:0/1133002173 >> v1:172.21.15.33:6789/0 conn(0x7f8d28153c80 legacy=0x7f8d2815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d277fe640 1 -- 172.21.15.31:0/1133002173 >> v1:172.21.15.31:6789/0 conn(0x7f8d280c4a10 legacy=0x7f8d280ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.762+0000 7f8d277fe640 1 -- 172.21.15.31:0/1133002173 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2815d620 con 0x7f8d2815cf90 2023-10-30T13:39:27.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d277fe640 1 --2- 172.21.15.31:0/1133002173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d2815cf90 0x7f8d280c42a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009d90 tx=0x7f8d2001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655ebcd64805c815 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:27.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d25ffb640 1 -- 172.21.15.31:0/1133002173 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d200af600 con 0x7f8d2815cf90 2023-10-30T13:39:27.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d25ffb640 1 -- 172.21.15.31:0/1133002173 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d200ab050 con 0x7f8d2815cf90 2023-10-30T13:39:27.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d25ffb640 1 -- 172.21.15.31:0/1133002173 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d200af980 con 0x7f8d2815cf90 2023-10-30T13:39:27.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/1133002173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d2815cf90 msgr2=0x7f8d280c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/1133002173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d2815cf90 0x7f8d280c42a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009d90 tx=0x7f8d2001f130 comp rx=0 tx=0).stop 2023-10-30T13:39:27.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/1133002173 shutdown_connections 2023-10-30T13:39:27.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/1133002173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d2815cf90 0x7f8d280c42a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/1133002173 >> 172.21.15.31:0/1133002173 conn(0x7f8d280715c0 msgr2=0x7f8d280719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:27.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/1133002173 shutdown_connections 2023-10-30T13:39:27.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/1133002173 wait complete. 2023-10-30T13:39:27.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.766+0000 7f8d2d5c4640 1 Processor -- start 2023-10-30T13:39:27.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 -- start start 2023-10-30T13:39:27.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d280c4a10 0x7f8d280ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d28153c80 0x7f8d280c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2815cf90 0x7f8d281f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280c8930 con 0x7f8d28153c80 2023-10-30T13:39:27.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280c7620 con 0x7f8d2815cf90 2023-10-30T13:39:27.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2814fb70 con 0x7f8d280c4a10 2023-10-30T13:39:27.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d28153c80 0x7f8d280c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d28153c80 0x7f8d280c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54590/0 (socket says 172.21.15.31:54590) 2023-10-30T13:39:27.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 -- 172.21.15.31:0/476166833 learned_addr learned my addr 172.21.15.31:0/476166833 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:27.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d26ffd640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d280c4a10 0x7f8d280ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d277fe640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2815cf90 0x7f8d281f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 -- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d280c4a10 msgr2=0x7f8d280ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d280c4a10 0x7f8d280ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 -- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2815cf90 msgr2=0x7f8d281f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2815cf90 0x7f8d281f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 -- 172.21.15.31:0/476166833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d20009a70 con 0x7f8d28153c80 2023-10-30T13:39:27.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d267fc640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d28153c80 0x7f8d280c8d50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c00c500 tx=0x7f8d1c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:27.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d0f7fe640 1 -- 172.21.15.31:0/476166833 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c005cf0 con 0x7f8d28153c80 2023-10-30T13:39:27.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d0f7fe640 1 -- 172.21.15.31:0/476166833 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c005e50 con 0x7f8d28153c80 2023-10-30T13:39:27.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2814fe50 con 0x7f8d28153c80 2023-10-30T13:39:27.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d0f7fe640 1 -- 172.21.15.31:0/476166833 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c0072f0 con 0x7f8d28153c80 2023-10-30T13:39:27.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2814e2b0 con 0x7f8d28153c80 2023-10-30T13:39:27.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.770+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d28159b90 con 0x7f8d28153c80 2023-10-30T13:39:27.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.774+0000 7f8d0f7fe640 1 -- 172.21.15.31:0/476166833 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d1c007450 con 0x7f8d28153c80 2023-10-30T13:39:27.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.774+0000 7f8d0f7fe640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d140664d0 0x7f8d14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.774+0000 7f8d0f7fe640 1 -- 172.21.15.31:0/476166833 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f8d1c083480 con 0x7f8d28153c80 2023-10-30T13:39:27.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.774+0000 7f8d26ffd640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d140664d0 0x7f8d14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.774+0000 7f8d26ffd640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d140664d0 0x7f8d14068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8d18007fd0 tx=0x7f8d18009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:27.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.778+0000 7f8d0f7fe640 1 -- 172.21.15.31:0/476166833 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d28159b90 con 0x7f8d28153c80 2023-10-30T13:39:27.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d28159b90 con 0x7f8d28153c80 2023-10-30T13:39:27.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d0f7fe640 1 -- 172.21.15.31:0/476166833 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f8d28159b90 con 0x7f8d28153c80 2023-10-30T13:39:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:27.979 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:23.871273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:27.979 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2023-10-30T13:39:27.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d140664d0 msgr2=0x7f8d14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d140664d0 0x7f8d14068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8d18007fd0 tx=0x7f8d18009040 comp rx=0 tx=0).stop 2023-10-30T13:39:27.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d28153c80 msgr2=0x7f8d280c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d28153c80 0x7f8d280c8d50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c00c500 tx=0x7f8d1c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 shutdown_connections 2023-10-30T13:39:27.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d140664d0 0x7f8d14068990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d2815cf90 0x7f8d281f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d28153c80 0x7f8d280c8d50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 --2- 172.21.15.31:0/476166833 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d280c4a10 0x7f8d280ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 >> 172.21.15.31:0/476166833 conn(0x7f8d280715c0 msgr2=0x7f8d28150150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:27.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 shutdown_connections 2023-10-30T13:39:27.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:27.974+0000 7f8d2d5c4640 1 -- 172.21.15.31:0/476166833 wait complete. 2023-10-30T13:39:28.001 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:30.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:30.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.150+0000 7f9facbc4640 1 Processor -- start 2023-10-30T13:39:30.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.150+0000 7f9facbc4640 1 -- start start 2023-10-30T13:39:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9facbc4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9fa806b760 0x7f9fa815e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9facbc4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806bb40 0x7f9fa8157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9facbc4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8071e00 con 0x7f9fa806c420 2023-10-30T13:39:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9facbc4640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa8070020 con 0x7f9fa806b760 2023-10-30T13:39:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9facbc4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa815eb90 con 0x7f9fa806bb40 2023-10-30T13:39:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9fa806b760 0x7f9fa815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9fa806b760 0x7f9fa815e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54592/0 (socket says 172.21.15.31:54592) 2023-10-30T13:39:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 -- 172.21.15.31:0/2987116702 learned_addr learned my addr 172.21.15.31:0/2987116702 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa5d9b640 1 --2- 172.21.15.31:0/2987116702 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806bb40 0x7f9fa8157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 -- 172.21.15.31:0/2987116702 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806bb40 msgr2=0x7f9fa8157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:30.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 --2- 172.21.15.31:0/2987116702 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806bb40 0x7f9fa8157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 -- 172.21.15.31:0/2987116702 >> v1:172.21.15.33:6789/0 conn(0x7f9fa806c420 legacy=0x7f9fa80c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:30.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 -- 172.21.15.31:0/2987116702 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8070f10 con 0x7f9fa806b760 2023-10-30T13:39:30.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.154+0000 7f9fa659c640 1 --2- 172.21.15.31:0/2987116702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806b760 0x7f9fa815e650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009870 tx=0x7f9f9801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346bd672d5e98f50 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9fa559a640 1 -- 172.21.15.31:0/2987116702 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f980aa670 con 0x7f9fa806b760 2023-10-30T13:39:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9fa559a640 1 -- 172.21.15.31:0/2987116702 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f980aa7d0 con 0x7f9fa806b760 2023-10-30T13:39:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9fa559a640 1 -- 172.21.15.31:0/2987116702 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f980aab60 con 0x7f9fa806b760 2023-10-30T13:39:30.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- 172.21.15.31:0/2987116702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806b760 msgr2=0x7f9fa815e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:30.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 --2- 172.21.15.31:0/2987116702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806b760 0x7f9fa815e650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009870 tx=0x7f9f9801c510 comp rx=0 tx=0).stop 2023-10-30T13:39:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- 172.21.15.31:0/2987116702 shutdown_connections 2023-10-30T13:39:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 --2- 172.21.15.31:0/2987116702 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806bb40 0x7f9fa8157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 --2- 172.21.15.31:0/2987116702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806b760 0x7f9fa815e650 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- 172.21.15.31:0/2987116702 >> 172.21.15.31:0/2987116702 conn(0x7f9fa80cac70 msgr2=0x7f9fa80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- 172.21.15.31:0/2987116702 shutdown_connections 2023-10-30T13:39:30.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- 172.21.15.31:0/2987116702 wait complete. 2023-10-30T13:39:30.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 Processor -- start 2023-10-30T13:39:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- start start 2023-10-30T13:39:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806b760 0x7f9fa8151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806bb40 0x7f9fa8149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fa806c420 0x7f9fa814e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa814b530 con 0x7f9fa806bb40 2023-10-30T13:39:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa81f67d0 con 0x7f9fa806c420 2023-10-30T13:39:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9facbc4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa814cab0 con 0x7f9fa806b760 2023-10-30T13:39:30.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.158+0000 7f9fa5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806bb40 0x7f9fa8149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:30.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806bb40 0x7f9fa8149e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54598/0 (socket says 172.21.15.31:54598) 2023-10-30T13:39:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 -- 172.21.15.31:0/3636867959 learned_addr learned my addr 172.21.15.31:0/3636867959 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa659c640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806b760 0x7f9fa8151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa6d9d640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fa806c420 0x7f9fa814e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 -- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806b760 msgr2=0x7f9fa8151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806b760 0x7f9fa8151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 -- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fa806c420 msgr2=0x7f9fa814e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fa806c420 0x7f9fa814e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 -- 172.21.15.31:0/3636867959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f980094d0 con 0x7f9fa806bb40 2023-10-30T13:39:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9fa5d9b640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806bb40 0x7f9fa8149e00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00d7d0 tx=0x7f9f9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9f96ffd640 1 -- 172.21.15.31:0/3636867959 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c018070 con 0x7f9fa806bb40 2023-10-30T13:39:30.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9f96ffd640 1 -- 172.21.15.31:0/3636867959 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c004020 con 0x7f9fa806bb40 2023-10-30T13:39:30.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9f96ffd640 1 -- 172.21.15.31:0/3636867959 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0041d0 con 0x7f9fa806bb40 2023-10-30T13:39:30.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa81f6a90 con 0x7f9fa806bb40 2023-10-30T13:39:30.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa814d810 con 0x7f9fa806bb40 2023-10-30T13:39:30.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8071a70 con 0x7f9fa806bb40 2023-10-30T13:39:30.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9f96ffd640 1 -- 172.21.15.31:0/3636867959 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f9c004330 con 0x7f9fa806bb40 2023-10-30T13:39:30.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.162+0000 7f9f96ffd640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f880664a0 0x7f9f88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.166+0000 7f9f96ffd640 1 -- 172.21.15.31:0/3636867959 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f9f9c083ec0 con 0x7f9fa806bb40 2023-10-30T13:39:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.166+0000 7f9fa659c640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f880664a0 0x7f9f88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.166+0000 7f9fa659c640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f880664a0 0x7f9f88068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f980ae000 tx=0x7f9f980b5040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:30.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.166+0000 7f9f96ffd640 1 -- 172.21.15.31:0/3636867959 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9fa8071a70 con 0x7f9fa806bb40 2023-10-30T13:39:30.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.358+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fa8071a70 con 0x7f9fa806bb40 2023-10-30T13:39:30.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.358+0000 7f9f96ffd640 1 -- 172.21.15.31:0/3636867959 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f9fa8071a70 con 0x7f9fa806bb40 2023-10-30T13:39:30.364 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:29.005610+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:30.364 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2023-10-30T13:39:30.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f880664a0 msgr2=0x7f9f88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f880664a0 0x7f9f88068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f980ae000 tx=0x7f9f980b5040 comp rx=0 tx=0).stop 2023-10-30T13:39:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806bb40 msgr2=0x7f9fa8149e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806bb40 0x7f9fa8149e00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00d7d0 tx=0x7f9f9c00dc90 comp rx=0 tx=0).stop 2023-10-30T13:39:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 shutdown_connections 2023-10-30T13:39:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9f880664a0 0x7f9f88068960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fa806c420 0x7f9fa814e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fa806bb40 0x7f9fa8149e00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 --2- 172.21.15.31:0/3636867959 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9fa806b760 0x7f9fa8151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:30.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 >> 172.21.15.31:0/3636867959 conn(0x7f9fa80cac70 msgr2=0x7f9fa80722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:30.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 shutdown_connections 2023-10-30T13:39:30.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:30.362+0000 7f9facbc4640 1 -- 172.21.15.31:0/3636867959 wait complete. 2023-10-30T13:39:30.384 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=2, replacing gid=4272 2023-10-30T13:39:30.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2023-10-30T13:39:30.564 INFO:tasks.ceph.mds.b.smithi033.stderr:2023-10-30T13:39:30.561+0000 7f01d8981640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:39:35.150 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-30T13:39:35.150 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-30T13:39:35.150 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-30T13:39:35.151 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-30T13:39:35.151 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-30T13:39:35.151 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-30T13:39:35.151 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-30T13:39:35.151 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-30T13:39:35.152 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-30T13:39:35.152 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-30T13:39:35.152 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-30T13:39:35.152 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-30T13:39:35.152 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-30T13:39:35.152 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-30T13:39:35.153 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-30T13:39:35.153 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-30T13:39:35.153 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-30T13:39:35.153 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-30T13:39:35.153 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-30T13:39:35.154 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-30T13:39:35.154 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-30T13:39:35.154 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-30T13:39:35.154 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-30T13:39:35.154 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-30T13:39:35.155 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-30T13:39:35.155 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-30T13:39:35.155 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-30T13:39:35.155 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-30T13:39:35.155 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-30T13:39:35.155 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-30T13:39:35.156 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-30T13:39:35.156 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-30T13:39:35.156 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-30T13:39:35.156 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-30T13:39:35.156 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-30T13:39:35.156 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-30T13:39:35.157 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-30T13:39:35.157 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-30T13:39:35.157 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-30T13:39:35.157 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-30T13:39:35.157 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-30T13:39:35.158 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-30T13:39:35.158 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-30T13:39:35.158 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-30T13:39:35.158 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-30T13:39:35.158 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-30T13:39:35.159 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-30T13:39:35.159 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-30T13:39:35.159 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-30T13:39:35.159 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-30T13:39:35.159 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-30T13:39:35.160 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-30T13:39:35.160 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-30T13:39:35.160 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-30T13:39:35.160 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-30T13:39:35.160 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-30T13:39:35.161 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-30T13:39:35.161 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-30T13:39:35.161 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-30T13:39:35.161 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-30T13:39:35.161 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-30T13:39:35.162 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-30T13:39:35.162 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-30T13:39:35.162 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-30T13:39:35.162 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-30T13:39:35.162 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-30T13:39:35.163 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-30T13:39:35.163 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-30T13:39:35.163 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-30T13:39:35.163 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-30T13:39:35.163 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-30T13:39:35.163 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-30T13:39:35.164 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-30T13:39:35.164 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-30T13:39:35.164 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-30T13:39:35.164 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-30T13:39:35.164 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-30T13:39:35.164 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-30T13:39:35.165 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-30T13:39:35.165 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-30T13:39:35.165 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-30T13:39:36.292 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-30T13:39:36.292 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-30T13:39:36.292 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-30T13:39:36.292 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-30T13:39:36.293 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-30T13:39:36.293 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-30T13:39:36.293 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-30T13:39:36.293 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-30T13:39:36.293 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-30T13:39:36.294 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-30T13:39:36.294 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-30T13:39:36.294 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-30T13:39:36.294 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-30T13:39:36.294 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-30T13:39:36.294 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-30T13:39:36.295 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-30T13:39:36.295 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-30T13:39:36.295 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-30T13:39:36.295 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-30T13:39:36.295 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-30T13:39:36.295 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-30T13:39:36.296 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-30T13:39:36.296 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-30T13:39:36.296 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-30T13:39:36.296 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-30T13:39:36.296 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-30T13:39:36.297 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-30T13:39:36.297 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-30T13:39:36.297 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-30T13:39:36.297 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-30T13:39:36.297 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-30T13:39:36.297 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-30T13:39:36.298 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-30T13:39:36.298 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-30T13:39:36.298 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-30T13:39:36.298 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-30T13:39:36.298 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-30T13:39:36.299 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-30T13:39:36.299 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-30T13:39:36.299 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-30T13:39:36.299 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-30T13:39:36.299 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-30T13:39:36.299 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-30T13:39:36.300 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-30T13:39:36.300 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-30T13:39:36.300 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-30T13:39:36.300 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-30T13:39:36.300 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-30T13:39:36.301 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-30T13:39:36.301 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-30T13:39:36.301 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-30T13:39:36.301 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-30T13:39:36.301 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-30T13:39:36.301 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-30T13:39:36.302 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-30T13:39:36.302 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-30T13:39:36.302 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-30T13:39:36.302 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-30T13:39:36.302 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-30T13:39:36.303 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-30T13:39:36.303 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-30T13:39:36.303 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-30T13:39:36.303 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-30T13:39:36.303 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-30T13:39:36.304 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-30T13:39:36.304 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-30T13:39:36.304 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-30T13:39:36.304 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-30T13:39:36.304 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-30T13:39:36.304 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-30T13:39:36.305 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-30T13:39:36.305 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-30T13:39:36.305 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-30T13:39:36.305 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-30T13:39:36.305 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-30T13:39:36.305 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-30T13:39:36.306 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-30T13:39:36.306 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-30T13:39:36.306 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-30T13:39:36.306 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-30T13:39:36.306 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-30T13:39:36.307 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-30T13:39:37.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-30T13:39:37.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-30T13:39:37.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-30T13:39:37.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-30T13:39:37.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-30T13:39:37.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-30T13:39:37.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-30T13:39:37.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-30T13:39:37.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-30T13:39:37.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-30T13:39:37.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-30T13:39:37.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-30T13:39:37.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-30T13:39:37.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-30T13:39:37.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-30T13:39:37.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-30T13:39:37.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-30T13:39:37.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-30T13:39:37.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-30T13:39:37.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-30T13:39:37.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-30T13:39:37.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-30T13:39:37.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-30T13:39:37.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-30T13:39:37.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-30T13:39:37.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-30T13:39:37.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-30T13:39:37.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-30T13:39:37.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-30T13:39:37.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-30T13:39:37.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-30T13:39:37.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-30T13:39:37.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-30T13:39:37.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-30T13:39:37.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-30T13:39:37.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-30T13:39:37.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-30T13:39:37.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-30T13:39:37.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-30T13:39:37.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-30T13:39:37.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-30T13:39:37.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-30T13:39:37.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-30T13:39:37.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-30T13:39:37.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-30T13:39:37.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-30T13:39:37.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-30T13:39:37.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-30T13:39:37.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-30T13:39:37.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-30T13:39:37.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-30T13:39:37.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-30T13:39:37.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-30T13:39:37.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-30T13:39:37.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-30T13:39:37.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-30T13:39:37.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-30T13:39:37.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-30T13:39:37.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-30T13:39:37.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-30T13:39:37.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-30T13:39:37.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-30T13:39:37.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-30T13:39:37.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-30T13:39:37.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-30T13:39:37.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-30T13:39:37.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-30T13:39:37.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-30T13:39:37.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-30T13:39:37.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-30T13:39:37.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-30T13:39:37.143 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-30T13:39:37.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-30T13:39:37.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-30T13:39:37.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-30T13:39:37.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-30T13:39:37.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-30T13:39:37.144 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-30T13:39:37.145 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-30T13:39:37.145 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-30T13:39:37.145 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-30T13:39:37.385 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:39:37.386 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:39:37.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:39:37.390 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:39:37.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:37.483 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2023-10-30T13:39:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f73072c9640 1 Processor -- start 2023-10-30T13:39:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f73072c9640 1 -- start start 2023-10-30T13:39:37.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f73072c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7300153c80 0x7f730015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:37.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f73072c9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f73000c42a0 0x7f73000ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:37.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f73072c9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7300159f20 con 0x7f73000c4a10 2023-10-30T13:39:37.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f73072c9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730015dec0 con 0x7f73000c42a0 2023-10-30T13:39:37.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f73072c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73001599a0 con 0x7f7300153c80 2023-10-30T13:39:37.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305065640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7300153c80 0x7f730015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:37.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305866640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f73000c42a0 0x7f73000ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:37.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305065640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7300153c80 0x7f730015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:52830/0 (socket says 172.21.15.31:52830) 2023-10-30T13:39:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305866640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f73000c42a0 0x7f73000ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:34524/0 (socket says 172.21.15.31:34524) 2023-10-30T13:39:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305065640 1 -- 172.21.15.31:0/4079969046 learned_addr learned my addr 172.21.15.31:0/4079969046 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305065640 1 -- 172.21.15.31:0/4079969046 >> v2:172.21.15.33:3300/0 conn(0x7f73000c42a0 msgr2=0x7f73000ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305065640 1 --2- 172.21.15.31:0/4079969046 >> v2:172.21.15.33:3300/0 conn(0x7f73000c42a0 0x7f73000ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305065640 1 -- 172.21.15.31:0/4079969046 >> v1:172.21.15.33:6789/0 conn(0x7f73000c4a10 legacy=0x7f730015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.538+0000 7f7305065640 1 -- 172.21.15.31:0/4079969046 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730015d620 con 0x7f7300153c80 2023-10-30T13:39:37.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f7305065640 1 --2- 172.21.15.31:0/4079969046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7300153c80 0x7f730015cf90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009870 tx=0x7f72fc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fbde63cd9b626d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:37.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f72f7fff640 1 -- 172.21.15.31:0/4079969046 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0a8a30 con 0x7f7300153c80 2023-10-30T13:39:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f72f7fff640 1 -- 172.21.15.31:0/4079969046 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc0a8b90 con 0x7f7300153c80 2023-10-30T13:39:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f72f7fff640 1 -- 172.21.15.31:0/4079969046 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0b0440 con 0x7f7300153c80 2023-10-30T13:39:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 -- 172.21.15.31:0/4079969046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7300153c80 msgr2=0x7f730015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 --2- 172.21.15.31:0/4079969046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7300153c80 0x7f730015cf90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009870 tx=0x7f72fc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:39:37.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 -- 172.21.15.31:0/4079969046 shutdown_connections 2023-10-30T13:39:37.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 --2- 172.21.15.31:0/4079969046 >> v2:172.21.15.33:3300/0 conn(0x7f73000c42a0 0x7f73000ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 --2- 172.21.15.31:0/4079969046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7300153c80 0x7f730015cf90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 -- 172.21.15.31:0/4079969046 >> 172.21.15.31:0/4079969046 conn(0x7f73000715c0 msgr2=0x7f73000719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:37.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 -- 172.21.15.31:0/4079969046 shutdown_connections 2023-10-30T13:39:37.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.542+0000 7f73072c9640 1 -- 172.21.15.31:0/4079969046 wait complete. 2023-10-30T13:39:37.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 Processor -- start 2023-10-30T13:39:37.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 -- start start 2023-10-30T13:39:37.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73000c42a0 0x7f730014fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:37.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73000c4a10 0x7f73000ce890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:37.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7300153c80 0x7f73001557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73001f30c0 con 0x7f7300153c80 2023-10-30T13:39:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73001554e0 con 0x7f73000c4a10 2023-10-30T13:39:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73001f2480 con 0x7f73000c42a0 2023-10-30T13:39:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7300153c80 0x7f73001557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7300153c80 0x7f73001557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49772/0 (socket says 172.21.15.31:49772) 2023-10-30T13:39:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 -- 172.21.15.31:0/2704444469 learned_addr learned my addr 172.21.15.31:0/2704444469 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305065640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73000c42a0 0x7f730014fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7304864640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73000c4a10 0x7f73000ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 -- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73000c42a0 msgr2=0x7f730014fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73000c42a0 0x7f730014fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 -- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73000c4a10 msgr2=0x7f73000ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73000c4a10 0x7f73000ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 -- 172.21.15.31:0/2704444469 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0094d0 con 0x7f7300153c80 2023-10-30T13:39:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305065640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73000c42a0 0x7f730014fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f7305866640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7300153c80 0x7f73001557a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f72f800efc0 tx=0x7f72f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:37.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f72f5ffb640 1 -- 172.21.15.31:0/2704444469 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8014480 con 0x7f7300153c80 2023-10-30T13:39:37.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f72f5ffb640 1 -- 172.21.15.31:0/2704444469 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f800f040 con 0x7f7300153c80 2023-10-30T13:39:37.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f72f5ffb640 1 -- 172.21.15.31:0/2704444469 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8014820 con 0x7f7300153c80 2023-10-30T13:39:37.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73001f2720 con 0x7f7300153c80 2023-10-30T13:39:37.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73001f2ce0 con 0x7f7300153c80 2023-10-30T13:39:37.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.546+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730004e170 con 0x7f7300153c80 2023-10-30T13:39:37.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.550+0000 7f72f5ffb640 1 -- 172.21.15.31:0/2704444469 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72f8014980 con 0x7f7300153c80 2023-10-30T13:39:37.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.550+0000 7f72f5ffb640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f72e40664d0 0x7f72e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:37.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.550+0000 7f72f5ffb640 1 -- 172.21.15.31:0/2704444469 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f72f8084bf0 con 0x7f7300153c80 2023-10-30T13:39:37.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.550+0000 7f7305065640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f72e40664d0 0x7f72e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:37.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.550+0000 7f7305065640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f72e40664d0 0x7f72e4068990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0b2000 tx=0x7f72fc0aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:37.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.554+0000 7f72f5ffb640 1 -- 172.21.15.31:0/2704444469 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f72f8051e00 con 0x7f7300153c80 2023-10-30T13:39:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.742+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73000b1b20 con 0x7f7300153c80 2023-10-30T13:39:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.742+0000 7f72f5ffb640 1 -- 172.21.15.31:0/2704444469 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f72f8018070 con 0x7f7300153c80 2023-10-30T13:39:37.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:37.748 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:29.005610+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2023-10-30T13:39:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.742+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f72e40664d0 msgr2=0x7f72e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.742+0000 7f73072c9640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f72e40664d0 0x7f72e4068990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0b2000 tx=0x7f72fc0aa070 comp rx=0 tx=0).stop 2023-10-30T13:39:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.742+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7300153c80 msgr2=0x7f73001557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.742+0000 7f73072c9640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7300153c80 0x7f73001557a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f72f800efc0 tx=0x7f72f800c490 comp rx=0 tx=0).stop 2023-10-30T13:39:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 shutdown_connections 2023-10-30T13:39:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f72e40664d0 0x7f72e4068990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7300153c80 0x7f73001557a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73000c4a10 0x7f73000ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 --2- 172.21.15.31:0/2704444469 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73000c42a0 0x7f730014fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 >> 172.21.15.31:0/2704444469 conn(0x7f73000715c0 msgr2=0x7f73001590e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 shutdown_connections 2023-10-30T13:39:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:37.746+0000 7f73072c9640 1 -- 172.21.15.31:0/2704444469 wait complete. 2023-10-30T13:39:37.768 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:39:38.128 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-30T13:39:38.128 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-30T13:39:38.128 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-30T13:39:38.129 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-30T13:39:38.129 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-30T13:39:38.129 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-30T13:39:38.129 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-30T13:39:38.129 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-30T13:39:38.129 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-30T13:39:38.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-30T13:39:38.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-30T13:39:38.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-30T13:39:38.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-30T13:39:38.130 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-30T13:39:38.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-30T13:39:38.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-30T13:39:38.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-30T13:39:38.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-30T13:39:38.131 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-30T13:39:38.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-30T13:39:38.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-30T13:39:38.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-30T13:39:38.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-30T13:39:38.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-30T13:39:38.132 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-30T13:39:38.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-30T13:39:38.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-30T13:39:38.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-30T13:39:38.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-30T13:39:38.133 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-30T13:39:38.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-30T13:39:38.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-30T13:39:38.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-30T13:39:38.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-30T13:39:38.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-30T13:39:38.134 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-30T13:39:38.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-30T13:39:38.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-30T13:39:38.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-30T13:39:38.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-30T13:39:38.135 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-30T13:39:38.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-30T13:39:38.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-30T13:39:38.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-30T13:39:38.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-30T13:39:38.136 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-30T13:39:38.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-30T13:39:38.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-30T13:39:38.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-30T13:39:38.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-30T13:39:38.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-30T13:39:38.137 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-30T13:39:38.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-30T13:39:38.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-30T13:39:38.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-30T13:39:38.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-30T13:39:38.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-30T13:39:38.138 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-30T13:39:38.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-30T13:39:38.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-30T13:39:38.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-30T13:39:38.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-30T13:39:38.139 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-30T13:39:38.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-30T13:39:38.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-30T13:39:38.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-30T13:39:38.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-30T13:39:38.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-30T13:39:38.140 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-30T13:39:38.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-30T13:39:38.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-30T13:39:38.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-30T13:39:38.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-30T13:39:38.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-30T13:39:38.141 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-30T13:39:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-30T13:39:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-30T13:39:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-30T13:39:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-30T13:39:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-30T13:39:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-30T13:39:39.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:39.877 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-30T13:39:39.878 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-30T13:39:39.878 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-30T13:39:39.878 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-30T13:39:39.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-30T13:39:39.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-30T13:39:39.879 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-30T13:39:39.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-30T13:39:39.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-30T13:39:39.880 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-30T13:39:39.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-30T13:39:39.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-30T13:39:39.881 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-30T13:39:39.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-30T13:39:39.882 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-30T13:39:39.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-30T13:39:39.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-30T13:39:39.883 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-30T13:39:39.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-30T13:39:39.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-30T13:39:39.884 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-30T13:39:39.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-30T13:39:39.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-30T13:39:39.885 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-30T13:39:39.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-30T13:39:39.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-30T13:39:39.886 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-30T13:39:39.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-30T13:39:39.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-30T13:39:39.887 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-30T13:39:39.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-30T13:39:39.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-30T13:39:39.888 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-30T13:39:39.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-30T13:39:39.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-30T13:39:39.889 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-30T13:39:39.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-30T13:39:39.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-30T13:39:39.890 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-30T13:39:39.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-30T13:39:39.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-30T13:39:39.891 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-30T13:39:39.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-30T13:39:39.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-30T13:39:39.892 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-30T13:39:39.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-30T13:39:39.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-30T13:39:39.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-30T13:39:39.893 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-30T13:39:39.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-30T13:39:39.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-30T13:39:39.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-30T13:39:39.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-30T13:39:39.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-30T13:39:39.894 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-30T13:39:39.895 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-30T13:39:39.895 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-30T13:39:39.895 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-30T13:39:39.895 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-30T13:39:39.895 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-30T13:39:39.896 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-30T13:39:39.896 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-30T13:39:39.896 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-30T13:39:39.896 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-30T13:39:39.896 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-30T13:39:39.896 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-30T13:39:39.897 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-30T13:39:39.897 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-30T13:39:39.897 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-30T13:39:39.897 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-30T13:39:39.897 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-30T13:39:39.898 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-30T13:39:39.898 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-30T13:39:39.898 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-30T13:39:39.898 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-30T13:39:39.898 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-30T13:39:39.899 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-30T13:39:39.899 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-30T13:39:39.899 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-30T13:39:39.900 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-30T13:39:39.900 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-30T13:39:39.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.922+0000 7fce4eec9640 1 Processor -- start 2023-10-30T13:39:39.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4eec9640 1 -- start start 2023-10-30T13:39:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4eec9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fce48070520 0x7fce48070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4eec9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce480c2310 con 0x7fce48150830 2023-10-30T13:39:39.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4eec9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce48156710 con 0x7fce480cb7d0 2023-10-30T13:39:39.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4eec9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4806bc10 con 0x7fce48070520 2023-10-30T13:39:39.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4cc65640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fce48150830 0x7fce4806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:40816/0 (socket says 172.21.15.31:40816) 2023-10-30T13:39:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4cc65640 1 -- 172.21.15.31:0/1555817970 learned_addr learned my addr 172.21.15.31:0/1555817970 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:39:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4d466640 1 --2- 172.21.15.31:0/1555817970 >> v2:172.21.15.33:3300/0 conn(0x7fce48070520 0x7fce48070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367836480 0 0) 0x7fce480c2310 con 0x7fce48150830 2023-10-30T13:39:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce38003530 con 0x7fce48150830 2023-10-30T13:39:39.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915038244 0 0) 0x7fce38003530 con 0x7fce48150830 2023-10-30T13:39:39.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 >> v2:172.21.15.33:3300/0 conn(0x7fce48070520 msgr2=0x7fce48070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:39.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 --2- 172.21.15.31:0/1555817970 >> v2:172.21.15.33:3300/0 conn(0x7fce48070520 0x7fce48070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 >> v1:172.21.15.33:6789/0 conn(0x7fce480cb7d0 legacy=0x7fce480c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce480c4360 con 0x7fce48150830 2023-10-30T13:39:39.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fce3c0016a0 con 0x7fce48150830 2023-10-30T13:39:39.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 >> v1:172.21.15.31:6789/0 conn(0x7fce48150830 legacy=0x7fce4806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce38003980 0x7fce38003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce380047d0 0x7fce38004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38005190 0x7fce3800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4806bc10 con 0x7fce380047d0 2023-10-30T13:39:39.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3800c040 con 0x7fce38005190 2023-10-30T13:39:39.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3800c200 con 0x7fce38003980 2023-10-30T13:39:39.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce47fff640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce380047d0 0x7fce38004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4cc65640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce38003980 0x7fce38003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4d466640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38005190 0x7fce3800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce47fff640 1 -- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce38003980 msgr2=0x7fce38003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:39.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce47fff640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce38003980 0x7fce38003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce47fff640 1 -- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38005190 msgr2=0x7fce3800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:39.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce47fff640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38005190 0x7fce3800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce47fff640 1 -- 172.21.15.31:0/1555817970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38003530 con 0x7fce380047d0 2023-10-30T13:39:39.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.926+0000 7fce4d466640 1 -- 172.21.15.31:0/1555817970 reap_dead start 2023-10-30T13:39:39.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce47fff640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce380047d0 0x7fce38004c50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce34001780 tx=0x7fce3401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe9f402f1808171 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:39.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340052d0 con 0x7fce380047d0 2023-10-30T13:39:39.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce340a7050 con 0x7fce380047d0 2023-10-30T13:39:39.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce477fe640 1 -- 172.21.15.31:0/1555817970 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34003600 con 0x7fce380047d0 2023-10-30T13:39:39.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce4eec9640 1 -- 172.21.15.31:0/1555817970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce380047d0 msgr2=0x7fce38004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce380047d0 0x7fce38004c50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce34001780 tx=0x7fce3401c3e0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce4eec9640 1 -- 172.21.15.31:0/1555817970 shutdown_connections 2023-10-30T13:39:39.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1555817970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce380047d0 0x7fce38004c50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce4eec9640 1 -- 172.21.15.31:0/1555817970 >> 172.21.15.31:0/1555817970 conn(0x7fce480cac70 msgr2=0x7fce480cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:39.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce4eec9640 1 -- 172.21.15.31:0/1555817970 shutdown_connections 2023-10-30T13:39:39.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.930+0000 7fce4eec9640 1 -- 172.21.15.31:0/1555817970 wait complete. 2023-10-30T13:39:39.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 Processor -- start 2023-10-30T13:39:39.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 -- start start 2023-10-30T13:39:39.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce380047d0 0x7fce48141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4814c370 0x7fce4814b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce4814b5e0 0x7fce48145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce48148a30 con 0x7fce4814c370 2023-10-30T13:39:39.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4814ba60 con 0x7fce4814b5e0 2023-10-30T13:39:39.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce481eb410 con 0x7fce380047d0 2023-10-30T13:39:39.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4814c370 0x7fce4814b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4814c370 0x7fce4814b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49796/0 (socket says 172.21.15.31:49796) 2023-10-30T13:39:39.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 -- 172.21.15.31:0/1547575348 learned_addr learned my addr 172.21.15.31:0/1547575348 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:39:39.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4cc65640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce380047d0 0x7fce48141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4d466640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce4814b5e0 0x7fce48145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 -- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce380047d0 msgr2=0x7fce48141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce380047d0 0x7fce48141890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 -- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce4814b5e0 msgr2=0x7fce48145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce4814b5e0 0x7fce48145870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 -- 172.21.15.31:0/1547575348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce480ca9f0 con 0x7fce4814c370 2023-10-30T13:39:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4d466640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce4814b5e0 0x7fce48145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce47fff640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4814c370 0x7fce4814b0a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fce34005040 tx=0x7fce38005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:39.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce457fa640 1 -- 172.21.15.31:0/1547575348 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34007500 con 0x7fce4814c370 2023-10-30T13:39:39.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce457fa640 1 -- 172.21.15.31:0/1547575348 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce34007680 con 0x7fce4814c370 2023-10-30T13:39:39.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce481ea800 con 0x7fce4814c370 2023-10-30T13:39:39.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce457fa640 1 -- 172.21.15.31:0/1547575348 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34006a20 con 0x7fce4814c370 2023-10-30T13:39:39.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.934+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce481eadc0 con 0x7fce4814c370 2023-10-30T13:39:39.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.938+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4804e170 con 0x7fce4814c370 2023-10-30T13:39:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.938+0000 7fce457fa640 1 -- 172.21.15.31:0/1547575348 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce34006b80 con 0x7fce4814c370 2023-10-30T13:39:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.938+0000 7fce457fa640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fce2c066490 0x7fce2c068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.938+0000 7fce457fa640 1 -- 172.21.15.31:0/1547575348 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fce3411e1e0 con 0x7fce4814c370 2023-10-30T13:39:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.938+0000 7fce4cc65640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fce2c066490 0x7fce2c068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.938+0000 7fce4cc65640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fce2c066490 0x7fce2c068950 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c10 tx=0x7fce3c001c50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:39.942+0000 7fce457fa640 1 -- 172.21.15.31:0/1547575348 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fce340ef030 con 0x7fce4814c370 2023-10-30T13:39:40.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce48070300 con 0x7fce4814c370 2023-10-30T13:39:40.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce457fa640 1 -- 172.21.15.31:0/1547575348 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6523 (secure 0 0 0) 0x7fce340eebc0 con 0x7fce4814c370 2023-10-30T13:39:40.170 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:39:40.171 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:39:29.005610+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:40.171 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2023-10-30T13:39:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fce2c066490 msgr2=0x7fce2c068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fce2c066490 0x7fce2c068950 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c10 tx=0x7fce3c001c50 comp rx=0 tx=0).stop 2023-10-30T13:39:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4814c370 msgr2=0x7fce4814b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4814c370 0x7fce4814b0a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fce34005040 tx=0x7fce38005800 comp rx=0 tx=0).stop 2023-10-30T13:39:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 shutdown_connections 2023-10-30T13:39:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fce2c066490 0x7fce2c068950 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce4814b5e0 0x7fce48145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4814c370 0x7fce4814b0a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 --2- 172.21.15.31:0/1547575348 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce380047d0 0x7fce48141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.166+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 >> 172.21.15.31:0/1547575348 conn(0x7fce480cac70 msgr2=0x7fce480c58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:40.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.170+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 shutdown_connections 2023-10-30T13:39:40.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:39:40.170+0000 7fce4eec9640 1 -- 172.21.15.31:0/1547575348 wait complete. 2023-10-30T13:39:40.197 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:39:40.197 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 61 secs before thrashing 2023-10-30T13:39:42.547 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-30T13:39:42.547 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-30T13:39:42.547 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-30T13:39:42.548 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-30T13:39:42.548 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-30T13:39:42.548 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-30T13:39:42.548 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-30T13:39:42.548 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-30T13:39:42.549 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-30T13:39:42.549 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-30T13:39:42.549 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-30T13:39:42.549 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-30T13:39:42.549 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-30T13:39:42.550 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-30T13:39:42.550 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-30T13:39:42.550 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-30T13:39:42.550 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-30T13:39:42.550 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-30T13:39:42.551 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-30T13:39:42.551 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-30T13:39:42.551 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-30T13:39:42.551 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-30T13:39:42.551 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-30T13:39:42.551 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-30T13:39:42.552 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-30T13:39:42.552 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-30T13:39:42.552 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-30T13:39:42.552 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-30T13:39:42.552 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-30T13:39:42.552 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-30T13:39:42.553 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-30T13:39:42.553 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-30T13:39:42.553 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-30T13:39:42.553 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-30T13:39:42.553 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-30T13:39:42.554 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-30T13:39:42.554 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-30T13:39:42.554 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-30T13:39:42.554 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-30T13:39:42.554 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-30T13:39:42.554 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-30T13:39:42.555 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-30T13:39:42.555 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-30T13:39:42.555 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-30T13:39:42.555 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-30T13:39:42.556 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-30T13:39:42.556 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-30T13:39:42.556 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-30T13:39:42.556 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-30T13:39:42.556 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-30T13:39:42.557 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-30T13:39:42.557 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-30T13:39:42.557 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-30T13:39:42.557 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-30T13:39:42.557 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-30T13:39:42.557 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-30T13:39:42.558 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-30T13:39:42.558 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-30T13:39:42.558 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-30T13:39:42.558 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-30T13:39:42.559 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-30T13:39:42.559 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-30T13:39:42.559 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-30T13:39:42.559 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-30T13:39:42.559 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-30T13:39:42.559 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-30T13:39:42.560 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-30T13:39:42.560 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-30T13:39:42.560 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-30T13:39:42.560 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-30T13:39:42.560 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-30T13:39:42.560 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-30T13:39:42.561 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-30T13:39:42.561 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-30T13:39:42.561 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-30T13:39:42.561 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-30T13:39:42.561 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-30T13:39:42.561 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-30T13:39:42.562 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-30T13:39:42.562 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-30T13:39:42.562 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-30T13:39:44.087 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-30T13:39:44.087 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-30T13:39:44.087 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-30T13:39:44.088 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-30T13:39:44.088 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-30T13:39:44.088 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-30T13:39:44.088 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-30T13:39:44.088 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-30T13:39:44.088 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-30T13:39:44.089 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-30T13:39:44.089 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-30T13:39:44.089 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-30T13:39:44.089 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-30T13:39:44.089 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-30T13:39:44.089 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-30T13:39:44.090 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-30T13:39:44.090 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-30T13:39:44.090 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-30T13:39:44.090 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-30T13:39:44.090 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-30T13:39:44.090 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-30T13:39:44.091 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-30T13:39:44.091 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-30T13:39:44.091 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-30T13:39:44.091 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-30T13:39:44.091 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-30T13:39:44.091 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-30T13:39:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-30T13:39:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-30T13:39:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-30T13:39:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-30T13:39:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-30T13:39:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-30T13:39:44.093 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-30T13:39:44.093 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-30T13:39:44.093 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-30T13:39:44.093 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-30T13:39:44.093 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-30T13:39:44.093 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-30T13:39:44.094 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-30T13:39:44.094 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-30T13:39:44.094 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-30T13:39:44.094 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-30T13:39:44.094 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-30T13:39:44.095 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-30T13:39:44.095 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-30T13:39:44.095 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-30T13:39:44.095 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-30T13:39:44.095 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-30T13:39:44.095 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-30T13:39:44.096 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-30T13:39:44.096 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-30T13:39:44.096 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-30T13:39:44.096 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-30T13:39:44.096 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-30T13:39:44.097 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-30T13:39:44.097 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-30T13:39:44.097 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-30T13:39:44.097 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-30T13:39:44.097 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-30T13:39:44.097 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-30T13:39:44.098 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-30T13:39:44.098 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-30T13:39:44.098 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-30T13:39:44.098 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-30T13:39:44.098 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-30T13:39:44.098 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-30T13:39:44.099 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-30T13:39:44.099 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-30T13:39:44.099 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-30T13:39:44.099 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-30T13:39:44.099 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-30T13:39:44.099 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-30T13:39:44.100 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-30T13:39:44.100 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-30T13:39:44.100 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-30T13:39:44.100 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-30T13:39:44.100 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-30T13:39:44.100 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-30T13:39:44.101 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-30T13:39:44.101 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-30T13:39:45.170 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-30T13:39:45.171 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-30T13:39:45.171 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-30T13:39:45.171 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-30T13:39:45.172 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-30T13:39:45.172 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-30T13:39:45.172 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-30T13:39:45.172 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-30T13:39:45.172 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-30T13:39:45.173 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-30T13:39:45.173 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-30T13:39:45.173 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-30T13:39:45.173 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-30T13:39:45.174 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-30T13:39:45.174 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-30T13:39:45.174 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-30T13:39:45.174 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-30T13:39:45.174 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-30T13:39:45.175 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-30T13:39:45.175 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-30T13:39:45.175 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-30T13:39:45.175 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-30T13:39:45.175 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-30T13:39:45.176 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-30T13:39:45.176 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-30T13:39:45.176 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-30T13:39:45.176 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-30T13:39:45.176 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-30T13:39:45.177 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-30T13:39:45.177 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-30T13:39:45.177 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-30T13:39:45.177 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-30T13:39:45.177 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-30T13:39:45.178 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-30T13:39:45.178 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-30T13:39:45.178 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-30T13:39:45.178 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-30T13:39:45.179 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-30T13:39:45.179 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-30T13:39:45.179 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-30T13:39:45.179 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-30T13:39:45.179 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-30T13:39:45.179 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-30T13:39:45.180 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-30T13:39:45.180 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-30T13:39:45.180 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-30T13:39:45.180 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-30T13:39:45.180 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-30T13:39:45.181 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-30T13:39:45.181 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-30T13:39:45.181 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-30T13:39:45.181 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-30T13:39:45.181 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-30T13:39:45.181 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-30T13:39:45.182 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-30T13:39:45.182 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-30T13:39:45.182 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-30T13:39:45.182 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-30T13:39:45.182 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-30T13:39:45.183 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-30T13:39:45.183 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-30T13:39:45.183 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-30T13:39:45.183 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-30T13:39:45.183 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-30T13:39:45.183 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-30T13:39:45.184 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-30T13:39:45.184 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-30T13:39:45.184 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-30T13:39:45.184 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-30T13:39:45.184 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-30T13:39:45.184 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-30T13:39:45.185 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-30T13:39:45.185 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-30T13:39:45.185 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-30T13:39:45.185 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-30T13:39:45.185 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-30T13:39:45.186 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-30T13:39:45.186 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-30T13:39:45.186 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-30T13:39:45.186 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-30T13:39:45.186 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-30T13:39:48.425 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-30T13:39:48.425 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-30T13:39:48.425 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-30T13:39:48.426 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-30T13:39:48.426 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-30T13:39:48.426 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-30T13:39:48.427 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-30T13:39:48.427 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-30T13:39:48.427 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-30T13:39:48.427 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-30T13:39:48.428 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-30T13:39:48.428 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-30T13:39:48.428 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-30T13:39:48.428 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-30T13:39:48.428 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-30T13:39:48.429 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-30T13:39:48.429 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-30T13:39:48.429 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-30T13:39:48.429 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-30T13:39:48.429 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-30T13:39:48.430 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-30T13:39:48.430 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-30T13:39:48.430 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-30T13:39:48.430 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-30T13:39:48.430 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-30T13:39:48.430 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-30T13:39:48.431 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-30T13:39:48.431 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-30T13:39:48.431 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-30T13:39:48.431 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-30T13:39:48.431 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-30T13:39:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-30T13:39:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-30T13:39:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-30T13:39:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-30T13:39:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-30T13:39:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-30T13:39:48.433 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-30T13:39:48.433 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-30T13:39:48.433 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-30T13:39:48.433 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-30T13:39:48.433 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-30T13:39:48.433 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-30T13:39:48.434 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-30T13:39:48.434 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-30T13:39:48.434 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-30T13:39:48.434 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-30T13:39:48.434 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-30T13:39:48.435 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-30T13:39:48.435 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-30T13:39:48.435 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-30T13:39:48.435 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-30T13:39:48.435 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-30T13:39:48.436 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-30T13:39:48.436 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-30T13:39:48.436 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-30T13:39:48.436 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-30T13:39:48.437 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-30T13:39:48.437 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-30T13:39:48.437 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-30T13:39:48.437 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-30T13:39:48.437 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-30T13:39:48.438 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-30T13:39:48.438 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-30T13:39:48.438 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-30T13:39:48.438 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-30T13:39:48.439 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-30T13:39:48.439 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-30T13:39:48.439 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-30T13:39:48.439 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-30T13:39:48.439 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-30T13:39:48.439 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-30T13:39:48.440 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-30T13:39:48.440 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-30T13:39:48.440 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-30T13:39:48.440 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-30T13:39:48.440 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-30T13:39:48.441 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-30T13:39:48.441 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-30T13:39:48.441 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-30T13:39:48.441 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-30T13:39:48.713 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-30T13:39:48.714 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-30T13:39:48.714 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-30T13:39:48.714 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-30T13:39:48.714 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-30T13:39:48.714 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-30T13:39:48.715 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-30T13:39:48.715 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-30T13:39:48.715 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-30T13:39:48.715 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-30T13:39:48.715 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-30T13:39:48.716 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-30T13:39:48.716 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-30T13:39:48.716 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-30T13:39:48.716 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-30T13:39:48.716 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-30T13:39:48.716 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-30T13:39:48.717 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-30T13:39:48.717 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-30T13:39:48.717 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-30T13:39:48.717 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-30T13:39:48.717 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-30T13:39:48.718 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-30T13:39:48.718 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-30T13:39:48.718 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-30T13:39:48.718 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-30T13:39:48.718 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-30T13:39:48.719 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-30T13:39:48.719 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-30T13:39:48.719 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-30T13:39:48.719 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-30T13:39:48.720 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-30T13:39:48.720 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-30T13:39:48.720 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-30T13:39:48.720 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-30T13:39:48.720 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-30T13:39:48.721 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-30T13:39:48.721 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-30T13:39:48.721 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-30T13:39:48.721 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-30T13:39:48.721 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-30T13:39:48.721 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-30T13:39:48.722 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-30T13:39:48.722 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-30T13:39:48.722 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-30T13:39:48.722 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-30T13:39:48.722 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-30T13:39:48.723 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-30T13:39:48.723 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-30T13:39:48.723 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-30T13:39:48.723 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-30T13:39:48.723 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-30T13:39:48.723 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-30T13:39:48.724 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-30T13:39:48.724 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-30T13:39:48.724 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-30T13:39:48.724 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-30T13:39:48.724 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-30T13:39:48.725 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-30T13:39:48.725 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-30T13:39:48.725 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-30T13:39:48.725 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-30T13:39:48.725 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-30T13:39:48.725 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-30T13:39:48.726 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-30T13:39:48.726 INFO:tasks.workunit.client.0.smithi031.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-30T13:39:48.726 INFO:tasks.workunit.client.0.smithi031.stdout:Take third snapshot .snap/test3 2023-10-30T13:39:48.736 INFO:tasks.workunit.client.0.smithi031.stdout:Delete the snapshots... 2023-10-30T13:39:51.748 INFO:tasks.workunit.client.0.smithi031.stdout:.//.snap/test1 2023-10-30T13:39:51.748 INFO:tasks.workunit.client.0.smithi031.stdout:.//.snap/test2 2023-10-30T13:39:51.748 INFO:tasks.workunit.client.0.smithi031.stdout:.//.snap/test3 2023-10-30T13:39:51.753 INFO:tasks.workunit.client.0.smithi031.stdout:./274/.snap/snap-subdir-test 2023-10-30T13:39:51.771 INFO:tasks.workunit.client.0.smithi031.stdout:./166/.snap/snap-subdir-test 2023-10-30T13:39:51.777 INFO:tasks.workunit.client.0.smithi031.stdout:./256/.snap/snap-subdir-test 2023-10-30T13:39:52.401 INFO:tasks.workunit.client.0.smithi031.stdout:./247/.snap/snap-subdir-test 2023-10-30T13:39:52.414 INFO:tasks.workunit.client.0.smithi031.stdout:./248/.snap/snap-subdir-test 2023-10-30T13:39:52.428 INFO:tasks.workunit.client.0.smithi031.stdout:./147/.snap/snap-subdir-test 2023-10-30T13:39:52.441 INFO:tasks.workunit.client.0.smithi031.stdout:./151/.snap/snap-subdir-test 2023-10-30T13:39:52.454 INFO:tasks.workunit.client.0.smithi031.stdout:./170/.snap/snap-subdir-test 2023-10-30T13:39:52.468 INFO:tasks.workunit.client.0.smithi031.stdout:./167/.snap/snap-subdir-test 2023-10-30T13:39:52.473 INFO:tasks.workunit.client.0.smithi031.stdout:./140/.snap/snap-subdir-test 2023-10-30T13:39:52.486 INFO:tasks.workunit.client.0.smithi031.stdout:./142/.snap/snap-subdir-test 2023-10-30T13:39:52.583 INFO:tasks.workunit.client.0.smithi031.stdout:./150/.snap/snap-subdir-test 2023-10-30T13:39:52.588 INFO:tasks.workunit.client.0.smithi031.stdout:./273/.snap/snap-subdir-test 2023-10-30T13:39:52.599 INFO:tasks.workunit.client.0.smithi031.stdout:./269/.snap/snap-subdir-test 2023-10-30T13:39:52.604 INFO:tasks.workunit.client.0.smithi031.stdout:./105/.snap/snap-subdir-test 2023-10-30T13:39:52.618 INFO:tasks.workunit.client.0.smithi031.stdout:./210/.snap/snap-subdir-test 2023-10-30T13:39:52.624 INFO:tasks.workunit.client.0.smithi031.stdout:./265/.snap/snap-subdir-test 2023-10-30T13:39:52.636 INFO:tasks.workunit.client.0.smithi031.stdout:./218/.snap/snap-subdir-test 2023-10-30T13:39:52.649 INFO:tasks.workunit.client.0.smithi031.stdout:./290/.snap/snap-subdir-test 2023-10-30T13:39:52.918 INFO:tasks.workunit.client.0.smithi031.stdout:./149/.snap/snap-subdir-test 2023-10-30T13:39:52.931 INFO:tasks.workunit.client.0.smithi031.stdout:./109/.snap/snap-subdir-test 2023-10-30T13:39:52.944 INFO:tasks.workunit.client.0.smithi031.stdout:./292/.snap/snap-subdir-test 2023-10-30T13:39:52.960 INFO:tasks.workunit.client.0.smithi031.stdout:./242/.snap/snap-subdir-test 2023-10-30T13:39:52.973 INFO:tasks.workunit.client.0.smithi031.stdout:./158/.snap/snap-subdir-test 2023-10-30T13:39:52.986 INFO:tasks.workunit.client.0.smithi031.stdout:./243/.snap/snap-subdir-test 2023-10-30T13:39:53.001 INFO:tasks.workunit.client.0.smithi031.stdout:./106/.snap/snap-subdir-test 2023-10-30T13:39:53.007 INFO:tasks.workunit.client.0.smithi031.stdout:./250/.snap/snap-subdir-test 2023-10-30T13:39:53.021 INFO:tasks.workunit.client.0.smithi031.stdout:./221/.snap/snap-subdir-test 2023-10-30T13:39:53.834 INFO:tasks.workunit.client.0.smithi031.stdout:./244/.snap/snap-subdir-test 2023-10-30T13:39:54.091 INFO:tasks.workunit.client.0.smithi031.stdout:./258/.snap/snap-subdir-test 2023-10-30T13:39:54.097 INFO:tasks.workunit.client.0.smithi031.stdout:./268/.snap/snap-subdir-test 2023-10-30T13:39:54.102 INFO:tasks.workunit.client.0.smithi031.stdout:./189/.snap/snap-subdir-test 2023-10-30T13:39:54.108 INFO:tasks.workunit.client.0.smithi031.stdout:./285/.snap/snap-subdir-test 2023-10-30T13:39:54.122 INFO:tasks.workunit.client.0.smithi031.stdout:./208/.snap/snap-subdir-test 2023-10-30T13:39:54.129 INFO:tasks.workunit.client.0.smithi031.stdout:./288/.snap/snap-subdir-test 2023-10-30T13:39:54.145 INFO:tasks.workunit.client.0.smithi031.stdout:./209/.snap/snap-subdir-test 2023-10-30T13:39:54.150 INFO:tasks.workunit.client.0.smithi031.stdout:./253/.snap/snap-subdir-test 2023-10-30T13:39:54.155 INFO:tasks.workunit.client.0.smithi031.stdout:./249/.snap/snap-subdir-test 2023-10-30T13:39:54.510 INFO:tasks.workunit.client.0.smithi031.stdout:./133/.snap/snap-subdir-test 2023-10-30T13:39:54.517 INFO:tasks.workunit.client.0.smithi031.stdout:./291/.snap/snap-subdir-test 2023-10-30T13:39:54.524 INFO:tasks.workunit.client.0.smithi031.stdout:./226/.snap/snap-subdir-test 2023-10-30T13:39:54.921 INFO:tasks.workunit.client.0.smithi031.stdout:./173/.snap/snap-subdir-test 2023-10-30T13:39:54.939 INFO:tasks.workunit.client.0.smithi031.stdout:./174/.snap/snap-subdir-test 2023-10-30T13:39:54.946 INFO:tasks.workunit.client.0.smithi031.stdout:./299/.snap/snap-subdir-test 2023-10-30T13:39:54.952 INFO:tasks.workunit.client.0.smithi031.stdout:./262/.snap/snap-subdir-test 2023-10-30T13:39:54.959 INFO:tasks.workunit.client.0.smithi031.stdout:./231/.snap/snap-subdir-test 2023-10-30T13:39:54.973 INFO:tasks.workunit.client.0.smithi031.stdout:./214/.snap/snap-subdir-test 2023-10-30T13:39:54.980 INFO:tasks.workunit.client.0.smithi031.stdout:./281/.snap/snap-subdir-test 2023-10-30T13:39:54.986 INFO:tasks.workunit.client.0.smithi031.stdout:./289/.snap/snap-subdir-test 2023-10-30T13:39:55.000 INFO:tasks.workunit.client.0.smithi031.stdout:./283/.snap/snap-subdir-test 2023-10-30T13:39:55.014 INFO:tasks.workunit.client.0.smithi031.stdout:./232/.snap/snap-subdir-test 2023-10-30T13:39:55.028 INFO:tasks.workunit.client.0.smithi031.stdout:./298/.snap/snap-subdir-test 2023-10-30T13:39:55.043 INFO:tasks.workunit.client.0.smithi031.stdout:./233/.snap/snap-subdir-test 2023-10-30T13:39:55.058 INFO:tasks.workunit.client.0.smithi031.stdout:./155/.snap/snap-subdir-test 2023-10-30T13:39:55.412 INFO:tasks.workunit.client.0.smithi031.stdout:./114/.snap/snap-subdir-test 2023-10-30T13:39:55.433 INFO:tasks.workunit.client.0.smithi031.stdout:./159/.snap/snap-subdir-test 2023-10-30T13:39:55.450 INFO:tasks.workunit.client.0.smithi031.stdout:./187/.snap/snap-subdir-test 2023-10-30T13:39:55.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:55.470 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:55.471 INFO:tasks.workunit.client.0.smithi031.stdout:./185/.snap/snap-subdir-test 2023-10-30T13:39:55.485 INFO:tasks.workunit.client.0.smithi031.stdout:./293/.snap/snap-subdir-test 2023-10-30T13:39:55.499 INFO:tasks.workunit.client.0.smithi031.stdout:./228/.snap/snap-subdir-test 2023-10-30T13:39:55.518 INFO:tasks.workunit.client.0.smithi031.stdout:./120/.snap/snap-subdir-test 2023-10-30T13:39:55.532 INFO:tasks.workunit.client.0.smithi031.stdout:./110/.snap/snap-subdir-test 2023-10-30T13:39:55.545 INFO:tasks.workunit.client.0.smithi031.stdout:./267/.snap/snap-subdir-test 2023-10-30T13:39:55.554 INFO:tasks.workunit.client.0.smithi031.stdout:./202/.snap/snap-subdir-test 2023-10-30T13:39:55.573 INFO:tasks.workunit.client.0.smithi031.stdout:./137/.snap/snap-subdir-test 2023-10-30T13:39:55.589 INFO:tasks.workunit.client.0.smithi031.stdout:./246/.snap/snap-subdir-test 2023-10-30T13:39:55.604 INFO:tasks.workunit.client.0.smithi031.stdout:./219/.snap/snap-subdir-test 2023-10-30T13:39:55.871 INFO:tasks.workunit.client.0.smithi031.stdout:./198/.snap/snap-subdir-test 2023-10-30T13:39:55.891 INFO:tasks.workunit.client.0.smithi031.stdout:./136/.snap/snap-subdir-test 2023-10-30T13:39:55.905 INFO:tasks.workunit.client.0.smithi031.stdout:./270/.snap/snap-subdir-test 2023-10-30T13:39:55.923 INFO:tasks.workunit.client.0.smithi031.stdout:./125/.snap/snap-subdir-test 2023-10-30T13:39:55.941 INFO:tasks.workunit.client.0.smithi031.stdout:./194/.snap/snap-subdir-test 2023-10-30T13:39:55.958 INFO:tasks.workunit.client.0.smithi031.stdout:./119/.snap/snap-subdir-test 2023-10-30T13:39:55.973 INFO:tasks.workunit.client.0.smithi031.stdout:./297/.snap/snap-subdir-test 2023-10-30T13:39:55.988 INFO:tasks.workunit.client.0.smithi031.stdout:./230/.snap/snap-subdir-test 2023-10-30T13:39:56.002 INFO:tasks.workunit.client.0.smithi031.stdout:./260/.snap/snap-subdir-test 2023-10-30T13:39:56.018 INFO:tasks.workunit.client.0.smithi031.stdout:./157/.snap/snap-subdir-test 2023-10-30T13:39:56.033 INFO:tasks.workunit.client.0.smithi031.stdout:./172/.snap/snap-subdir-test 2023-10-30T13:39:56.047 INFO:tasks.workunit.client.0.smithi031.stdout:./277/.snap/snap-subdir-test 2023-10-30T13:39:56.062 INFO:tasks.workunit.client.0.smithi031.stdout:./186/.snap/snap-subdir-test 2023-10-30T13:39:56.075 INFO:tasks.workunit.client.0.smithi031.stdout:./237/.snap/snap-subdir-test 2023-10-30T13:39:56.085 INFO:tasks.workunit.client.0.smithi031.stdout:./190/.snap/snap-subdir-test 2023-10-30T13:39:56.100 INFO:tasks.workunit.client.0.smithi031.stdout:./213/.snap/snap-subdir-test 2023-10-30T13:39:56.118 INFO:tasks.workunit.client.0.smithi031.stdout:./104/.snap/snap-subdir-test 2023-10-30T13:39:56.132 INFO:tasks.workunit.client.0.smithi031.stdout:./164/.snap/snap-subdir-test 2023-10-30T13:39:56.655 INFO:tasks.workunit.client.0.smithi031.stdout:./112/.snap/snap-subdir-test 2023-10-30T13:39:56.670 INFO:tasks.workunit.client.0.smithi031.stdout:./295/.snap/snap-subdir-test 2023-10-30T13:39:56.684 INFO:tasks.workunit.client.0.smithi031.stdout:./272/.snap/snap-subdir-test 2023-10-30T13:39:56.702 INFO:tasks.workunit.client.0.smithi031.stdout:./100/.snap/snap-subdir-test 2023-10-30T13:39:56.716 INFO:tasks.workunit.client.0.smithi031.stdout:./134/.snap/snap-subdir-test 2023-10-30T13:39:56.730 INFO:tasks.workunit.client.0.smithi031.stdout:./200/.snap/snap-subdir-test 2023-10-30T13:39:56.747 INFO:tasks.workunit.client.0.smithi031.stdout:./115/.snap/snap-subdir-test 2023-10-30T13:39:56.763 INFO:tasks.workunit.client.0.smithi031.stdout:./284/.snap/snap-subdir-test 2023-10-30T13:39:56.780 INFO:tasks.workunit.client.0.smithi031.stdout:./280/.snap/snap-subdir-test 2023-10-30T13:39:56.800 INFO:tasks.workunit.client.0.smithi031.stdout:./116/.snap/snap-subdir-test 2023-10-30T13:39:56.816 INFO:tasks.workunit.client.0.smithi031.stdout:./245/.snap/snap-subdir-test 2023-10-30T13:39:56.830 INFO:tasks.workunit.client.0.smithi031.stdout:./195/.snap/snap-subdir-test 2023-10-30T13:39:56.846 INFO:tasks.workunit.client.0.smithi031.stdout:./138/.snap/snap-subdir-test 2023-10-30T13:39:56.863 INFO:tasks.workunit.client.0.smithi031.stdout:./135/.snap/snap-subdir-test 2023-10-30T13:39:56.880 INFO:tasks.workunit.client.0.smithi031.stdout:./117/.snap/snap-subdir-test 2023-10-30T13:39:56.895 INFO:tasks.workunit.client.0.smithi031.stdout:./241/.snap/snap-subdir-test 2023-10-30T13:39:56.909 INFO:tasks.workunit.client.0.smithi031.stdout:./143/.snap/snap-subdir-test 2023-10-30T13:39:57.177 INFO:tasks.workunit.client.0.smithi031.stdout:./180/.snap/snap-subdir-test 2023-10-30T13:39:57.833 INFO:tasks.workunit.client.0.smithi031.stdout:./126/.snap/snap-subdir-test 2023-10-30T13:39:57.849 INFO:tasks.workunit.client.0.smithi031.stdout:./161/.snap/snap-subdir-test 2023-10-30T13:39:57.864 INFO:tasks.workunit.client.0.smithi031.stdout:./204/.snap/snap-subdir-test 2023-10-30T13:39:57.879 INFO:tasks.workunit.client.0.smithi031.stdout:./176/.snap/snap-subdir-test 2023-10-30T13:39:57.893 INFO:tasks.workunit.client.0.smithi031.stdout:./255/.snap/snap-subdir-test 2023-10-30T13:39:57.907 INFO:tasks.workunit.client.0.smithi031.stdout:./215/.snap/snap-subdir-test 2023-10-30T13:39:57.922 INFO:tasks.workunit.client.0.smithi031.stdout:./111/.snap/snap-subdir-test 2023-10-30T13:39:57.939 INFO:tasks.workunit.client.0.smithi031.stdout:./238/.snap/snap-subdir-test 2023-10-30T13:39:57.955 INFO:tasks.workunit.client.0.smithi031.stdout:./191/.snap/snap-subdir-test 2023-10-30T13:39:57.971 INFO:tasks.workunit.client.0.smithi031.stdout:./184/.snap/snap-subdir-test 2023-10-30T13:39:57.988 INFO:tasks.workunit.client.0.smithi031.stdout:./192/.snap/snap-subdir-test 2023-10-30T13:39:57.994 INFO:tasks.workunit.client.0.smithi031.stdout:./103/.snap/snap-subdir-test 2023-10-30T13:39:58.011 INFO:tasks.workunit.client.0.smithi031.stdout:./128/.snap/snap-subdir-test 2023-10-30T13:39:58.029 INFO:tasks.workunit.client.0.smithi031.stdout:./229/.snap/snap-subdir-test 2023-10-30T13:39:58.046 INFO:tasks.workunit.client.0.smithi031.stdout:./144/.snap/snap-subdir-test 2023-10-30T13:39:58.062 INFO:tasks.workunit.client.0.smithi031.stdout:./146/.snap/snap-subdir-test 2023-10-30T13:39:58.078 INFO:tasks.workunit.client.0.smithi031.stdout:./227/.snap/snap-subdir-test 2023-10-30T13:39:58.093 INFO:tasks.workunit.client.0.smithi031.stdout:./278/.snap/snap-subdir-test 2023-10-30T13:39:58.109 INFO:tasks.workunit.client.0.smithi031.stdout:./141/.snap/snap-subdir-test 2023-10-30T13:39:58.126 INFO:tasks.workunit.client.0.smithi031.stdout:./234/.snap/snap-subdir-test 2023-10-30T13:39:58.142 INFO:tasks.workunit.client.0.smithi031.stdout:./225/.snap/snap-subdir-test 2023-10-30T13:39:59.123 INFO:tasks.workunit.client.0.smithi031.stdout:./182/.snap/snap-subdir-test 2023-10-30T13:39:59.134 INFO:tasks.workunit.client.0.smithi031.stdout:./107/.snap/snap-subdir-test 2023-10-30T13:39:59.147 INFO:tasks.workunit.client.0.smithi031.stdout:./178/.snap/snap-subdir-test 2023-10-30T13:39:59.159 INFO:tasks.workunit.client.0.smithi031.stdout:./266/.snap/snap-subdir-test 2023-10-30T13:39:59.170 INFO:tasks.workunit.client.0.smithi031.stdout:./240/.snap/snap-subdir-test 2023-10-30T13:39:59.183 INFO:tasks.workunit.client.0.smithi031.stdout:./156/.snap/snap-subdir-test 2023-10-30T13:39:59.196 INFO:tasks.workunit.client.0.smithi031.stdout:./131/.snap/snap-subdir-test 2023-10-30T13:39:59.210 INFO:tasks.workunit.client.0.smithi031.stdout:./123/.snap/snap-subdir-test 2023-10-30T13:39:59.225 INFO:tasks.workunit.client.0.smithi031.stdout:./160/.snap/snap-subdir-test 2023-10-30T13:39:59.870 INFO:tasks.workunit.client.0.smithi031.stdout:./165/.snap/snap-subdir-test 2023-10-30T13:40:00.138 INFO:tasks.workunit.client.0.smithi031.stdout:./101/.snap/snap-subdir-test 2023-10-30T13:40:00.151 INFO:tasks.workunit.client.0.smithi031.stdout:./148/.snap/snap-subdir-test 2023-10-30T13:40:00.164 INFO:tasks.workunit.client.0.smithi031.stdout:./168/.snap/snap-subdir-test 2023-10-30T13:40:01.068 INFO:tasks.workunit.client.0.smithi031.stdout:./179/.snap/snap-subdir-test 2023-10-30T13:40:01.083 INFO:tasks.workunit.client.0.smithi031.stdout:./257/.snap/snap-subdir-test 2023-10-30T13:40:02.070 INFO:tasks.workunit.client.0.smithi031.stdout:./252/.snap/snap-subdir-test 2023-10-30T13:40:02.086 INFO:tasks.workunit.client.0.smithi031.stdout:./132/.snap/snap-subdir-test 2023-10-30T13:40:02.102 INFO:tasks.workunit.client.0.smithi031.stdout:./236/.snap/snap-subdir-test 2023-10-30T13:40:05.954 INFO:tasks.workunit.client.0.smithi031.stdout:./211/.snap/snap-subdir-test 2023-10-30T13:40:05.970 INFO:tasks.workunit.client.0.smithi031.stdout:./199/.snap/snap-subdir-test 2023-10-30T13:40:05.989 INFO:tasks.workunit.client.0.smithi031.stdout:./205/.snap/snap-subdir-test 2023-10-30T13:40:06.807 INFO:tasks.workunit.client.0.smithi031.stdout:./163/.snap/snap-subdir-test 2023-10-30T13:40:07.076 INFO:tasks.workunit.client.0.smithi031.stdout:./261/.snap/snap-subdir-test 2023-10-30T13:40:07.095 INFO:tasks.workunit.client.0.smithi031.stdout:./201/.snap/snap-subdir-test 2023-10-30T13:40:07.110 INFO:tasks.workunit.client.0.smithi031.stdout:./282/.snap/snap-subdir-test 2023-10-30T13:40:07.125 INFO:tasks.workunit.client.0.smithi031.stdout:./296/.snap/snap-subdir-test 2023-10-30T13:40:07.140 INFO:tasks.workunit.client.0.smithi031.stdout:./217/.snap/snap-subdir-test 2023-10-30T13:40:07.154 INFO:tasks.workunit.client.0.smithi031.stdout:./153/.snap/snap-subdir-test 2023-10-30T13:40:07.170 INFO:tasks.workunit.client.0.smithi031.stdout:./206/.snap/snap-subdir-test 2023-10-30T13:40:07.188 INFO:tasks.workunit.client.0.smithi031.stdout:./122/.snap/snap-subdir-test 2023-10-30T13:40:07.201 INFO:tasks.workunit.client.0.smithi031.stdout:./162/.snap/snap-subdir-test 2023-10-30T13:40:07.217 INFO:tasks.workunit.client.0.smithi031.stdout:./197/.snap/snap-subdir-test 2023-10-30T13:40:07.233 INFO:tasks.workunit.client.0.smithi031.stdout:./216/.snap/snap-subdir-test 2023-10-30T13:40:10.133 INFO:tasks.workunit.client.0.smithi031.stdout:./254/.snap/snap-subdir-test 2023-10-30T13:40:10.140 INFO:tasks.workunit.client.0.smithi031.stdout:./113/.snap/snap-subdir-test 2023-10-30T13:40:10.156 INFO:tasks.workunit.client.0.smithi031.stdout:./152/.snap/snap-subdir-test 2023-10-30T13:40:10.177 INFO:tasks.workunit.client.0.smithi031.stdout:./121/.snap/snap-subdir-test 2023-10-30T13:40:10.192 INFO:tasks.workunit.client.0.smithi031.stdout:./118/.snap/snap-subdir-test 2023-10-30T13:40:10.207 INFO:tasks.workunit.client.0.smithi031.stdout:./196/.snap/snap-subdir-test 2023-10-30T13:40:11.219 INFO:tasks.workunit.client.0.smithi031.stdout:./124/.snap/snap-subdir-test 2023-10-30T13:40:11.776 INFO:tasks.workunit.client.0.smithi031.stdout:./239/.snap/snap-subdir-test 2023-10-30T13:40:11.986 INFO:tasks.workunit.client.0.smithi031.stdout:./188/.snap/snap-subdir-test 2023-10-30T13:40:12.491 INFO:tasks.workunit.client.0.smithi031.stdout:./287/.snap/snap-subdir-test 2023-10-30T13:40:12.984 INFO:tasks.workunit.client.0.smithi031.stdout:./275/.snap/snap-subdir-test 2023-10-30T13:40:13.001 INFO:tasks.workunit.client.0.smithi031.stdout:./259/.snap/snap-subdir-test 2023-10-30T13:40:13.021 INFO:tasks.workunit.client.0.smithi031.stdout:./224/.snap/snap-subdir-test 2023-10-30T13:40:13.035 INFO:tasks.workunit.client.0.smithi031.stdout:./264/.snap/snap-subdir-test 2023-10-30T13:40:13.052 INFO:tasks.workunit.client.0.smithi031.stdout:./145/.snap/snap-subdir-test 2023-10-30T13:40:13.067 INFO:tasks.workunit.client.0.smithi031.stdout:./279/.snap/snap-subdir-test 2023-10-30T13:40:13.085 INFO:tasks.workunit.client.0.smithi031.stdout:./263/.snap/snap-subdir-test 2023-10-30T13:40:13.102 INFO:tasks.workunit.client.0.smithi031.stdout:./130/.snap/snap-subdir-test 2023-10-30T13:40:13.116 INFO:tasks.workunit.client.0.smithi031.stdout:./193/.snap/snap-subdir-test 2023-10-30T13:40:13.128 INFO:tasks.workunit.client.0.smithi031.stdout:./223/.snap/snap-subdir-test 2023-10-30T13:40:13.141 INFO:tasks.workunit.client.0.smithi031.stdout:./207/.snap/snap-subdir-test 2023-10-30T13:40:13.155 INFO:tasks.workunit.client.0.smithi031.stdout:./276/.snap/snap-subdir-test 2023-10-30T13:40:13.170 INFO:tasks.workunit.client.0.smithi031.stdout:./271/.snap/snap-subdir-test 2023-10-30T13:40:13.183 INFO:tasks.workunit.client.0.smithi031.stdout:./251/.snap/snap-subdir-test 2023-10-30T13:40:13.199 INFO:tasks.workunit.client.0.smithi031.stdout:./175/.snap/snap-subdir-test 2023-10-30T13:40:13.214 INFO:tasks.workunit.client.0.smithi031.stdout:./222/.snap/snap-subdir-test 2023-10-30T13:40:13.228 INFO:tasks.workunit.client.0.smithi031.stdout:./220/.snap/snap-subdir-test 2023-10-30T13:40:13.244 INFO:tasks.workunit.client.0.smithi031.stdout:./203/.snap/snap-subdir-test 2023-10-30T13:40:13.257 INFO:tasks.workunit.client.0.smithi031.stdout:./181/.snap/snap-subdir-test 2023-10-30T13:40:13.271 INFO:tasks.workunit.client.0.smithi031.stdout:./286/.snap/snap-subdir-test 2023-10-30T13:40:13.286 INFO:tasks.workunit.client.0.smithi031.stdout:./127/.snap/snap-subdir-test 2023-10-30T13:40:13.301 INFO:tasks.workunit.client.0.smithi031.stdout:./108/.snap/snap-subdir-test 2023-10-30T13:40:13.314 INFO:tasks.workunit.client.0.smithi031.stdout:./183/.snap/snap-subdir-test 2023-10-30T13:40:13.328 INFO:tasks.workunit.client.0.smithi031.stdout:./294/.snap/snap-subdir-test 2023-10-30T13:40:13.343 INFO:tasks.workunit.client.0.smithi031.stdout:./235/.snap/snap-subdir-test 2023-10-30T13:40:13.359 INFO:tasks.workunit.client.0.smithi031.stdout:./177/.snap/snap-subdir-test 2023-10-30T13:40:13.373 INFO:tasks.workunit.client.0.smithi031.stdout:./169/.snap/snap-subdir-test 2023-10-30T13:40:13.387 INFO:tasks.workunit.client.0.smithi031.stdout:./212/.snap/snap-subdir-test 2023-10-30T13:40:13.401 INFO:tasks.workunit.client.0.smithi031.stdout:./154/.snap/snap-subdir-test 2023-10-30T13:40:13.417 INFO:tasks.workunit.client.0.smithi031.stdout:./171/.snap/snap-subdir-test 2023-10-30T13:40:13.433 INFO:tasks.workunit.client.0.smithi031.stdout:./139/.snap/snap-subdir-test 2023-10-30T13:40:13.448 INFO:tasks.workunit.client.0.smithi031.stdout:./102/.snap/snap-subdir-test 2023-10-30T13:40:13.597 INFO:tasks.workunit.client.0.smithi031.stdout:./129/.snap/snap-subdir-test 2023-10-30T13:40:25.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:25.542 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:26.566 INFO:tasks.workunit.client.0.smithi031.stdout:Delete all the files and directories ... 2023-10-30T13:40:29.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/19' 2023-10-30T13:40:29.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/15' 2023-10-30T13:40:29.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/31' 2023-10-30T13:40:29.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/36' 2023-10-30T13:40:29.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/20' 2023-10-30T13:40:29.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/12' 2023-10-30T13:40:29.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/39' 2023-10-30T13:40:29.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/10' 2023-10-30T13:40:29.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/34' 2023-10-30T13:40:29.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/38' 2023-10-30T13:40:29.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/11' 2023-10-30T13:40:29.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/16' 2023-10-30T13:40:29.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/32' 2023-10-30T13:40:29.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/30' 2023-10-30T13:40:29.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/14' 2023-10-30T13:40:29.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/18' 2023-10-30T13:40:29.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/13' 2023-10-30T13:40:29.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/33' 2023-10-30T13:40:29.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/17' 2023-10-30T13:40:29.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/35' 2023-10-30T13:40:29.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './100/37' 2023-10-30T13:40:29.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './100' 2023-10-30T13:40:29.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/19' 2023-10-30T13:40:29.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/15' 2023-10-30T13:40:29.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/31' 2023-10-30T13:40:29.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/36' 2023-10-30T13:40:29.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/20' 2023-10-30T13:40:29.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/12' 2023-10-30T13:40:29.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/39' 2023-10-30T13:40:29.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/10' 2023-10-30T13:40:29.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/34' 2023-10-30T13:40:29.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/38' 2023-10-30T13:40:29.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/11' 2023-10-30T13:40:29.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/16' 2023-10-30T13:40:29.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/32' 2023-10-30T13:40:29.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/30' 2023-10-30T13:40:29.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/14' 2023-10-30T13:40:29.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/18' 2023-10-30T13:40:29.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/13' 2023-10-30T13:40:29.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/33' 2023-10-30T13:40:29.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/17' 2023-10-30T13:40:29.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/35' 2023-10-30T13:40:29.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './101/37' 2023-10-30T13:40:29.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './101' 2023-10-30T13:40:29.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/19' 2023-10-30T13:40:29.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/15' 2023-10-30T13:40:29.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/31' 2023-10-30T13:40:29.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/36' 2023-10-30T13:40:29.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/20' 2023-10-30T13:40:29.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/12' 2023-10-30T13:40:29.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/39' 2023-10-30T13:40:29.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/10' 2023-10-30T13:40:29.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/34' 2023-10-30T13:40:29.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/38' 2023-10-30T13:40:29.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/11' 2023-10-30T13:40:29.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/16' 2023-10-30T13:40:29.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/32' 2023-10-30T13:40:29.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/30' 2023-10-30T13:40:29.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/14' 2023-10-30T13:40:29.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/18' 2023-10-30T13:40:29.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/13' 2023-10-30T13:40:29.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/33' 2023-10-30T13:40:29.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/17' 2023-10-30T13:40:29.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/35' 2023-10-30T13:40:29.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './102/37' 2023-10-30T13:40:29.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './102' 2023-10-30T13:40:29.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/19' 2023-10-30T13:40:29.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/15' 2023-10-30T13:40:29.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/31' 2023-10-30T13:40:29.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/36' 2023-10-30T13:40:29.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/20' 2023-10-30T13:40:29.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/12' 2023-10-30T13:40:29.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/39' 2023-10-30T13:40:29.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/10' 2023-10-30T13:40:29.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/34' 2023-10-30T13:40:29.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/38' 2023-10-30T13:40:29.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/11' 2023-10-30T13:40:29.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/16' 2023-10-30T13:40:29.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/32' 2023-10-30T13:40:29.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/30' 2023-10-30T13:40:29.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/14' 2023-10-30T13:40:29.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/18' 2023-10-30T13:40:29.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/13' 2023-10-30T13:40:29.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/33' 2023-10-30T13:40:29.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/17' 2023-10-30T13:40:29.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/35' 2023-10-30T13:40:29.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './103/37' 2023-10-30T13:40:29.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './103' 2023-10-30T13:40:29.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/19' 2023-10-30T13:40:29.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/15' 2023-10-30T13:40:29.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/31' 2023-10-30T13:40:29.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/36' 2023-10-30T13:40:29.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/20' 2023-10-30T13:40:29.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/12' 2023-10-30T13:40:29.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/39' 2023-10-30T13:40:29.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/10' 2023-10-30T13:40:29.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/34' 2023-10-30T13:40:29.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/38' 2023-10-30T13:40:29.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/11' 2023-10-30T13:40:29.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/16' 2023-10-30T13:40:29.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/32' 2023-10-30T13:40:29.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/30' 2023-10-30T13:40:29.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/14' 2023-10-30T13:40:29.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/18' 2023-10-30T13:40:29.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/13' 2023-10-30T13:40:29.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/33' 2023-10-30T13:40:29.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/17' 2023-10-30T13:40:29.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/35' 2023-10-30T13:40:29.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './104/37' 2023-10-30T13:40:29.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './104' 2023-10-30T13:40:29.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/19' 2023-10-30T13:40:29.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/15' 2023-10-30T13:40:29.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/31' 2023-10-30T13:40:29.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/36' 2023-10-30T13:40:29.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/20' 2023-10-30T13:40:29.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/12' 2023-10-30T13:40:29.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/39' 2023-10-30T13:40:29.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/10' 2023-10-30T13:40:29.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/34' 2023-10-30T13:40:29.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/38' 2023-10-30T13:40:29.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/11' 2023-10-30T13:40:29.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/16' 2023-10-30T13:40:29.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/32' 2023-10-30T13:40:29.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/30' 2023-10-30T13:40:29.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/14' 2023-10-30T13:40:29.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/18' 2023-10-30T13:40:29.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/13' 2023-10-30T13:40:29.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/33' 2023-10-30T13:40:29.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/17' 2023-10-30T13:40:29.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/35' 2023-10-30T13:40:29.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './105/37' 2023-10-30T13:40:29.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './105' 2023-10-30T13:40:29.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/19' 2023-10-30T13:40:29.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/15' 2023-10-30T13:40:29.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/31' 2023-10-30T13:40:29.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/36' 2023-10-30T13:40:29.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/20' 2023-10-30T13:40:29.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/12' 2023-10-30T13:40:29.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/39' 2023-10-30T13:40:29.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/10' 2023-10-30T13:40:29.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/34' 2023-10-30T13:40:29.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/38' 2023-10-30T13:40:29.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/11' 2023-10-30T13:40:29.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/16' 2023-10-30T13:40:29.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/32' 2023-10-30T13:40:29.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/30' 2023-10-30T13:40:29.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/14' 2023-10-30T13:40:29.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/18' 2023-10-30T13:40:29.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/13' 2023-10-30T13:40:29.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/33' 2023-10-30T13:40:29.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/17' 2023-10-30T13:40:29.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/35' 2023-10-30T13:40:29.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './106/37' 2023-10-30T13:40:29.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './106' 2023-10-30T13:40:29.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/19' 2023-10-30T13:40:29.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/15' 2023-10-30T13:40:29.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/31' 2023-10-30T13:40:29.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/36' 2023-10-30T13:40:29.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/20' 2023-10-30T13:40:29.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/12' 2023-10-30T13:40:29.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/39' 2023-10-30T13:40:29.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/10' 2023-10-30T13:40:29.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/34' 2023-10-30T13:40:29.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/38' 2023-10-30T13:40:29.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/11' 2023-10-30T13:40:29.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/16' 2023-10-30T13:40:29.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/32' 2023-10-30T13:40:29.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/30' 2023-10-30T13:40:29.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/14' 2023-10-30T13:40:29.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/18' 2023-10-30T13:40:29.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/13' 2023-10-30T13:40:29.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/33' 2023-10-30T13:40:29.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/17' 2023-10-30T13:40:29.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/35' 2023-10-30T13:40:29.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './107/37' 2023-10-30T13:40:29.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './107' 2023-10-30T13:40:29.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/19' 2023-10-30T13:40:29.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/15' 2023-10-30T13:40:29.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/31' 2023-10-30T13:40:29.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/36' 2023-10-30T13:40:29.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/20' 2023-10-30T13:40:29.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/12' 2023-10-30T13:40:29.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/39' 2023-10-30T13:40:29.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/10' 2023-10-30T13:40:29.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/34' 2023-10-30T13:40:29.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/38' 2023-10-30T13:40:29.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/11' 2023-10-30T13:40:29.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/16' 2023-10-30T13:40:29.109 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/32' 2023-10-30T13:40:29.109 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/30' 2023-10-30T13:40:29.109 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/14' 2023-10-30T13:40:29.109 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/18' 2023-10-30T13:40:29.109 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/13' 2023-10-30T13:40:29.109 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/33' 2023-10-30T13:40:29.110 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/17' 2023-10-30T13:40:29.110 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/35' 2023-10-30T13:40:29.110 INFO:tasks.workunit.client.0.smithi031.stdout:removed './108/37' 2023-10-30T13:40:29.110 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './108' 2023-10-30T13:40:29.110 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/19' 2023-10-30T13:40:29.111 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/15' 2023-10-30T13:40:29.111 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/31' 2023-10-30T13:40:29.111 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/36' 2023-10-30T13:40:29.111 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/20' 2023-10-30T13:40:29.111 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/12' 2023-10-30T13:40:29.111 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/39' 2023-10-30T13:40:29.112 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/10' 2023-10-30T13:40:29.112 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/34' 2023-10-30T13:40:29.112 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/38' 2023-10-30T13:40:29.112 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/11' 2023-10-30T13:40:29.112 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/16' 2023-10-30T13:40:29.112 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/32' 2023-10-30T13:40:29.113 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/30' 2023-10-30T13:40:32.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/14' 2023-10-30T13:40:32.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/18' 2023-10-30T13:40:32.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/13' 2023-10-30T13:40:32.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/33' 2023-10-30T13:40:32.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/17' 2023-10-30T13:40:32.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/35' 2023-10-30T13:40:32.675 INFO:tasks.workunit.client.0.smithi031.stdout:removed './109/37' 2023-10-30T13:40:32.675 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './109' 2023-10-30T13:40:32.675 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/19' 2023-10-30T13:40:32.675 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/15' 2023-10-30T13:40:32.675 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/31' 2023-10-30T13:40:32.676 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/36' 2023-10-30T13:40:32.676 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/20' 2023-10-30T13:40:32.676 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/12' 2023-10-30T13:40:32.676 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/39' 2023-10-30T13:40:32.676 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/10' 2023-10-30T13:40:32.676 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/34' 2023-10-30T13:40:32.677 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/38' 2023-10-30T13:40:32.677 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/11' 2023-10-30T13:40:32.677 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/16' 2023-10-30T13:40:32.677 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/32' 2023-10-30T13:40:32.677 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/30' 2023-10-30T13:40:32.677 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/14' 2023-10-30T13:40:32.678 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/18' 2023-10-30T13:40:32.678 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/13' 2023-10-30T13:40:32.678 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/33' 2023-10-30T13:40:32.678 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/17' 2023-10-30T13:40:32.678 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/35' 2023-10-30T13:40:32.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './110/37' 2023-10-30T13:40:32.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './110' 2023-10-30T13:40:32.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/19' 2023-10-30T13:40:32.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/15' 2023-10-30T13:40:32.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/31' 2023-10-30T13:40:32.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/36' 2023-10-30T13:40:32.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/20' 2023-10-30T13:40:32.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/12' 2023-10-30T13:40:32.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/39' 2023-10-30T13:40:32.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/10' 2023-10-30T13:40:32.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/34' 2023-10-30T13:40:32.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/38' 2023-10-30T13:40:32.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/11' 2023-10-30T13:40:32.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/16' 2023-10-30T13:40:32.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/32' 2023-10-30T13:40:32.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/30' 2023-10-30T13:40:32.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/14' 2023-10-30T13:40:32.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/18' 2023-10-30T13:40:32.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/13' 2023-10-30T13:40:32.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/33' 2023-10-30T13:40:32.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/17' 2023-10-30T13:40:32.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/35' 2023-10-30T13:40:32.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './111/37' 2023-10-30T13:40:32.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './111' 2023-10-30T13:40:32.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/19' 2023-10-30T13:40:32.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/15' 2023-10-30T13:40:32.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/31' 2023-10-30T13:40:32.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/36' 2023-10-30T13:40:32.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/20' 2023-10-30T13:40:32.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/12' 2023-10-30T13:40:32.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/39' 2023-10-30T13:40:32.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/10' 2023-10-30T13:40:32.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/34' 2023-10-30T13:40:32.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/38' 2023-10-30T13:40:32.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/11' 2023-10-30T13:40:32.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/16' 2023-10-30T13:40:32.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/32' 2023-10-30T13:40:32.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/30' 2023-10-30T13:40:32.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/14' 2023-10-30T13:40:32.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/18' 2023-10-30T13:40:32.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/13' 2023-10-30T13:40:32.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/33' 2023-10-30T13:40:32.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/17' 2023-10-30T13:40:32.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/35' 2023-10-30T13:40:32.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './112/37' 2023-10-30T13:40:32.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './112' 2023-10-30T13:40:32.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/19' 2023-10-30T13:40:32.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/15' 2023-10-30T13:40:32.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/31' 2023-10-30T13:40:32.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/36' 2023-10-30T13:40:32.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/20' 2023-10-30T13:40:32.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/12' 2023-10-30T13:40:32.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/39' 2023-10-30T13:40:32.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/10' 2023-10-30T13:40:32.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/34' 2023-10-30T13:40:32.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/38' 2023-10-30T13:40:32.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/11' 2023-10-30T13:40:32.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/16' 2023-10-30T13:40:32.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/32' 2023-10-30T13:40:32.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/30' 2023-10-30T13:40:32.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/14' 2023-10-30T13:40:32.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/18' 2023-10-30T13:40:32.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/13' 2023-10-30T13:40:32.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/33' 2023-10-30T13:40:32.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/17' 2023-10-30T13:40:32.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/35' 2023-10-30T13:40:32.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './113/37' 2023-10-30T13:40:32.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './113' 2023-10-30T13:40:32.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/19' 2023-10-30T13:40:32.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/15' 2023-10-30T13:40:32.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/31' 2023-10-30T13:40:32.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/36' 2023-10-30T13:40:32.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/20' 2023-10-30T13:40:32.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/12' 2023-10-30T13:40:32.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/39' 2023-10-30T13:40:32.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/10' 2023-10-30T13:40:32.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/34' 2023-10-30T13:40:32.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/38' 2023-10-30T13:40:32.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/11' 2023-10-30T13:40:32.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/16' 2023-10-30T13:40:32.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/32' 2023-10-30T13:40:32.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/30' 2023-10-30T13:40:32.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/14' 2023-10-30T13:40:32.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/18' 2023-10-30T13:40:32.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/13' 2023-10-30T13:40:32.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/33' 2023-10-30T13:40:32.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/17' 2023-10-30T13:40:32.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/35' 2023-10-30T13:40:32.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './114/37' 2023-10-30T13:40:32.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './114' 2023-10-30T13:40:32.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/19' 2023-10-30T13:40:32.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/15' 2023-10-30T13:40:32.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/31' 2023-10-30T13:40:32.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/36' 2023-10-30T13:40:32.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/20' 2023-10-30T13:40:32.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/12' 2023-10-30T13:40:32.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/39' 2023-10-30T13:40:32.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/10' 2023-10-30T13:40:32.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/34' 2023-10-30T13:40:32.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/38' 2023-10-30T13:40:32.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/11' 2023-10-30T13:40:32.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/16' 2023-10-30T13:40:32.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/32' 2023-10-30T13:40:32.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/30' 2023-10-30T13:40:32.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/14' 2023-10-30T13:40:32.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/18' 2023-10-30T13:40:32.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/13' 2023-10-30T13:40:32.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/33' 2023-10-30T13:40:32.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/17' 2023-10-30T13:40:32.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/35' 2023-10-30T13:40:32.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './115/37' 2023-10-30T13:40:32.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './115' 2023-10-30T13:40:32.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/19' 2023-10-30T13:40:32.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/15' 2023-10-30T13:40:32.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/31' 2023-10-30T13:40:32.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/36' 2023-10-30T13:40:32.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/20' 2023-10-30T13:40:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/12' 2023-10-30T13:40:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/39' 2023-10-30T13:40:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/10' 2023-10-30T13:40:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/34' 2023-10-30T13:40:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/38' 2023-10-30T13:40:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/11' 2023-10-30T13:40:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/16' 2023-10-30T13:40:32.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/32' 2023-10-30T13:40:32.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/30' 2023-10-30T13:40:32.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/14' 2023-10-30T13:40:32.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/18' 2023-10-30T13:40:32.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/13' 2023-10-30T13:40:32.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/33' 2023-10-30T13:40:32.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/17' 2023-10-30T13:40:32.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/35' 2023-10-30T13:40:32.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './116/37' 2023-10-30T13:40:32.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './116' 2023-10-30T13:40:32.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/19' 2023-10-30T13:40:32.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/15' 2023-10-30T13:40:32.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/31' 2023-10-30T13:40:32.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/36' 2023-10-30T13:40:32.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/20' 2023-10-30T13:40:32.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/12' 2023-10-30T13:40:32.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/39' 2023-10-30T13:40:32.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/10' 2023-10-30T13:40:32.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/34' 2023-10-30T13:40:32.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/38' 2023-10-30T13:40:32.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/11' 2023-10-30T13:40:32.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/16' 2023-10-30T13:40:32.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/32' 2023-10-30T13:40:32.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/30' 2023-10-30T13:40:32.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/14' 2023-10-30T13:40:32.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/18' 2023-10-30T13:40:32.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/13' 2023-10-30T13:40:32.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/33' 2023-10-30T13:40:32.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/17' 2023-10-30T13:40:32.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/35' 2023-10-30T13:40:32.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './117/37' 2023-10-30T13:40:32.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './117' 2023-10-30T13:40:32.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/19' 2023-10-30T13:40:32.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/15' 2023-10-30T13:40:32.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/31' 2023-10-30T13:40:32.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/36' 2023-10-30T13:40:32.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/20' 2023-10-30T13:40:32.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/12' 2023-10-30T13:40:32.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/39' 2023-10-30T13:40:32.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/10' 2023-10-30T13:40:32.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/34' 2023-10-30T13:40:32.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/38' 2023-10-30T13:40:32.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/11' 2023-10-30T13:40:32.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/16' 2023-10-30T13:40:32.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/32' 2023-10-30T13:40:32.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/30' 2023-10-30T13:40:32.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/14' 2023-10-30T13:40:32.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/18' 2023-10-30T13:40:32.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/13' 2023-10-30T13:40:32.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/33' 2023-10-30T13:40:32.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/17' 2023-10-30T13:40:32.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/35' 2023-10-30T13:40:32.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './118/37' 2023-10-30T13:40:32.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './118' 2023-10-30T13:40:32.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/19' 2023-10-30T13:40:32.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/15' 2023-10-30T13:40:32.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/31' 2023-10-30T13:40:32.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/36' 2023-10-30T13:40:32.713 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/20' 2023-10-30T13:40:32.713 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/12' 2023-10-30T13:40:40.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/39' 2023-10-30T13:40:40.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/10' 2023-10-30T13:40:40.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/34' 2023-10-30T13:40:40.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/38' 2023-10-30T13:40:40.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/11' 2023-10-30T13:40:40.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/16' 2023-10-30T13:40:40.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/32' 2023-10-30T13:40:40.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/30' 2023-10-30T13:40:40.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/14' 2023-10-30T13:40:40.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/18' 2023-10-30T13:40:40.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/13' 2023-10-30T13:40:40.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/33' 2023-10-30T13:40:40.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/17' 2023-10-30T13:40:40.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/35' 2023-10-30T13:40:40.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './119/37' 2023-10-30T13:40:40.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './119' 2023-10-30T13:40:40.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/19' 2023-10-30T13:40:40.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/15' 2023-10-30T13:40:40.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/31' 2023-10-30T13:40:40.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/36' 2023-10-30T13:40:40.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/20' 2023-10-30T13:40:40.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/12' 2023-10-30T13:40:40.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/39' 2023-10-30T13:40:40.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/10' 2023-10-30T13:40:40.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/34' 2023-10-30T13:40:40.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/38' 2023-10-30T13:40:40.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/11' 2023-10-30T13:40:40.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/16' 2023-10-30T13:40:40.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/32' 2023-10-30T13:40:40.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/30' 2023-10-30T13:40:40.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/14' 2023-10-30T13:40:40.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/18' 2023-10-30T13:40:40.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/13' 2023-10-30T13:40:40.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/33' 2023-10-30T13:40:40.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/17' 2023-10-30T13:40:40.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/35' 2023-10-30T13:40:40.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './120/37' 2023-10-30T13:40:40.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './120' 2023-10-30T13:40:40.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/19' 2023-10-30T13:40:40.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/15' 2023-10-30T13:40:40.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/31' 2023-10-30T13:40:40.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/36' 2023-10-30T13:40:40.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/20' 2023-10-30T13:40:40.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/12' 2023-10-30T13:40:40.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/39' 2023-10-30T13:40:40.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/10' 2023-10-30T13:40:40.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/34' 2023-10-30T13:40:40.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/38' 2023-10-30T13:40:40.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/11' 2023-10-30T13:40:40.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/16' 2023-10-30T13:40:40.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/32' 2023-10-30T13:40:40.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/30' 2023-10-30T13:40:40.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/14' 2023-10-30T13:40:40.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/18' 2023-10-30T13:40:40.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/13' 2023-10-30T13:40:40.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/33' 2023-10-30T13:40:40.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/17' 2023-10-30T13:40:40.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/35' 2023-10-30T13:40:40.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './121/37' 2023-10-30T13:40:40.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './121' 2023-10-30T13:40:40.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/19' 2023-10-30T13:40:40.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/15' 2023-10-30T13:40:40.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/31' 2023-10-30T13:40:40.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/36' 2023-10-30T13:40:40.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/20' 2023-10-30T13:40:40.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/12' 2023-10-30T13:40:40.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/39' 2023-10-30T13:40:40.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/10' 2023-10-30T13:40:40.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/34' 2023-10-30T13:40:40.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/38' 2023-10-30T13:40:40.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/11' 2023-10-30T13:40:40.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/16' 2023-10-30T13:40:40.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/32' 2023-10-30T13:40:40.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/30' 2023-10-30T13:40:40.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/14' 2023-10-30T13:40:40.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/18' 2023-10-30T13:40:40.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/13' 2023-10-30T13:40:40.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/33' 2023-10-30T13:40:40.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/17' 2023-10-30T13:40:40.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/35' 2023-10-30T13:40:40.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './122/37' 2023-10-30T13:40:40.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './122' 2023-10-30T13:40:40.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/19' 2023-10-30T13:40:40.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/15' 2023-10-30T13:40:40.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/31' 2023-10-30T13:40:40.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/36' 2023-10-30T13:40:40.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/20' 2023-10-30T13:40:40.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/12' 2023-10-30T13:40:40.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/39' 2023-10-30T13:40:40.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/10' 2023-10-30T13:40:40.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/34' 2023-10-30T13:40:40.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/38' 2023-10-30T13:40:40.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/11' 2023-10-30T13:40:40.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/16' 2023-10-30T13:40:40.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/32' 2023-10-30T13:40:40.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/30' 2023-10-30T13:40:40.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/14' 2023-10-30T13:40:40.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/18' 2023-10-30T13:40:40.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/13' 2023-10-30T13:40:40.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/33' 2023-10-30T13:40:40.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/17' 2023-10-30T13:40:40.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/35' 2023-10-30T13:40:40.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './123/37' 2023-10-30T13:40:40.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './123' 2023-10-30T13:40:40.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/19' 2023-10-30T13:40:40.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/15' 2023-10-30T13:40:40.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/31' 2023-10-30T13:40:40.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/36' 2023-10-30T13:40:40.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/20' 2023-10-30T13:40:40.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/12' 2023-10-30T13:40:40.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/39' 2023-10-30T13:40:40.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/10' 2023-10-30T13:40:40.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/34' 2023-10-30T13:40:40.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/38' 2023-10-30T13:40:40.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/11' 2023-10-30T13:40:40.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/16' 2023-10-30T13:40:40.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/32' 2023-10-30T13:40:40.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/30' 2023-10-30T13:40:40.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/14' 2023-10-30T13:40:40.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/18' 2023-10-30T13:40:40.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/13' 2023-10-30T13:40:40.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/33' 2023-10-30T13:40:40.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/17' 2023-10-30T13:40:40.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/35' 2023-10-30T13:40:40.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './124/37' 2023-10-30T13:40:40.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './124' 2023-10-30T13:40:40.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/19' 2023-10-30T13:40:40.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/15' 2023-10-30T13:40:40.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/31' 2023-10-30T13:40:40.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/36' 2023-10-30T13:40:40.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/20' 2023-10-30T13:40:40.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/12' 2023-10-30T13:40:40.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/39' 2023-10-30T13:40:40.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/10' 2023-10-30T13:40:40.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/34' 2023-10-30T13:40:40.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/38' 2023-10-30T13:40:40.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/11' 2023-10-30T13:40:40.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/16' 2023-10-30T13:40:40.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/32' 2023-10-30T13:40:40.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/30' 2023-10-30T13:40:40.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/14' 2023-10-30T13:40:40.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/18' 2023-10-30T13:40:40.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/13' 2023-10-30T13:40:40.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/33' 2023-10-30T13:40:40.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/17' 2023-10-30T13:40:40.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/35' 2023-10-30T13:40:40.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './125/37' 2023-10-30T13:40:40.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './125' 2023-10-30T13:40:40.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/19' 2023-10-30T13:40:40.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/15' 2023-10-30T13:40:40.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/31' 2023-10-30T13:40:40.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/36' 2023-10-30T13:40:40.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/20' 2023-10-30T13:40:40.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/12' 2023-10-30T13:40:40.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/39' 2023-10-30T13:40:40.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/10' 2023-10-30T13:40:40.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/34' 2023-10-30T13:40:40.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/38' 2023-10-30T13:40:40.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/11' 2023-10-30T13:40:40.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/16' 2023-10-30T13:40:40.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/32' 2023-10-30T13:40:40.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/30' 2023-10-30T13:40:40.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/14' 2023-10-30T13:40:40.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/18' 2023-10-30T13:40:40.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/13' 2023-10-30T13:40:40.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/33' 2023-10-30T13:40:40.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/17' 2023-10-30T13:40:40.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/35' 2023-10-30T13:40:40.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './126/37' 2023-10-30T13:40:40.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './126' 2023-10-30T13:40:40.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/19' 2023-10-30T13:40:40.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/15' 2023-10-30T13:40:40.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/31' 2023-10-30T13:40:40.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/36' 2023-10-30T13:40:40.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/20' 2023-10-30T13:40:40.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/12' 2023-10-30T13:40:40.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/39' 2023-10-30T13:40:40.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/10' 2023-10-30T13:40:40.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/34' 2023-10-30T13:40:40.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/38' 2023-10-30T13:40:40.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/11' 2023-10-30T13:40:40.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/16' 2023-10-30T13:40:40.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/32' 2023-10-30T13:40:40.246 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/30' 2023-10-30T13:40:40.246 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/14' 2023-10-30T13:40:40.246 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/18' 2023-10-30T13:40:40.246 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/13' 2023-10-30T13:40:40.246 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/33' 2023-10-30T13:40:40.247 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/17' 2023-10-30T13:40:40.247 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/35' 2023-10-30T13:40:40.247 INFO:tasks.workunit.client.0.smithi031.stdout:removed './127/37' 2023-10-30T13:40:40.247 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './127' 2023-10-30T13:40:40.247 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/19' 2023-10-30T13:40:40.248 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/15' 2023-10-30T13:40:40.248 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/31' 2023-10-30T13:40:40.248 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/36' 2023-10-30T13:40:40.248 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/20' 2023-10-30T13:40:40.248 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/12' 2023-10-30T13:40:40.248 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/39' 2023-10-30T13:40:40.249 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/10' 2023-10-30T13:40:40.249 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/34' 2023-10-30T13:40:40.249 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/38' 2023-10-30T13:40:40.249 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/11' 2023-10-30T13:40:40.249 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/16' 2023-10-30T13:40:40.250 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/32' 2023-10-30T13:40:40.250 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/30' 2023-10-30T13:40:40.250 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/14' 2023-10-30T13:40:40.250 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/18' 2023-10-30T13:40:40.250 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/13' 2023-10-30T13:40:40.251 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/33' 2023-10-30T13:40:40.251 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/17' 2023-10-30T13:40:40.251 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/35' 2023-10-30T13:40:41.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:41.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.341+0000 7fef3413e640 1 Processor -- start 2023-10-30T13:40:41.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.341+0000 7fef3413e640 1 -- start start 2023-10-30T13:40:41.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fef2c151c20 0x7fef2c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c0cb8b0 0x7fef2c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c0c5030 con 0x7fef2c151840 2023-10-30T13:40:41.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2c157830 con 0x7fef2c151c20 2023-10-30T13:40:41.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c157b10 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef326db640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c0cb8b0 0x7fef2c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef316d9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fef2c151c20 0x7fef2c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef326db640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c0cb8b0 0x7fef2c06b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:46552/0 (socket says 172.21.15.31:46552) 2023-10-30T13:40:41.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef326db640 1 -- 172.21.15.31:0/1174063375 learned_addr learned my addr 172.21.15.31:0/1174063375 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:41.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef316d9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fef2c151c20 0x7fef2c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:58032/0 (socket says 172.21.15.31:58032) 2023-10-30T13:40:41.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef31eda640 1 --1- 172.21.15.31:0/1174063375 >> v1:172.21.15.33:6789/0 conn(0x7fef2c151840 0x7fef2c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:53476/0 (socket says 172.21.15.31:53476) 2023-10-30T13:40:41.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef316d9640 1 -- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c0cb8b0 msgr2=0x7fef2c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:41.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef316d9640 1 --2- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c0cb8b0 0x7fef2c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef316d9640 1 -- 172.21.15.31:0/1174063375 >> v1:172.21.15.33:6789/0 conn(0x7fef2c151840 legacy=0x7fef2c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef316d9640 1 -- 172.21.15.31:0/1174063375 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c157df0 con 0x7fef2c151c20 2023-10-30T13:40:41.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef326db640 1 --2- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c0cb8b0 0x7fef2c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:40:41.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef316d9640 1 --2- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 0x7fef2c0c49a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fef1c0097c0 tx=0x7fef1c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3308e08ecb7acdfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:41.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef30ed8640 1 -- 172.21.15.31:0/1174063375 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0a8970 con 0x7fef2c151c20 2023-10-30T13:40:41.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef30ed8640 1 -- 172.21.15.31:0/1174063375 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c0a8ad0 con 0x7fef2c151c20 2023-10-30T13:40:41.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef30ed8640 1 -- 172.21.15.31:0/1174063375 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0a8e60 con 0x7fef2c151c20 2023-10-30T13:40:41.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 msgr2=0x7fef2c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 --2- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 0x7fef2c0c49a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fef1c0097c0 tx=0x7fef1c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:40:41.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- 172.21.15.31:0/1174063375 shutdown_connections 2023-10-30T13:40:41.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 --2- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c0cb8b0 0x7fef2c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 --2- 172.21.15.31:0/1174063375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 0x7fef2c0c49a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- 172.21.15.31:0/1174063375 >> 172.21.15.31:0/1174063375 conn(0x7fef2c0cac70 msgr2=0x7fef2c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:41.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- 172.21.15.31:0/1174063375 shutdown_connections 2023-10-30T13:40:41.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- 172.21.15.31:0/1174063375 wait complete. 2023-10-30T13:40:41.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 Processor -- start 2023-10-30T13:40:41.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.345+0000 7fef3413e640 1 -- start start 2023-10-30T13:40:41.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef2c0cb8b0 0x7fef2c0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c151840 0x7fef2c14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 0x7fef2c148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c14b3f0 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c1effb0 con 0x7fef2c151c20 2023-10-30T13:40:41.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c1efd40 con 0x7fef2c151840 2023-10-30T13:40:41.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef2c0cb8b0 0x7fef2c0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef2c0cb8b0 0x7fef2c0c9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57538/0 (socket says 172.21.15.31:57538) 2023-10-30T13:40:41.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 -- 172.21.15.31:0/2123126458 learned_addr learned my addr 172.21.15.31:0/2123126458 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:41.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef316d9640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c151840 0x7fef2c14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef326db640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 0x7fef2c148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 -- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c151840 msgr2=0x7fef2c14a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c151840 0x7fef2c14a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 -- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 msgr2=0x7fef2c148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 0x7fef2c148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 -- 172.21.15.31:0/2123126458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24009b30 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef31eda640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef2c0cb8b0 0x7fef2c0c9c70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c4c0 tx=0x7fef2800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:41.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef227fc640 1 -- 172.21.15.31:0/2123126458 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28007480 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef227fc640 1 -- 172.21.15.31:0/2123126458 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef280075e0 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c009420 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef227fc640 1 -- 172.21.15.31:0/2123126458 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28004a10 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c1487b0 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c04e170 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef227fc640 1 -- 172.21.15.31:0/2123126458 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef28004b70 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef227fc640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fef100664a0 0x7fef10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef227fc640 1 -- 172.21.15.31:0/2123126458 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6802+0+0 (secure 0 0 0) 0x7fef280835d0 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef316d9640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fef100664a0 0x7fef10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.353+0000 7fef316d9640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fef100664a0 0x7fef10068960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef1c0b2000 tx=0x7fef1c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:41.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.361+0000 7fef227fc640 1 -- 172.21.15.31:0/2123126458 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef28050720 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef2c150830 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef227fc640 1 -- 172.21.15.31:0/2123126458 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fef28053d40 con 0x7fef2c0cb8b0 2023-10-30T13:40:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:41.555 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fef100664a0 msgr2=0x7fef10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fef100664a0 0x7fef10068960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef1c0b2000 tx=0x7fef1c008040 comp rx=0 tx=0).stop 2023-10-30T13:40:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef2c0cb8b0 msgr2=0x7fef2c0c9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef2c0cb8b0 0x7fef2c0c9c70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c4c0 tx=0x7fef2800c980 comp rx=0 tx=0).stop 2023-10-30T13:40:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 shutdown_connections 2023-10-30T13:40:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fef100664a0 0x7fef10068960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef2c151c20 0x7fef2c148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fef2c151840 0x7fef2c14a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 --2- 172.21.15.31:0/2123126458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef2c0cb8b0 0x7fef2c0c9c70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 >> 172.21.15.31:0/2123126458 conn(0x7fef2c0cac70 msgr2=0x7fef2c072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:41.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 shutdown_connections 2023-10-30T13:40:41.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.549+0000 7fef3413e640 1 -- 172.21.15.31:0/2123126458 wait complete. 2023-10-30T13:40:41.575 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2023-10-30T13:40:41.575 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T13:40:41.576 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:40:41.599 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T13:40:41.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5628cea640 1 Processor -- start 2023-10-30T13:40:41.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5628cea640 1 -- start start 2023-10-30T13:40:41.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5628cea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f5624151840 0x7f56240c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5628cea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240c5030 con 0x7f56240cb8b0 2023-10-30T13:40:41.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5628cea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5624157830 con 0x7f5624151c20 2023-10-30T13:40:41.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5628cea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5624157b10 con 0x7f5624151840 2023-10-30T13:40:41.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5622d9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f56240cb8b0 0x7f562406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:44940/0 (socket says 172.21.15.31:44940) 2023-10-30T13:40:41.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f5622d9d640 1 -- 172.21.15.31:0/3046294787 learned_addr learned my addr 172.21.15.31:0/3046294787 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:40:41.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f562259c640 1 --2- 172.21.15.31:0/3046294787 >> v2:172.21.15.33:3300/0 conn(0x7f5624151840 0x7f56240c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f562159a640 1 -- 172.21.15.31:0/3046294787 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948268988 0 0) 0x7f56240c5030 con 0x7f56240cb8b0 2023-10-30T13:40:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f562159a640 1 -- 172.21.15.31:0/3046294787 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c003530 con 0x7f56240cb8b0 2023-10-30T13:40:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f562159a640 1 -- 172.21.15.31:0/3046294787 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593732353 0 0) 0x7f560c003530 con 0x7f56240cb8b0 2023-10-30T13:40:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f562259c640 1 -- 172.21.15.31:0/3046294787 >> v1:172.21.15.33:6789/0 conn(0x7f5624151c20 legacy=0x7f5624072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f562259c640 1 -- 172.21.15.31:0/3046294787 >> v1:172.21.15.31:6789/0 conn(0x7f56240cb8b0 legacy=0x7f562406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.741+0000 7f562259c640 1 -- 172.21.15.31:0/3046294787 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5624157df0 con 0x7f5624151840 2023-10-30T13:40:41.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f562259c640 1 --2- 172.21.15.31:0/3046294787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5624151840 0x7f56240c49a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f56140097c0 tx=0x7f561401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b8adbb884945154 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:41.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f562159a640 1 -- 172.21.15.31:0/3046294787 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c001a00 con 0x7f5624151840 2023-10-30T13:40:41.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f562159a640 1 -- 172.21.15.31:0/3046294787 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56140a8610 con 0x7f5624151840 2023-10-30T13:40:41.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f562159a640 1 -- 172.21.15.31:0/3046294787 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56140a89a0 con 0x7f5624151840 2023-10-30T13:40:41.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f5628cea640 1 -- 172.21.15.31:0/3046294787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5624151840 msgr2=0x7f56240c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f5628cea640 1 --2- 172.21.15.31:0/3046294787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5624151840 0x7f56240c49a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f56140097c0 tx=0x7f561401eb90 comp rx=0 tx=0).stop 2023-10-30T13:40:41.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f5628cea640 1 -- 172.21.15.31:0/3046294787 shutdown_connections 2023-10-30T13:40:41.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f5628cea640 1 --2- 172.21.15.31:0/3046294787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5624151840 0x7f56240c49a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f5628cea640 1 -- 172.21.15.31:0/3046294787 >> 172.21.15.31:0/3046294787 conn(0x7f56240cac70 msgr2=0x7f56240cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:41.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f5628cea640 1 -- 172.21.15.31:0/3046294787 shutdown_connections 2023-10-30T13:40:41.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.745+0000 7f5628cea640 1 -- 172.21.15.31:0/3046294787 wait complete. 2023-10-30T13:40:41.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 Processor -- start 2023-10-30T13:40:41.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 -- start start 2023-10-30T13:40:41.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56240cb8b0 0x7f56241ddb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5624151840 0x7f5624144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5624151c20 0x7f5624144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5624070220 con 0x7f5624151c20 2023-10-30T13:40:41.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56241df860 con 0x7f56240cb8b0 2023-10-30T13:40:41.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5628cea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56241df5c0 con 0x7f5624151840 2023-10-30T13:40:41.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5624151c20 0x7f5624144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f562259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56240cb8b0 0x7f56241ddb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5624151c20 0x7f5624144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57544/0 (socket says 172.21.15.31:57544) 2023-10-30T13:40:41.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 -- 172.21.15.31:0/3791101397 learned_addr learned my addr 172.21.15.31:0/3791101397 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:41.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5621d9b640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5624151840 0x7f5624144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 -- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5624151840 msgr2=0x7f5624144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5624151840 0x7f5624144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 -- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56240cb8b0 msgr2=0x7f56241ddb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56240cb8b0 0x7f56241ddb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 -- 172.21.15.31:0/3791101397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5614009420 con 0x7f5624151c20 2023-10-30T13:40:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5622d9d640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5624151c20 0x7f5624144da0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f561c0018a0 tx=0x7f561c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5612ffd640 1 -- 172.21.15.31:0/3791101397 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c012400 con 0x7f5624151c20 2023-10-30T13:40:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.749+0000 7f5612ffd640 1 -- 172.21.15.31:0/3791101397 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c012560 con 0x7f5624151c20 2023-10-30T13:40:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f5612ffd640 1 -- 172.21.15.31:0/3791101397 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c012910 con 0x7f5624151c20 2023-10-30T13:40:41.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5624071280 con 0x7f5624151c20 2023-10-30T13:40:41.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5624071800 con 0x7f5624151c20 2023-10-30T13:40:41.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562404e170 con 0x7f5624151c20 2023-10-30T13:40:41.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f5612ffd640 1 -- 172.21.15.31:0/3791101397 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f561c012a70 con 0x7f5624151c20 2023-10-30T13:40:41.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f5612ffd640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5604066470 0x7f5604068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:41.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f5612ffd640 1 -- 172.21.15.31:0/3791101397 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6802+0+0 (secure 0 0 0) 0x7f561c084a30 con 0x7f5624151c20 2023-10-30T13:40:41.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.753+0000 7f562259c640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5604066470 0x7f5604068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:41.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.761+0000 7f562259c640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5604066470 0x7f5604068930 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f56140b2000 tx=0x7f5614008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:41.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.761+0000 7f5612ffd640 1 -- 172.21.15.31:0/3791101397 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f561c051af0 con 0x7f5624151c20 2023-10-30T13:40:41.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56240c4aa0 con 0x7f5624151c20 2023-10-30T13:40:41.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5612ffd640 1 -- 172.21.15.31:0/3791101397 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f561c055110 con 0x7f5624151c20 2023-10-30T13:40:41.954 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:41.955 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:41.955 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:41.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5604066470 msgr2=0x7f5604068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5604066470 0x7f5604068930 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f56140b2000 tx=0x7f5614008040 comp rx=0 tx=0).stop 2023-10-30T13:40:41.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5624151c20 msgr2=0x7f5624144da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:41.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5624151c20 0x7f5624144da0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f561c0018a0 tx=0x7f561c00a430 comp rx=0 tx=0).stop 2023-10-30T13:40:41.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 shutdown_connections 2023-10-30T13:40:41.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5604066470 0x7f5604068930 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5624151c20 0x7f5624144da0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5624151840 0x7f5624144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 --2- 172.21.15.31:0/3791101397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56240cb8b0 0x7f56241ddb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:41.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 >> 172.21.15.31:0/3791101397 conn(0x7f56240cac70 msgr2=0x7f5624150490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:41.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 shutdown_connections 2023-10-30T13:40:41.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:41.949+0000 7f5628cea640 1 -- 172.21.15.31:0/3791101397 wait complete. 2023-10-30T13:40:41.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:40:43.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b4b66b640 1 Processor -- start 2023-10-30T13:40:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b4b66b640 1 -- start start 2023-10-30T13:40:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b4b66b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b44151840 0x7f5b440c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b4b66b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5b440cb8b0 0x7f5b4406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b4b66b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b440c5030 con 0x7f5b44151c20 2023-10-30T13:40:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b4b66b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b44157830 con 0x7f5b440cb8b0 2023-10-30T13:40:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b4b66b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b44157b10 con 0x7f5b44151840 2023-10-30T13:40:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49c08640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5b440cb8b0 0x7f5b4406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49c08640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5b440cb8b0 0x7f5b4406b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57556/0 (socket says 172.21.15.31:57556) 2023-10-30T13:40:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49c08640 1 -- 172.21.15.31:0/651659551 learned_addr learned my addr 172.21.15.31:0/651659551 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b48c06640 1 --1- 172.21.15.31:0/651659551 >> v1:172.21.15.31:6789/0 conn(0x7f5b44151c20 0x7f5b44072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:44956/0 (socket says 172.21.15.31:44956) 2023-10-30T13:40:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49407640 1 --2- 172.21.15.31:0/651659551 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b44151840 0x7f5b440c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49c08640 1 -- 172.21.15.31:0/651659551 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b44151840 msgr2=0x7f5b440c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49c08640 1 --2- 172.21.15.31:0/651659551 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b44151840 0x7f5b440c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49c08640 1 -- 172.21.15.31:0/651659551 >> v1:172.21.15.31:6789/0 conn(0x7f5b44151c20 legacy=0x7f5b44072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:44.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b49c08640 1 -- 172.21.15.31:0/651659551 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b44157df0 con 0x7f5b440cb8b0 2023-10-30T13:40:44.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.129+0000 7f5b3bfff640 1 -- 172.21.15.31:0/651659551 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225062296 0 0) 0x7f5b440c5030 con 0x7f5b44151c20 2023-10-30T13:40:44.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b49c08640 1 --2- 172.21.15.31:0/651659551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b440cb8b0 0x7f5b4406b600 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009e50 tx=0x7f5b3c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf990f18d4fb8da9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:44.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b3bfff640 1 -- 172.21.15.31:0/651659551 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0af400 con 0x7f5b440cb8b0 2023-10-30T13:40:44.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b3bfff640 1 -- 172.21.15.31:0/651659551 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3c0ab050 con 0x7f5b440cb8b0 2023-10-30T13:40:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b3bfff640 1 -- 172.21.15.31:0/651659551 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0af780 con 0x7f5b440cb8b0 2023-10-30T13:40:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 -- 172.21.15.31:0/651659551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b440cb8b0 msgr2=0x7f5b4406b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/651659551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b440cb8b0 0x7f5b4406b600 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009e50 tx=0x7f5b3c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 -- 172.21.15.31:0/651659551 shutdown_connections 2023-10-30T13:40:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/651659551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b440cb8b0 0x7f5b4406b600 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/651659551 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b44151840 0x7f5b440c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 -- 172.21.15.31:0/651659551 >> 172.21.15.31:0/651659551 conn(0x7f5b440cac70 msgr2=0x7f5b440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 -- 172.21.15.31:0/651659551 shutdown_connections 2023-10-30T13:40:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.133+0000 7f5b4b66b640 1 -- 172.21.15.31:0/651659551 wait complete. 2023-10-30T13:40:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 Processor -- start 2023-10-30T13:40:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 -- start start 2023-10-30T13:40:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b440cb8b0 0x7f5b44156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b44151840 0x7f5b44147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b44151c20 0x7f5b44147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b44156100 con 0x7f5b44151840 2023-10-30T13:40:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b441e1d10 con 0x7f5b44151c20 2023-10-30T13:40:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b44071800 con 0x7f5b440cb8b0 2023-10-30T13:40:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b44151840 0x7f5b44147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b44151840 0x7f5b44147560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57566/0 (socket says 172.21.15.31:57566) 2023-10-30T13:40:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 -- 172.21.15.31:0/1038555411 learned_addr learned my addr 172.21.15.31:0/1038555411 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b49407640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b440cb8b0 0x7f5b44156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b49c08640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b44151c20 0x7f5b44147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 -- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b440cb8b0 msgr2=0x7f5b44156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b440cb8b0 0x7f5b44156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 -- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b44151c20 msgr2=0x7f5b44147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b44151c20 0x7f5b44147aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 -- 172.21.15.31:0/1038555411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3c009b30 con 0x7f5b44151840 2023-10-30T13:40:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b49c08640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b44151c20 0x7f5b44147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b48c06640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b44151840 0x7f5b44147560 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400c450 tx=0x7f5b3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b39ffb640 1 -- 172.21.15.31:0/1038555411 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b340074e0 con 0x7f5b44151840 2023-10-30T13:40:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b39ffb640 1 -- 172.21.15.31:0/1038555411 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b34007640 con 0x7f5b44151840 2023-10-30T13:40:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b39ffb640 1 -- 172.21.15.31:0/1038555411 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b340049f0 con 0x7f5b44151840 2023-10-30T13:40:44.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b44156660 con 0x7f5b44151840 2023-10-30T13:40:44.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.137+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b44155e20 con 0x7f5b44151840 2023-10-30T13:40:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.141+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b441567f0 con 0x7f5b44151840 2023-10-30T13:40:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.141+0000 7f5b39ffb640 1 -- 172.21.15.31:0/1038555411 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b34004b50 con 0x7f5b44151840 2023-10-30T13:40:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.141+0000 7f5b39ffb640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5b280664d0 0x7f5b28068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.141+0000 7f5b39ffb640 1 -- 172.21.15.31:0/1038555411 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f5b34083280 con 0x7f5b44151840 2023-10-30T13:40:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.141+0000 7f5b49407640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5b280664d0 0x7f5b28068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:44.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.145+0000 7f5b49407640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5b280664d0 0x7f5b28068990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000b9d0 tx=0x7f5b40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:44.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.145+0000 7f5b39ffb640 1 -- 172.21.15.31:0/1038555411 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b34053f80 con 0x7f5b44151840 2023-10-30T13:40:44.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.333+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b4404e170 con 0x7f5b44151840 2023-10-30T13:40:44.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b39ffb640 1 -- 172.21.15.31:0/1038555411 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f5b34018090 con 0x7f5b44151840 2023-10-30T13:40:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:44.337 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5b280664d0 msgr2=0x7f5b28068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5b280664d0 0x7f5b28068990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000b9d0 tx=0x7f5b40008040 comp rx=0 tx=0).stop 2023-10-30T13:40:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b44151840 msgr2=0x7f5b44147560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b44151840 0x7f5b44147560 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400c450 tx=0x7f5b3400c910 comp rx=0 tx=0).stop 2023-10-30T13:40:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 shutdown_connections 2023-10-30T13:40:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5b280664d0 0x7f5b28068990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b44151c20 0x7f5b44147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b44151840 0x7f5b44147560 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 --2- 172.21.15.31:0/1038555411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b440cb8b0 0x7f5b44156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:44.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 >> 172.21.15.31:0/1038555411 conn(0x7f5b440cac70 msgr2=0x7f5b440c8030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:44.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 shutdown_connections 2023-10-30T13:40:44.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:44.337+0000 7f5b4b66b640 1 -- 172.21.15.31:0/1038555411 wait complete. 2023-10-30T13:40:44.359 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:40:46.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:46.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.513+0000 7fc47bff5640 1 Processor -- start 2023-10-30T13:40:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.513+0000 7fc47bff5640 1 -- start start 2023-10-30T13:40:46.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc47bff5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc474153c10 0x7fc474065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:46.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc47bff5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47406e390 con 0x7fc474159970 2023-10-30T13:40:46.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc47bff5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740660e0 con 0x7fc474070c30 2023-10-30T13:40:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc47bff5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc474072390 con 0x7fc474153c10 2023-10-30T13:40:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc47a592640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc474159970 0x7fc474159dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41060/0 (socket says 172.21.15.31:41060) 2023-10-30T13:40:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc47a592640 1 -- 172.21.15.31:0/78508401 learned_addr learned my addr 172.21.15.31:0/78508401 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:40:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc479d91640 1 --2- 172.21.15.31:0/78508401 >> v2:172.21.15.33:3300/0 conn(0x7fc474153c10 0x7fc474065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:46.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc478d8f640 1 -- 172.21.15.31:0/78508401 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677809651 0 0) 0x7fc47406e390 con 0x7fc474159970 2023-10-30T13:40:46.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc478d8f640 1 -- 172.21.15.31:0/78508401 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc460003530 con 0x7fc474159970 2023-10-30T13:40:46.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc479d91640 1 -- 172.21.15.31:0/78508401 >> v1:172.21.15.33:6789/0 conn(0x7fc474070c30 legacy=0x7fc474071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:46.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc478d8f640 1 -- 172.21.15.31:0/78508401 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500885702 0 0) 0x7fc460003530 con 0x7fc474159970 2023-10-30T13:40:46.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc479d91640 1 -- 172.21.15.31:0/78508401 >> v1:172.21.15.31:6789/0 conn(0x7fc474159970 legacy=0x7fc474159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:46.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.517+0000 7fc479d91640 1 -- 172.21.15.31:0/78508401 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740bec10 con 0x7fc474153c10 2023-10-30T13:40:46.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc479d91640 1 --2- 172.21.15.31:0/78508401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc474065ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc4700097c0 tx=0x7fc47001c1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f470ba32d7e1b89d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:46.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc478d8f640 1 -- 172.21.15.31:0/78508401 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c001a00 con 0x7fc474153c10 2023-10-30T13:40:46.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc478d8f640 1 -- 172.21.15.31:0/78508401 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4700a4c00 con 0x7fc474153c10 2023-10-30T13:40:46.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc478d8f640 1 -- 172.21.15.31:0/78508401 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4700ac440 con 0x7fc474153c10 2023-10-30T13:40:46.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 -- 172.21.15.31:0/78508401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 msgr2=0x7fc474065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:46.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 --2- 172.21.15.31:0/78508401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc474065ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc4700097c0 tx=0x7fc47001c1c0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 -- 172.21.15.31:0/78508401 shutdown_connections 2023-10-30T13:40:46.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 --2- 172.21.15.31:0/78508401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc474065ba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 -- 172.21.15.31:0/78508401 >> 172.21.15.31:0/78508401 conn(0x7fc4740c7f60 msgr2=0x7fc4740c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 -- 172.21.15.31:0/78508401 shutdown_connections 2023-10-30T13:40:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 -- 172.21.15.31:0/78508401 wait complete. 2023-10-30T13:40:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 Processor -- start 2023-10-30T13:40:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.521+0000 7fc47bff5640 1 -- start start 2023-10-30T13:40:46.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc474070c30 0x7fc4741511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:46.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc47414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:46.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc474159970 0x7fc47414deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:46.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc474148750 con 0x7fc474070c30 2023-10-30T13:40:46.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc474151720 con 0x7fc474153c10 2023-10-30T13:40:46.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4741501d0 con 0x7fc474159970 2023-10-30T13:40:46.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc474070c30 0x7fc4741511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:46.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc474070c30 0x7fc4741511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55928/0 (socket says 172.21.15.31:55928) 2023-10-30T13:40:46.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 -- 172.21.15.31:0/116280571 learned_addr learned my addr 172.21.15.31:0/116280571 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:46.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47a592640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc474159970 0x7fc47414deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:46.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479590640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc47414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:46.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 -- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc474159970 msgr2=0x7fc47414deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:46.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc474159970 0x7fc47414deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 -- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 msgr2=0x7fc47414d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:46.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc47414d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 -- 172.21.15.31:0/116280571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc470009420 con 0x7fc474070c30 2023-10-30T13:40:46.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479590640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc47414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:40:46.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47a592640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc474159970 0x7fc47414deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:40:46.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc479d91640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc474070c30 0x7fc4741511e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc47001c6a0 tx=0x7fc470018f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:46.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc46a7fc640 1 -- 172.21.15.31:0/116280571 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470004020 con 0x7fc474070c30 2023-10-30T13:40:46.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc46a7fc640 1 -- 172.21.15.31:0/116280571 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc470004180 con 0x7fc474070c30 2023-10-30T13:40:46.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4741f2ca0 con 0x7fc474070c30 2023-10-30T13:40:46.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc46a7fc640 1 -- 172.21.15.31:0/116280571 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470003620 con 0x7fc474070c30 2023-10-30T13:40:46.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc474150700 con 0x7fc474070c30 2023-10-30T13:40:46.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47404e170 con 0x7fc474070c30 2023-10-30T13:40:46.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.525+0000 7fc46a7fc640 1 -- 172.21.15.31:0/116280571 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc4700b3040 con 0x7fc474070c30 2023-10-30T13:40:46.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.529+0000 7fc46a7fc640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4580664a0 0x7fc458068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:46.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.529+0000 7fc46a7fc640 1 -- 172.21.15.31:0/116280571 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc47011d1a0 con 0x7fc474070c30 2023-10-30T13:40:46.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.529+0000 7fc479590640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4580664a0 0x7fc458068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:46.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.529+0000 7fc479590640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4580664a0 0x7fc458068960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4740c7970 tx=0x7fc464009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:46.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.533+0000 7fc46a7fc640 1 -- 172.21.15.31:0/116280571 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4700ea380 con 0x7fc474070c30 2023-10-30T13:40:46.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc474158c30 con 0x7fc474070c30 2023-10-30T13:40:46.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc46a7fc640 1 -- 172.21.15.31:0/116280571 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fc4700ed9a0 con 0x7fc474070c30 2023-10-30T13:40:46.726 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:46.726 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:46.726 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4580664a0 msgr2=0x7fc458068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4580664a0 0x7fc458068960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4740c7970 tx=0x7fc464009040 comp rx=0 tx=0).stop 2023-10-30T13:40:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc474070c30 msgr2=0x7fc4741511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc474070c30 0x7fc4741511e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc47001c6a0 tx=0x7fc470018f60 comp rx=0 tx=0).stop 2023-10-30T13:40:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 shutdown_connections 2023-10-30T13:40:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc4580664a0 0x7fc458068960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc474159970 0x7fc47414deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc474153c10 0x7fc47414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 --2- 172.21.15.31:0/116280571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc474070c30 0x7fc4741511e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 >> 172.21.15.31:0/116280571 conn(0x7fc4740c7f60 msgr2=0x7fc474153110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 shutdown_connections 2023-10-30T13:40:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:46.725+0000 7fc47bff5640 1 -- 172.21.15.31:0/116280571 wait complete. 2023-10-30T13:40:46.746 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:40:48.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd708268640 1 Processor -- start 2023-10-30T13:40:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd708268640 1 -- start start 2023-10-30T13:40:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd708268640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd700153c80 0x7fd70015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd708268640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd700159f20 con 0x7fd7000c4a10 2023-10-30T13:40:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd708268640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70015d620 con 0x7fd7000c42a0 2023-10-30T13:40:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd708268640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70015dec0 con 0x7fd700153c80 2023-10-30T13:40:48.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd705803640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fd7000c4a10 0x7fd70015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41062/0 (socket says 172.21.15.31:41062) 2023-10-30T13:40:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd705803640 1 -- 172.21.15.31:0/2576679378 learned_addr learned my addr 172.21.15.31:0/2576679378 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:40:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd706004640 1 --2- 172.21.15.31:0/2576679378 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd700153c80 0x7fd70015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681233110 0 0) 0x7fd700159f20 con 0x7fd7000c4a10 2023-10-30T13:40:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6ec003530 con 0x7fd7000c4a10 2023-10-30T13:40:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210039767 0 0) 0x7fd70015d620 con 0x7fd7000c42a0 2023-10-30T13:40:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd700159f20 con 0x7fd7000c42a0 2023-10-30T13:40:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd706004640 1 -- 172.21.15.31:0/2576679378 >> v1:172.21.15.33:6789/0 conn(0x7fd7000c42a0 legacy=0x7fd7000ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd706004640 1 -- 172.21.15.31:0/2576679378 >> v1:172.21.15.31:6789/0 conn(0x7fd7000c4a10 legacy=0x7fd70015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437147709 0 0) 0x7fd6ec003530 con 0x7fd7000c4a10 2023-10-30T13:40:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.901+0000 7fd706004640 1 -- 172.21.15.31:0/2576679378 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70015d900 con 0x7fd700153c80 2023-10-30T13:40:48.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.905+0000 7fd706004640 1 --2- 172.21.15.31:0/2576679378 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd700153c80 0x7fd70015cf90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097f0 tx=0x7fd6fc01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f208935a5a0ba61 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:48.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0001680 con 0x7fd700153c80 2023-10-30T13:40:48.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc0a8650 con 0x7fd700153c80 2023-10-30T13:40:48.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd705002640 1 -- 172.21.15.31:0/2576679378 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0a89c0 con 0x7fd700153c80 2023-10-30T13:40:48.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 -- 172.21.15.31:0/2576679378 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd700153c80 msgr2=0x7fd70015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 --2- 172.21.15.31:0/2576679378 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd700153c80 0x7fd70015cf90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097f0 tx=0x7fd6fc01ebd0 comp rx=0 tx=0).stop 2023-10-30T13:40:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 -- 172.21.15.31:0/2576679378 shutdown_connections 2023-10-30T13:40:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 --2- 172.21.15.31:0/2576679378 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd700153c80 0x7fd70015cf90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 -- 172.21.15.31:0/2576679378 >> 172.21.15.31:0/2576679378 conn(0x7fd7000715c0 msgr2=0x7fd7000719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 -- 172.21.15.31:0/2576679378 shutdown_connections 2023-10-30T13:40:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 -- 172.21.15.31:0/2576679378 wait complete. 2023-10-30T13:40:48.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 Processor -- start 2023-10-30T13:40:48.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.913+0000 7fd708268640 1 -- start start 2023-10-30T13:40:48.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7000c42a0 0x7fd7000c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:48.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd7000c4a10 0x7fd70014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:48.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd700153c80 0x7fd70014ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:48.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70014fd80 con 0x7fd7000c42a0 2023-10-30T13:40:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70015bf40 con 0x7fd700153c80 2023-10-30T13:40:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70015c220 con 0x7fd7000c4a10 2023-10-30T13:40:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7000c42a0 0x7fd7000c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7000c42a0 0x7fd7000c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55940/0 (socket says 172.21.15.31:55940) 2023-10-30T13:40:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 -- 172.21.15.31:0/2305014197 learned_addr learned my addr 172.21.15.31:0/2305014197 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd705803640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd7000c4a10 0x7fd70014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706805640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd700153c80 0x7fd70014ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 -- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd7000c4a10 msgr2=0x7fd70014e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd7000c4a10 0x7fd70014e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 -- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd700153c80 msgr2=0x7fd70014ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd700153c80 0x7fd70014ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 -- 172.21.15.31:0/2305014197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc009420 con 0x7fd7000c42a0 2023-10-30T13:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706805640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd700153c80 0x7fd70014ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd706004640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7000c42a0 0x7fd7000c8a70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097f0 tx=0x7fd6fc01b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd6f6ffd640 1 -- 172.21.15.31:0/2305014197 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0c1070 con 0x7fd7000c42a0 2023-10-30T13:40:48.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd6f6ffd640 1 -- 172.21.15.31:0/2305014197 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc006e20 con 0x7fd7000c42a0 2023-10-30T13:40:48.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd6f6ffd640 1 -- 172.21.15.31:0/2305014197 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0bc860 con 0x7fd7000c42a0 2023-10-30T13:40:48.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70015c500 con 0x7fd7000c42a0 2023-10-30T13:40:48.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7001533e0 con 0x7fd7000c42a0 2023-10-30T13:40:48.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70015c690 con 0x7fd7000c42a0 2023-10-30T13:40:48.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.917+0000 7fd6f6ffd640 1 -- 172.21.15.31:0/2305014197 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6fc0bc9c0 con 0x7fd7000c42a0 2023-10-30T13:40:48.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.921+0000 7fd6f6ffd640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6e40664a0 0x7fd6e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.921+0000 7fd6f6ffd640 1 -- 172.21.15.31:0/2305014197 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fd6fc121930 con 0x7fd7000c42a0 2023-10-30T13:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.921+0000 7fd705803640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6e40664a0 0x7fd6e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.921+0000 7fd705803640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6e40664a0 0x7fd6e4068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd70014f8f0 tx=0x7fd6f0000fa0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:48.925+0000 7fd6f6ffd640 1 -- 172.21.15.31:0/2305014197 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6fc0eeb10 con 0x7fd7000c42a0 2023-10-30T13:40:49.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.113+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd70015c360 con 0x7fd7000c42a0 2023-10-30T13:40:49.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.113+0000 7fd6f6ffd640 1 -- 172.21.15.31:0/2305014197 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fd70015c360 con 0x7fd7000c42a0 2023-10-30T13:40:49.116 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:49.116 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6e40664a0 msgr2=0x7fd6e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6e40664a0 0x7fd6e4068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd70014f8f0 tx=0x7fd6f0000fa0 comp rx=0 tx=0).stop 2023-10-30T13:40:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7000c42a0 msgr2=0x7fd7000c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7000c42a0 0x7fd7000c8a70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097f0 tx=0x7fd6fc01b970 comp rx=0 tx=0).stop 2023-10-30T13:40:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 shutdown_connections 2023-10-30T13:40:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6e40664a0 0x7fd6e4068960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd700153c80 0x7fd70014ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd7000c4a10 0x7fd70014e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 --2- 172.21.15.31:0/2305014197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7000c42a0 0x7fd7000c8a70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 >> 172.21.15.31:0/2305014197 conn(0x7fd7000715c0 msgr2=0x7fd7000ca5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 shutdown_connections 2023-10-30T13:40:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:49.117+0000 7fd708268640 1 -- 172.21.15.31:0/2305014197 wait complete. 2023-10-30T13:40:49.137 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:40:51.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:51.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f863733e640 1 Processor -- start 2023-10-30T13:40:51.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f863733e640 1 -- start start 2023-10-30T13:40:51.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f863733e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8630151840 0x7f86300c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f863733e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300c5030 con 0x7f86300cb8b0 2023-10-30T13:40:51.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f863733e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630157830 con 0x7f8630151c20 2023-10-30T13:40:51.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f863733e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8630157b10 con 0x7f8630151840 2023-10-30T13:40:51.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f86358db640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f86300cb8b0 0x7f863006b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41074/0 (socket says 172.21.15.31:41074) 2023-10-30T13:40:51.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f86358db640 1 -- 172.21.15.31:0/1689530084 learned_addr learned my addr 172.21.15.31:0/1689530084 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:40:51.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f86350da640 1 --2- 172.21.15.31:0/1689530084 >> v2:172.21.15.33:3300/0 conn(0x7f8630151840 0x7f86300c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831395965 0 0) 0x7f86300c5030 con 0x7f86300cb8b0 2023-10-30T13:40:51.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c003530 con 0x7f86300cb8b0 2023-10-30T13:40:51.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 724033509 0 0) 0x7f861c003530 con 0x7f86300cb8b0 2023-10-30T13:40:51.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 >> v2:172.21.15.33:3300/0 conn(0x7f8630151840 msgr2=0x7f86300c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:51.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 --2- 172.21.15.31:0/1689530084 >> v2:172.21.15.33:3300/0 conn(0x7f8630151840 0x7f86300c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 >> v1:172.21.15.33:6789/0 conn(0x7f8630151c20 legacy=0x7f8630072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:51.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630157df0 con 0x7f86300cb8b0 2023-10-30T13:40:51.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f8628001a00 con 0x7f86300cb8b0 2023-10-30T13:40:51.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 >> v1:172.21.15.31:6789/0 conn(0x7f86300cb8b0 legacy=0x7f863006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:51.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c003980 0x7f861c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f861c0047d0 0x7f861c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f861c005190 0x7f861c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8630157b10 con 0x7f861c005190 2023-10-30T13:40:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c00c040 con 0x7f861c0047d0 2023-10-30T13:40:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c00c200 con 0x7f861c003980 2023-10-30T13:40:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f86358db640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f861c005190 0x7f861c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f86350da640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c003980 0x7f861c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.293+0000 7f86348d9640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f861c0047d0 0x7f861c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f86358db640 1 -- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c003980 msgr2=0x7f861c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f86358db640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c003980 0x7f861c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f86358db640 1 -- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f861c0047d0 msgr2=0x7f861c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f86358db640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f861c0047d0 0x7f861c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f86358db640 1 -- 172.21.15.31:0/1689530084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c003530 con 0x7f861c005190 2023-10-30T13:40:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f86358db640 1 -- 172.21.15.31:0/1689530084 reap_dead start 2023-10-30T13:40:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f86358db640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f861c005190 0x7f861c00b990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f862800a830 tx=0x7f862801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1609670e2e748df3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86280072d0 con 0x7f861c005190 2023-10-30T13:40:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86280a8070 con 0x7f861c005190 2023-10-30T13:40:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f8627fff640 1 -- 172.21.15.31:0/1689530084 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8628007690 con 0x7f861c005190 2023-10-30T13:40:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f863733e640 1 -- 172.21.15.31:0/1689530084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f861c005190 msgr2=0x7f861c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f863733e640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f861c005190 0x7f861c00b990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f862800a830 tx=0x7f862801d340 comp rx=0 tx=0).stop 2023-10-30T13:40:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f863733e640 1 -- 172.21.15.31:0/1689530084 shutdown_connections 2023-10-30T13:40:51.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f863733e640 1 --2- 172.21.15.31:0/1689530084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f861c005190 0x7f861c00b990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f863733e640 1 -- 172.21.15.31:0/1689530084 >> 172.21.15.31:0/1689530084 conn(0x7f86300cac70 msgr2=0x7f86300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:51.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.297+0000 7f863733e640 1 -- 172.21.15.31:0/1689530084 shutdown_connections 2023-10-30T13:40:51.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 -- 172.21.15.31:0/1689530084 wait complete. 2023-10-30T13:40:51.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 Processor -- start 2023-10-30T13:40:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 -- start start 2023-10-30T13:40:51.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c005190 0x7f86301ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8630072a20 0x7f86301469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f863014bad0 0x7f8630155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86300718d0 con 0x7f863014bad0 2023-10-30T13:40:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86301eec90 con 0x7f8630072a20 2023-10-30T13:40:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f863733e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863014d320 con 0x7f861c005190 2023-10-30T13:40:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f863014bad0 0x7f8630155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f863014bad0 0x7f8630155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55966/0 (socket says 172.21.15.31:55966) 2023-10-30T13:40:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 -- 172.21.15.31:0/4081393697 learned_addr learned my addr 172.21.15.31:0/4081393697 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86348d9640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8630072a20 0x7f86301469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86350da640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c005190 0x7f86301ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 -- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c005190 msgr2=0x7f86301ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c005190 0x7f86301ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 -- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8630072a20 msgr2=0x7f86301469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8630072a20 0x7f86301469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 -- 172.21.15.31:0/4081393697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86301eb1d0 con 0x7f863014bad0 2023-10-30T13:40:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86348d9640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8630072a20 0x7f86301469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.301+0000 7f86358db640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f863014bad0 0x7f8630155a80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8628007040 tx=0x7f861c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f8625ffb640 1 -- 172.21.15.31:0/4081393697 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86280072d0 con 0x7f863014bad0 2023-10-30T13:40:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f8625ffb640 1 -- 172.21.15.31:0/4081393697 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8628007430 con 0x7f863014bad0 2023-10-30T13:40:51.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f8625ffb640 1 -- 172.21.15.31:0/4081393697 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86280077e0 con 0x7f863014bad0 2023-10-30T13:40:51.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86301ea8e0 con 0x7f863014bad0 2023-10-30T13:40:51.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863014d570 con 0x7f863014bad0 2023-10-30T13:40:51.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86301eb360 con 0x7f863014bad0 2023-10-30T13:40:51.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f8625ffb640 1 -- 172.21.15.31:0/4081393697 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f86280aa070 con 0x7f863014bad0 2023-10-30T13:40:51.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f8625ffb640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f86140664d0 0x7f8614068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:51.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f8625ffb640 1 -- 172.21.15.31:0/4081393697 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f862811d890 con 0x7f863014bad0 2023-10-30T13:40:51.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.305+0000 7f86350da640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f86140664d0 0x7f8614068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.309+0000 7f86350da640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f86140664d0 0x7f8614068990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f862c009aa0 tx=0x7f862c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.309+0000 7f8625ffb640 1 -- 172.21.15.31:0/4081393697 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86280eab00 con 0x7f863014bad0 2023-10-30T13:40:51.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.501+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86301583b0 con 0x7f863014bad0 2023-10-30T13:40:51.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.501+0000 7f8625ffb640 1 -- 172.21.15.31:0/4081393697 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f86280ee120 con 0x7f863014bad0 2023-10-30T13:40:51.503 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:51.503 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:51.504 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:51.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f86140664d0 msgr2=0x7f8614068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:51.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f86140664d0 0x7f8614068990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f862c009aa0 tx=0x7f862c008040 comp rx=0 tx=0).stop 2023-10-30T13:40:51.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f863014bad0 msgr2=0x7f8630155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:51.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f863014bad0 0x7f8630155a80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8628007040 tx=0x7f861c005800 comp rx=0 tx=0).stop 2023-10-30T13:40:51.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 shutdown_connections 2023-10-30T13:40:51.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f86140664d0 0x7f8614068990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f863014bad0 0x7f8630155a80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8630072a20 0x7f86301469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 --2- 172.21.15.31:0/4081393697 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f861c005190 0x7f86301ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:51.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 >> 172.21.15.31:0/4081393697 conn(0x7f86300cac70 msgr2=0x7f86301479f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:51.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 shutdown_connections 2023-10-30T13:40:51.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:51.505+0000 7f863733e640 1 -- 172.21.15.31:0/4081393697 wait complete. 2023-10-30T13:40:51.524 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:40:53.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:53.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc817491640 1 Processor -- start 2023-10-30T13:40:53.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc817491640 1 -- start start 2023-10-30T13:40:53.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc817491640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc810151c20 0x7fc8100c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc817491640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8100c5030 con 0x7fc810151840 2023-10-30T13:40:53.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc817491640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc810157830 con 0x7fc8100cb8b0 2023-10-30T13:40:53.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc817491640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc810157b10 con 0x7fc810151c20 2023-10-30T13:40:53.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc81522d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc810151840 0x7fc810072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41090/0 (socket says 172.21.15.31:41090) 2023-10-30T13:40:53.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc81522d640 1 -- 172.21.15.31:0/1925436260 learned_addr learned my addr 172.21.15.31:0/1925436260 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:40:53.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.677+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1925436260 >> v2:172.21.15.33:3300/0 conn(0x7fc810151c20 0x7fc8100c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911318425 0 0) 0x7fc8100c5030 con 0x7fc810151840 2023-10-30T13:40:53.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc003530 con 0x7fc810151840 2023-10-30T13:40:53.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552559935 0 0) 0x7fc7fc003530 con 0x7fc810151840 2023-10-30T13:40:53.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 >> v2:172.21.15.33:3300/0 conn(0x7fc810151c20 msgr2=0x7fc8100c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:53.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 --2- 172.21.15.31:0/1925436260 >> v2:172.21.15.33:3300/0 conn(0x7fc810151c20 0x7fc8100c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 >> v1:172.21.15.33:6789/0 conn(0x7fc8100cb8b0 legacy=0x7fc81006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:53.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc810157df0 con 0x7fc810151840 2023-10-30T13:40:53.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fc80c001680 con 0x7fc810151840 2023-10-30T13:40:53.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 >> v1:172.21.15.31:6789/0 conn(0x7fc810151840 legacy=0x7fc810072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:53.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7fc003980 0x7fc7fc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc7fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7fc005190 0x7fc7fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc810157b10 con 0x7fc7fc0047d0 2023-10-30T13:40:53.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc00c040 con 0x7fc7fc005190 2023-10-30T13:40:53.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc00c200 con 0x7fc7fc003980 2023-10-30T13:40:53.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc7fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc81522d640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7fc003980 0x7fc7fc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc815a2e640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7fc005190 0x7fc7fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc814a2c640 1 -- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7fc003980 msgr2=0x7fc7fc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc7fc003980 0x7fc7fc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc814a2c640 1 -- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7fc005190 msgr2=0x7fc7fc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7fc005190 0x7fc7fc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc814a2c640 1 -- 172.21.15.31:0/1925436260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc003530 con 0x7fc7fc0047d0 2023-10-30T13:40:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc815a2e640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7fc005190 0x7fc7fc00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.681+0000 7fc815a2e640 1 -- 172.21.15.31:0/1925436260 reap_dead start 2023-10-30T13:40:53.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc7fc004c50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc800009ae0 tx=0x7fc80001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcce602a2ee05aa5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:53.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a5640 con 0x7fc7fc0047d0 2023-10-30T13:40:53.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8000a57a0 con 0x7fc7fc0047d0 2023-10-30T13:40:53.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc807fff640 1 -- 172.21.15.31:0/1925436260 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a5b50 con 0x7fc7fc0047d0 2023-10-30T13:40:53.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- 172.21.15.31:0/1925436260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 msgr2=0x7fc7fc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:53.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc7fc004c50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc800009ae0 tx=0x7fc80001ca50 comp rx=0 tx=0).stop 2023-10-30T13:40:53.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- 172.21.15.31:0/1925436260 shutdown_connections 2023-10-30T13:40:53.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 --2- 172.21.15.31:0/1925436260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc7fc004c50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- 172.21.15.31:0/1925436260 >> 172.21.15.31:0/1925436260 conn(0x7fc8100cac70 msgr2=0x7fc8100cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:53.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- 172.21.15.31:0/1925436260 shutdown_connections 2023-10-30T13:40:53.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- 172.21.15.31:0/1925436260 wait complete. 2023-10-30T13:40:53.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 Processor -- start 2023-10-30T13:40:53.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- start start 2023-10-30T13:40:53.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc8101ef500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc810071c00 0x7fc81014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc810155f00 0x7fc8101547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8101efa40 con 0x7fc7fc0047d0 2023-10-30T13:40:53.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8101564b0 con 0x7fc810155f00 2023-10-30T13:40:53.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.685+0000 7fc817491640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8101f2090 con 0x7fc810071c00 2023-10-30T13:40:53.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc8101ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc8101ef500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55978/0 (socket says 172.21.15.31:55978) 2023-10-30T13:40:53.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 -- 172.21.15.31:0/1839997832 learned_addr learned my addr 172.21.15.31:0/1839997832 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:53.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc810071c00 0x7fc81014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc815a2e640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc810155f00 0x7fc8101547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 -- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc810071c00 msgr2=0x7fc81014cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc810071c00 0x7fc81014cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 -- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc810155f00 msgr2=0x7fc8101547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc810155f00 0x7fc8101547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 -- 172.21.15.31:0/1839997832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8101f2370 con 0x7fc7fc0047d0 2023-10-30T13:40:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc81522d640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc8101ef500 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007320 tx=0x7fc80c007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:53.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc805ffb640 1 -- 172.21.15.31:0/1839997832 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c00fa00 con 0x7fc7fc0047d0 2023-10-30T13:40:53.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc805ffb640 1 -- 172.21.15.31:0/1839997832 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80c00fb80 con 0x7fc7fc0047d0 2023-10-30T13:40:53.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8101f2650 con 0x7fc7fc0047d0 2023-10-30T13:40:53.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc805ffb640 1 -- 172.21.15.31:0/1839997832 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c0133f0 con 0x7fc7fc0047d0 2023-10-30T13:40:53.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8101f2c10 con 0x7fc7fc0047d0 2023-10-30T13:40:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8101f2e00 con 0x7fc7fc0047d0 2023-10-30T13:40:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.689+0000 7fc805ffb640 1 -- 172.21.15.31:0/1839997832 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc80c013550 con 0x7fc7fc0047d0 2023-10-30T13:40:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.693+0000 7fc805ffb640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc7f40664a0 0x7fc7f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.693+0000 7fc805ffb640 1 -- 172.21.15.31:0/1839997832 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc80c084400 con 0x7fc7fc0047d0 2023-10-30T13:40:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.693+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc7f40664a0 0x7fc7f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.693+0000 7fc814a2c640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc7f40664a0 0x7fc7f4068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8000af000 tx=0x7fc800008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:53.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.697+0000 7fc805ffb640 1 -- 172.21.15.31:0/1839997832 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc80c051670 con 0x7fc7fc0047d0 2023-10-30T13:40:53.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.881+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8100c6330 con 0x7fc7fc0047d0 2023-10-30T13:40:53.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc805ffb640 1 -- 172.21.15.31:0/1839997832 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fc80c054c90 con 0x7fc7fc0047d0 2023-10-30T13:40:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:53.886 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:53.886 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:53.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc7f40664a0 msgr2=0x7fc7f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:53.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc7f40664a0 0x7fc7f4068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8000af000 tx=0x7fc800008040 comp rx=0 tx=0).stop 2023-10-30T13:40:53.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 msgr2=0x7fc8101ef500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:53.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc8101ef500 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007320 tx=0x7fc80c007720 comp rx=0 tx=0).stop 2023-10-30T13:40:53.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 shutdown_connections 2023-10-30T13:40:53.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc7f40664a0 0x7fc7f4068960 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc810155f00 0x7fc8101547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc810071c00 0x7fc81014cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 --2- 172.21.15.31:0/1839997832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7fc0047d0 0x7fc8101ef500 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:53.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 >> 172.21.15.31:0/1839997832 conn(0x7fc8100cac70 msgr2=0x7fc8100733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:53.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 shutdown_connections 2023-10-30T13:40:53.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:53.885+0000 7fc817491640 1 -- 172.21.15.31:0/1839997832 wait complete. 2023-10-30T13:40:53.906 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:40:55.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:55.593 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:55.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:56.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.057+0000 7f02dd52a640 1 Processor -- start 2023-10-30T13:40:56.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.057+0000 7f02dd52a640 1 -- start start 2023-10-30T13:40:56.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02dd52a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8153bb0 0x7f02d80c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:56.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02dd52a640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f02d8159980 0x7f02d8159df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:56.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02dd52a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80c5f80 con 0x7f02d806c420 2023-10-30T13:40:56.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02dd52a640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d815a330 con 0x7f02d8159980 2023-10-30T13:40:56.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02dd52a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80c5690 con 0x7f02d8153bb0 2023-10-30T13:40:56.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d77fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f02d8159980 0x7f02d8159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:56.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d77fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f02d8159980 0x7f02d8159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36998/0 (socket says 172.21.15.31:36998) 2023-10-30T13:40:56.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d77fe640 1 -- 172.21.15.31:0/2900654608 learned_addr learned my addr 172.21.15.31:0/2900654608 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:56.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d6ffd640 1 --2- 172.21.15.31:0/2900654608 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8153bb0 0x7f02d80c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:56.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d77fe640 1 -- 172.21.15.31:0/2900654608 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8153bb0 msgr2=0x7f02d80c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:56.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d77fe640 1 --2- 172.21.15.31:0/2900654608 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8153bb0 0x7f02d80c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d77fe640 1 -- 172.21.15.31:0/2900654608 >> v1:172.21.15.33:6789/0 conn(0x7f02d806c420 legacy=0x7f02d80c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:56.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.061+0000 7f02d77fe640 1 -- 172.21.15.31:0/2900654608 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d815a580 con 0x7f02d8159980 2023-10-30T13:40:56.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02d77fe640 1 --2- 172.21.15.31:0/2900654608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8159980 0x7f02d8159df0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009e50 tx=0x7f02d001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6dcf563ed7a0b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:56.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02d67fc640 1 -- 172.21.15.31:0/2900654608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00af400 con 0x7f02d8159980 2023-10-30T13:40:56.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02d67fc640 1 -- 172.21.15.31:0/2900654608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d00ab050 con 0x7f02d8159980 2023-10-30T13:40:56.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02d67fc640 1 -- 172.21.15.31:0/2900654608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00af780 con 0x7f02d8159980 2023-10-30T13:40:56.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 -- 172.21.15.31:0/2900654608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8159980 msgr2=0x7f02d8159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:56.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 --2- 172.21.15.31:0/2900654608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8159980 0x7f02d8159df0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009e50 tx=0x7f02d001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 -- 172.21.15.31:0/2900654608 shutdown_connections 2023-10-30T13:40:56.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 --2- 172.21.15.31:0/2900654608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8159980 0x7f02d8159df0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 --2- 172.21.15.31:0/2900654608 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8153bb0 0x7f02d80c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 -- 172.21.15.31:0/2900654608 >> 172.21.15.31:0/2900654608 conn(0x7f02d80cac70 msgr2=0x7f02d80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:56.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 -- 172.21.15.31:0/2900654608 shutdown_connections 2023-10-30T13:40:56.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 -- 172.21.15.31:0/2900654608 wait complete. 2023-10-30T13:40:56.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 Processor -- start 2023-10-30T13:40:56.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 -- start start 2023-10-30T13:40:56.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.065+0000 7f02dd52a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02d806c420 0x7f02d81ee690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:56.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8153bb0 0x7f02d8150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:56.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8159980 0x7f02d80bc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:56.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80bd250 con 0x7f02d8153bb0 2023-10-30T13:40:56.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80bd440 con 0x7f02d806c420 2023-10-30T13:40:56.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80bd720 con 0x7f02d8159980 2023-10-30T13:40:56.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8153bb0 0x7f02d8150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:56.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8153bb0 0x7f02d8150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37002/0 (socket says 172.21.15.31:37002) 2023-10-30T13:40:56.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 -- 172.21.15.31:0/428487605 learned_addr learned my addr 172.21.15.31:0/428487605 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:56.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02d77fe640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8159980 0x7f02d80bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:56.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02d6ffd640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02d806c420 0x7f02d81ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:56.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 -- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8159980 msgr2=0x7f02d80bc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:56.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8159980 0x7f02d80bc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 -- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02d806c420 msgr2=0x7f02d81ee690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:56.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02d806c420 0x7f02d81ee690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 -- 172.21.15.31:0/428487605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d0009b30 con 0x7f02d8153bb0 2023-10-30T13:40:56.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cffff640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8153bb0 0x7f02d8150960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c3c0 tx=0x7f02c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:56.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cf7fe640 1 -- 172.21.15.31:0/428487605 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c00052d0 con 0x7f02d8153bb0 2023-10-30T13:40:56.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cf7fe640 1 -- 172.21.15.31:0/428487605 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0005430 con 0x7f02d8153bb0 2023-10-30T13:40:56.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cf7fe640 1 -- 172.21.15.31:0/428487605 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0003840 con 0x7f02d8153bb0 2023-10-30T13:40:56.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d80bda00 con 0x7f02d8153bb0 2023-10-30T13:40:56.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d80bdfc0 con 0x7f02d8153bb0 2023-10-30T13:40:56.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d804e170 con 0x7f02d8153bb0 2023-10-30T13:40:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.069+0000 7f02cf7fe640 1 -- 172.21.15.31:0/428487605 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02c00039a0 con 0x7f02d8153bb0 2023-10-30T13:40:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.073+0000 7f02cf7fe640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f02b80664d0 0x7f02b8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.073+0000 7f02cf7fe640 1 -- 172.21.15.31:0/428487605 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f02c0083550 con 0x7f02d8153bb0 2023-10-30T13:40:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.073+0000 7f02d6ffd640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f02b80664d0 0x7f02b8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.073+0000 7f02d6ffd640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f02b80664d0 0x7f02b8068990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f02c800b930 tx=0x7f02c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.077+0000 7f02cf7fe640 1 -- 172.21.15.31:0/428487605 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02c0050850 con 0x7f02d8153bb0 2023-10-30T13:40:56.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.261+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02d80c5be0 con 0x7f02d8153bb0 2023-10-30T13:40:56.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02cf7fe640 1 -- 172.21.15.31:0/428487605 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f02c0053e70 con 0x7f02d8153bb0 2023-10-30T13:40:56.265 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:56.265 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:09.794606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4276,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.33:6837/3402459191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":3402459191},{"type":"v1","addr":"172.21.15.33:6837","nonce":3402459191}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:56.266 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2023-10-30T13:40:56.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f02b80664d0 msgr2=0x7f02b8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:56.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f02b80664d0 0x7f02b8068990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f02c800b930 tx=0x7f02c8008040 comp rx=0 tx=0).stop 2023-10-30T13:40:56.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8153bb0 msgr2=0x7f02d8150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:56.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8153bb0 0x7f02d8150960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c3c0 tx=0x7f02c000c880 comp rx=0 tx=0).stop 2023-10-30T13:40:56.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 shutdown_connections 2023-10-30T13:40:56.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f02b80664d0 0x7f02b8068990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f02d8159980 0x7f02d80bc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8153bb0 0x7f02d8150960 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 --2- 172.21.15.31:0/428487605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02d806c420 0x7f02d81ee690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:56.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 >> 172.21.15.31:0/428487605 conn(0x7f02d80cac70 msgr2=0x7f02d80c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:56.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 shutdown_connections 2023-10-30T13:40:56.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:56.265+0000 7f02dd52a640 1 -- 172.21.15.31:0/428487605 wait complete. 2023-10-30T13:40:56.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:40:58.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:58.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe9a19c4640 1 Processor -- start 2023-10-30T13:40:58.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe9a19c4640 1 -- start start 2023-10-30T13:40:58.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe9a19c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 0x7fe99c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe9a19c4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c0c2310 con 0x7fe99c150830 2023-10-30T13:40:58.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe9a19c4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c156710 con 0x7fe99c0cb7d0 2023-10-30T13:40:58.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe9a19c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c06bc10 con 0x7fe99c070520 2023-10-30T13:40:58.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe99c150830 0x7fe99c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:40896/0 (socket says 172.21.15.31:40896) 2023-10-30T13:40:58.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99affd640 1 -- 172.21.15.31:0/3981658193 learned_addr learned my addr 172.21.15.31:0/3981658193 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:40:58.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99b7fe640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 0x7fe99c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550335871 0 0) 0x7fe99c0c2310 con 0x7fe99c150830 2023-10-30T13:40:58.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe988005740 con 0x7fe99c150830 2023-10-30T13:40:58.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407544816 0 0) 0x7fe99c156710 con 0x7fe99c0cb7d0 2023-10-30T13:40:58.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe99c0c2310 con 0x7fe99c0cb7d0 2023-10-30T13:40:58.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 847652426 0 0) 0x7fe988005740 con 0x7fe99c150830 2023-10-30T13:40:58.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 msgr2=0x7fe99c0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:58.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 0x7fe99c0709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 >> v1:172.21.15.33:6789/0 conn(0x7fe99c0cb7d0 legacy=0x7fe99c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c0c4360 con 0x7fe99c150830 2023-10-30T13:40:58.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.441+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fe98c001680 con 0x7fe99c150830 2023-10-30T13:40:58.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 >> v1:172.21.15.31:6789/0 conn(0x7fe99c150830 legacy=0x7fe99c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 0x7fe988005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe9880069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe988006f20 0x7fe9880073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c06bc10 con 0x7fe9880062b0 2023-10-30T13:40:58.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98800dc80 con 0x7fe988006f20 2023-10-30T13:40:58.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98800de40 con 0x7fe99c070520 2023-10-30T13:40:58.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe993fff640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe9880069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99b7fe640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe988006f20 0x7fe9880073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99affd640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 0x7fe988005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe993fff640 1 -- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 msgr2=0x7fe988005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:58.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe993fff640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c070520 0x7fe988005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe993fff640 1 -- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe988006f20 msgr2=0x7fe9880073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:58.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe993fff640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe988006f20 0x7fe9880073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe993fff640 1 -- 172.21.15.31:0/3981658193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988005740 con 0x7fe9880062b0 2023-10-30T13:40:58.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe993fff640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe9880069e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe9840018d0 tx=0x7fe98401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1079f03b292df49b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:58.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9840a6040 con 0x7fe9880062b0 2023-10-30T13:40:58.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9840052f0 con 0x7fe9880062b0 2023-10-30T13:40:58.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.445+0000 7fe99a7fc640 1 -- 172.21.15.31:0/3981658193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984003600 con 0x7fe9880062b0 2023-10-30T13:40:58.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3981658193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 msgr2=0x7fe9880069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 --2- 172.21.15.31:0/3981658193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe9880069e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe9840018d0 tx=0x7fe98401c560 comp rx=0 tx=0).stop 2023-10-30T13:40:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99b7fe640 1 -- 172.21.15.31:0/3981658193 reap_dead start 2023-10-30T13:40:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3981658193 shutdown_connections 2023-10-30T13:40:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3981658193 >> 172.21.15.31:0/3981658193 conn(0x7fe99c0cac70 msgr2=0x7fe99c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3981658193 shutdown_connections 2023-10-30T13:40:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3981658193 wait complete. 2023-10-30T13:40:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 Processor -- start 2023-10-30T13:40:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- start start 2023-10-30T13:40:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe99c14b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe99c1eeb20 0x7fe99c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c149f90 0x7fe99c144410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c1500c0 con 0x7fe9880062b0 2023-10-30T13:40:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c14af10 con 0x7fe99c1eeb20 2023-10-30T13:40:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe9a19c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c14a570 con 0x7fe99c149f90 2023-10-30T13:40:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe99c14b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe99c14b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37048/0 (socket says 172.21.15.31:37048) 2023-10-30T13:40:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 -- 172.21.15.31:0/3656249049 learned_addr learned my addr 172.21.15.31:0/3656249049 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe993fff640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe99c1eeb20 0x7fe99c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99b7fe640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c149f90 0x7fe99c144410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 -- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c149f90 msgr2=0x7fe99c144410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c149f90 0x7fe99c144410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 -- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe99c1eeb20 msgr2=0x7fe99c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe99c1eeb20 0x7fe99c157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.449+0000 7fe99affd640 1 -- 172.21.15.31:0/3656249049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c14b8c0 con 0x7fe9880062b0 2023-10-30T13:40:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe99affd640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe99c14b380 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe98c011970 tx=0x7fe98c011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe998ff9640 1 -- 172.21.15.31:0/3656249049 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c00fc20 con 0x7fe9880062b0 2023-10-30T13:40:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe998ff9640 1 -- 172.21.15.31:0/3656249049 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c00fd80 con 0x7fe9880062b0 2023-10-30T13:40:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe998ff9640 1 -- 172.21.15.31:0/3656249049 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c015610 con 0x7fe9880062b0 2023-10-30T13:40:58.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99c15a520 con 0x7fe9880062b0 2023-10-30T13:40:58.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c14a830 con 0x7fe9880062b0 2023-10-30T13:40:58.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99c066510 con 0x7fe9880062b0 2023-10-30T13:40:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe998ff9640 1 -- 172.21.15.31:0/3656249049 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe98c015770 con 0x7fe9880062b0 2023-10-30T13:40:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe998ff9640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe97c0664a0 0x7fe97c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe998ff9640 1 -- 172.21.15.31:0/3656249049 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fe98c0874c0 con 0x7fe9880062b0 2023-10-30T13:40:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.453+0000 7fe993fff640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe97c0664a0 0x7fe97c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.457+0000 7fe993fff640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe97c0664a0 0x7fe97c068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9840ac000 tx=0x7fe9840a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.457+0000 7fe998ff9640 1 -- 172.21.15.31:0/3656249049 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe98c054730 con 0x7fe9880062b0 2023-10-30T13:40:58.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.645+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe99c04e170 con 0x7fe9880062b0 2023-10-30T13:40:58.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.645+0000 7fe998ff9640 1 -- 172.21.15.31:0/3656249049 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fe98c057d50 con 0x7fe9880062b0 2023-10-30T13:40:58.647 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:58.648 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:56.840836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:58.648 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2023-10-30T13:40:58.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe97c0664a0 msgr2=0x7fe97c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe97c0664a0 0x7fe97c068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9840ac000 tx=0x7fe9840a7000 comp rx=0 tx=0).stop 2023-10-30T13:40:58.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 msgr2=0x7fe99c14b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe99c14b380 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe98c011970 tx=0x7fe98c011e30 comp rx=0 tx=0).stop 2023-10-30T13:40:58.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 shutdown_connections 2023-10-30T13:40:58.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe97c0664a0 0x7fe97c068960 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99c149f90 0x7fe99c144410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe99c1eeb20 0x7fe99c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 --2- 172.21.15.31:0/3656249049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9880062b0 0x7fe99c14b380 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 >> 172.21.15.31:0/3656249049 conn(0x7fe99c0cac70 msgr2=0x7fe99c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:58.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 shutdown_connections 2023-10-30T13:40:58.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.649+0000 7fe9a19c4640 1 -- 172.21.15.31:0/3656249049 wait complete. 2023-10-30T13:40:58.668 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T13:40:58.668 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:40:58.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.809+0000 7f357a103640 1 Processor -- start 2023-10-30T13:40:58.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.809+0000 7f357a103640 1 -- start start 2023-10-30T13:40:58.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f357a103640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f357415ac70 0x7f357415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f357a103640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f357406aeb0 0x7f357406b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f357a103640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357415dcc0 con 0x7f35740c8eb0 2023-10-30T13:40:58.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f357a103640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3574159800 con 0x7f357406aeb0 2023-10-30T13:40:58.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f357a103640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3574159d80 con 0x7f357415ac70 2023-10-30T13:40:58.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3579902640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f357406aeb0 0x7f357406b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3579902640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f357406aeb0 0x7f357406b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37050/0 (socket says 172.21.15.31:37050) 2023-10-30T13:40:58.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3579902640 1 -- 172.21.15.31:0/1130912000 learned_addr learned my addr 172.21.15.31:0/1130912000 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:58.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3578900640 1 --2- 172.21.15.31:0/1130912000 >> v2:172.21.15.33:3300/0 conn(0x7f357415ac70 0x7f357415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3579902640 1 -- 172.21.15.31:0/1130912000 >> v2:172.21.15.33:3300/0 conn(0x7f357415ac70 msgr2=0x7f357415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:58.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3579902640 1 --2- 172.21.15.31:0/1130912000 >> v2:172.21.15.33:3300/0 conn(0x7f357415ac70 0x7f357415cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3579902640 1 -- 172.21.15.31:0/1130912000 >> v1:172.21.15.33:6789/0 conn(0x7f35740c8eb0 legacy=0x7f357415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.813+0000 7f3579902640 1 -- 172.21.15.31:0/1130912000 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357415d420 con 0x7f357406aeb0 2023-10-30T13:40:58.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f3579902640 1 --2- 172.21.15.31:0/1130912000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f357406aeb0 0x7f357406b300 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f35740b4130 tx=0x7f356c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d181a0ab64b39b77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:58.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f356bfff640 1 -- 172.21.15.31:0/1130912000 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0af400 con 0x7f357406aeb0 2023-10-30T13:40:58.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f356bfff640 1 -- 172.21.15.31:0/1130912000 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c0ab050 con 0x7f357406aeb0 2023-10-30T13:40:58.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f356bfff640 1 -- 172.21.15.31:0/1130912000 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0af780 con 0x7f357406aeb0 2023-10-30T13:40:58.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 -- 172.21.15.31:0/1130912000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f357406aeb0 msgr2=0x7f357406b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 --2- 172.21.15.31:0/1130912000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f357406aeb0 0x7f357406b300 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f35740b4130 tx=0x7f356c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 -- 172.21.15.31:0/1130912000 shutdown_connections 2023-10-30T13:40:58.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 --2- 172.21.15.31:0/1130912000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f357406aeb0 0x7f357406b300 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 --2- 172.21.15.31:0/1130912000 >> v2:172.21.15.33:3300/0 conn(0x7f357415ac70 0x7f357415cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 -- 172.21.15.31:0/1130912000 >> 172.21.15.31:0/1130912000 conn(0x7f35740cef00 msgr2=0x7f35740cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:58.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 -- 172.21.15.31:0/1130912000 shutdown_connections 2023-10-30T13:40:58.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 -- 172.21.15.31:0/1130912000 wait complete. 2023-10-30T13:40:58.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 Processor -- start 2023-10-30T13:40:58.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 -- start start 2023-10-30T13:40:58.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.817+0000 7f357a103640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f357406aeb0 0x7f35740c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35740c8eb0 0x7f357414d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f357415ac70 0x7f357414d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35741f2b90 con 0x7f35740c8eb0 2023-10-30T13:40:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35741eeb90 con 0x7f357406aeb0 2023-10-30T13:40:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35741f30c0 con 0x7f357415ac70 2023-10-30T13:40:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35740c8eb0 0x7f357414d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35740c8eb0 0x7f357414d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37062/0 (socket says 172.21.15.31:37062) 2023-10-30T13:40:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 -- 172.21.15.31:0/1814606859 learned_addr learned my addr 172.21.15.31:0/1814606859 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:40:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3579101640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f357406aeb0 0x7f35740c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3579902640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f357415ac70 0x7f357414d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 -- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f357415ac70 msgr2=0x7f357414d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f357415ac70 0x7f357414d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 -- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f357406aeb0 msgr2=0x7f35740c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f357406aeb0 0x7f35740c5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 -- 172.21.15.31:0/1814606859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c009b30 con 0x7f35740c8eb0 2023-10-30T13:40:58.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3579101640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f357406aeb0 0x7f35740c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:58.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3578900640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35740c8eb0 0x7f357414d180 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f356400e870 tx=0x7f356400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:58.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3569ffb640 1 -- 172.21.15.31:0/1814606859 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564009510 con 0x7f35740c8eb0 2023-10-30T13:40:58.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3569ffb640 1 -- 172.21.15.31:0/1814606859 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356400eec0 con 0x7f35740c8eb0 2023-10-30T13:40:58.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3569ffb640 1 -- 172.21.15.31:0/1814606859 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356400cb40 con 0x7f35740c8eb0 2023-10-30T13:40:58.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3574155e30 con 0x7f35740c8eb0 2023-10-30T13:40:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35741560d0 con 0x7f35740c8eb0 2023-10-30T13:40:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357404e170 con 0x7f35740c8eb0 2023-10-30T13:40:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.821+0000 7f3569ffb640 1 -- 172.21.15.31:0/1814606859 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f356400cca0 con 0x7f35740c8eb0 2023-10-30T13:40:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.825+0000 7f3569ffb640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35580664d0 0x7f3558068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.825+0000 7f3569ffb640 1 -- 172.21.15.31:0/1814606859 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f3564084470 con 0x7f35740c8eb0 2023-10-30T13:40:58.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.825+0000 7f3579101640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35580664d0 0x7f3558068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:58.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.825+0000 7f3579101640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35580664d0 0x7f3558068990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3570000f70 tx=0x7f3570009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:58.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:58.829+0000 7f3569ffb640 1 -- 172.21.15.31:0/1814606859 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3564051770 con 0x7f35740c8eb0 2023-10-30T13:40:59.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.021+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3574153b60 con 0x7f35740c8eb0 2023-10-30T13:40:59.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.021+0000 7f3569ffb640 1 -- 172.21.15.31:0/1814606859 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f3564054d90 con 0x7f35740c8eb0 2023-10-30T13:40:59.023 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:40:59.024 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:56.840836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:59.024 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2023-10-30T13:40:59.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35580664d0 msgr2=0x7f3558068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:59.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35580664d0 0x7f3558068990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3570000f70 tx=0x7f3570009040 comp rx=0 tx=0).stop 2023-10-30T13:40:59.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35740c8eb0 msgr2=0x7f357414d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:59.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35740c8eb0 0x7f357414d180 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f356400e870 tx=0x7f356400ec70 comp rx=0 tx=0).stop 2023-10-30T13:40:59.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 shutdown_connections 2023-10-30T13:40:59.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35580664d0 0x7f3558068990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:59.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f357415ac70 0x7f357414d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:59.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35740c8eb0 0x7f357414d180 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:59.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 --2- 172.21.15.31:0/1814606859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f357406aeb0 0x7f35740c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:59.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 >> 172.21.15.31:0/1814606859 conn(0x7f35740cef00 msgr2=0x7f3574157bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:59.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 shutdown_connections 2023-10-30T13:40:59.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:40:59.025+0000 7f357a103640 1 -- 172.21.15.31:0/1814606859 wait complete. 2023-10-30T13:40:59.045 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:40:56.840836+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:41:01.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:01.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 Processor -- start 2023-10-30T13:41:01.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 -- start start 2023-10-30T13:41:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f193c0c62b0 0x7f193c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f193c0cb180 0x7f193c06ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c159cf0 con 0x7f193c0c62b0 2023-10-30T13:41:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c06aa30 con 0x7f193c0cb180 2023-10-30T13:41:01.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941874640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c15a770 con 0x7f193c153b50 2023-10-30T13:41:01.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f193c0c62b0 0x7f193c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:01.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f193c0c62b0 0x7f193c0ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37074/0 (socket says 172.21.15.31:37074) 2023-10-30T13:41:01.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 -- 172.21.15.31:0/2408325012 learned_addr learned my addr 172.21.15.31:0/2408325012 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:01.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1941073640 1 --2- 172.21.15.31:0/2408325012 >> v2:172.21.15.33:3300/0 conn(0x7f193c0cb180 0x7f193c06ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f1940872640 1 --2- 172.21.15.31:0/2408325012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 -- 172.21.15.31:0/2408325012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 msgr2=0x7f193c1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 --2- 172.21.15.31:0/2408325012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 -- 172.21.15.31:0/2408325012 >> v2:172.21.15.33:3300/0 conn(0x7f193c0cb180 msgr2=0x7f193c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 --2- 172.21.15.31:0/2408325012 >> v2:172.21.15.33:3300/0 conn(0x7f193c0cb180 0x7f193c06ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.201+0000 7f193bfff640 1 -- 172.21.15.31:0/2408325012 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c0c9d40 con 0x7f193c0c62b0 2023-10-30T13:41:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f193bfff640 1 --2- 172.21.15.31:0/2408325012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0c62b0 0x7f193c0ca1d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f193c06ad60 tx=0x7f192801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75f34e41998aec7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f193b7fe640 1 -- 172.21.15.31:0/2408325012 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19280ab600 con 0x7f193c0c62b0 2023-10-30T13:41:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f193b7fe640 1 -- 172.21.15.31:0/2408325012 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19280a7050 con 0x7f193c0c62b0 2023-10-30T13:41:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f193b7fe640 1 -- 172.21.15.31:0/2408325012 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19280ab980 con 0x7f193c0c62b0 2023-10-30T13:41:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 -- 172.21.15.31:0/2408325012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0c62b0 msgr2=0x7f193c0ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 --2- 172.21.15.31:0/2408325012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0c62b0 0x7f193c0ca1d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f193c06ad60 tx=0x7f192801c500 comp rx=0 tx=0).stop 2023-10-30T13:41:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 -- 172.21.15.31:0/2408325012 shutdown_connections 2023-10-30T13:41:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 --2- 172.21.15.31:0/2408325012 >> v2:172.21.15.33:3300/0 conn(0x7f193c0cb180 0x7f193c06ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 --2- 172.21.15.31:0/2408325012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0c62b0 0x7f193c0ca1d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 --2- 172.21.15.31:0/2408325012 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 -- 172.21.15.31:0/2408325012 >> 172.21.15.31:0/2408325012 conn(0x7f193c065a80 msgr2=0x7f193c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:01.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 -- 172.21.15.31:0/2408325012 shutdown_connections 2023-10-30T13:41:01.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.205+0000 7f1941874640 1 -- 172.21.15.31:0/2408325012 wait complete. 2023-10-30T13:41:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 Processor -- start 2023-10-30T13:41:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 -- start start 2023-10-30T13:41:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f193c0c62b0 0x7f193c144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0cb180 0x7f193c1f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c15b830 con 0x7f193c0cb180 2023-10-30T13:41:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c15bdf0 con 0x7f193c0c62b0 2023-10-30T13:41:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c149d60 con 0x7f193c153b50 2023-10-30T13:41:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0cb180 0x7f193c1f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0cb180 0x7f193c1f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37090/0 (socket says 172.21.15.31:37090) 2023-10-30T13:41:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 -- 172.21.15.31:0/807027152 learned_addr learned my addr 172.21.15.31:0/807027152 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941073640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1940872640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f193c0c62b0 0x7f193c144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 -- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 msgr2=0x7f193c1f5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1f5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 -- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f193c0c62b0 msgr2=0x7f193c144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f193c0c62b0 0x7f193c144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 -- 172.21.15.31:0/807027152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280094d0 con 0x7f193c0cb180 2023-10-30T13:41:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1940872640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f193c0c62b0 0x7f193c144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f193bfff640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0cb180 0x7f193c1f58b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f19280ae040 tx=0x7f192801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f19397fa640 1 -- 172.21.15.31:0/807027152 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19280b6420 con 0x7f193c0cb180 2023-10-30T13:41:01.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f19397fa640 1 -- 172.21.15.31:0/807027152 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19280b6580 con 0x7f193c0cb180 2023-10-30T13:41:01.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f19397fa640 1 -- 172.21.15.31:0/807027152 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19280b6910 con 0x7f193c0cb180 2023-10-30T13:41:01.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c15c6a0 con 0x7f193c0cb180 2023-10-30T13:41:01.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.209+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c15ba80 con 0x7f193c0cb180 2023-10-30T13:41:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.213+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c15bf60 con 0x7f193c0cb180 2023-10-30T13:41:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.213+0000 7f19397fa640 1 -- 172.21.15.31:0/807027152 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19280b2040 con 0x7f193c0cb180 2023-10-30T13:41:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.213+0000 7f19397fa640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19200664d0 0x7f1920068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.213+0000 7f19397fa640 1 -- 172.21.15.31:0/807027152 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f192811e2d0 con 0x7f193c0cb180 2023-10-30T13:41:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.213+0000 7f1940872640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19200664d0 0x7f1920068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.213+0000 7f1940872640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19200664d0 0x7f1920068990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f193000b9f0 tx=0x7f1930008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.217+0000 7f19397fa640 1 -- 172.21.15.31:0/807027152 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19280eb650 con 0x7f193c0cb180 2023-10-30T13:41:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f193c067540 con 0x7f193c0cb180 2023-10-30T13:41:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f19397fa640 1 -- 172.21.15.31:0/807027152 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f19280eec70 con 0x7f193c0cb180 2023-10-30T13:41:01.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:01.411 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:40:56.840836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2023-10-30T13:41:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19200664d0 msgr2=0x7f1920068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19200664d0 0x7f1920068990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f193000b9f0 tx=0x7f1930008040 comp rx=0 tx=0).stop 2023-10-30T13:41:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0cb180 msgr2=0x7f193c1f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0cb180 0x7f193c1f58b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f19280ae040 tx=0x7f192801ad40 comp rx=0 tx=0).stop 2023-10-30T13:41:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 shutdown_connections 2023-10-30T13:41:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f19200664d0 0x7f1920068990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f193c153b50 0x7f193c1f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c0cb180 0x7f193c1f58b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 --2- 172.21.15.31:0/807027152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f193c0c62b0 0x7f193c144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:01.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 >> 172.21.15.31:0/807027152 conn(0x7f193c065a80 msgr2=0x7f193c14ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:01.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 shutdown_connections 2023-10-30T13:41:01.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:01.409+0000 7f1941874640 1 -- 172.21.15.31:0/807027152 wait complete. 2023-10-30T13:41:01.431 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:03.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:03.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a61b0e640 1 Processor -- start 2023-10-30T13:41:03.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a61b0e640 1 -- start start 2023-10-30T13:41:03.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a61b0e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3a5c06b3b0 0x7f3a5c06b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a61b0e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5c0c7b00 con 0x7f3a5c153aa0 2023-10-30T13:41:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a61b0e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5c06bd20 con 0x7f3a5c15d840 2023-10-30T13:41:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a61b0e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5c070a50 con 0x7f3a5c06b3b0 2023-10-30T13:41:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f3a5c153aa0 0x7f3a5c0701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:40912/0 (socket says 172.21.15.31:40912) 2023-10-30T13:41:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5b7fe640 1 -- 172.21.15.31:0/842997909 learned_addr learned my addr 172.21.15.31:0/842997909 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5bfff640 1 --2- 172.21.15.31:0/842997909 >> v2:172.21.15.31:3300/0 conn(0x7f3a5c06b3b0 0x7f3a5c06b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5affd640 1 -- 172.21.15.31:0/842997909 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231818377 0 0) 0x7f3a5c0c7b00 con 0x7f3a5c153aa0 2023-10-30T13:41:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5affd640 1 -- 172.21.15.31:0/842997909 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a48003530 con 0x7f3a5c153aa0 2023-10-30T13:41:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5bfff640 1 -- 172.21.15.31:0/842997909 >> v1:172.21.15.33:6789/0 conn(0x7f3a5c15d840 legacy=0x7f3a5c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5bfff640 1 -- 172.21.15.31:0/842997909 >> v1:172.21.15.31:6789/0 conn(0x7f3a5c153aa0 legacy=0x7f3a5c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:03.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.589+0000 7f3a5bfff640 1 -- 172.21.15.31:0/842997909 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c0c2050 con 0x7f3a5c06b3b0 2023-10-30T13:41:03.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a5bfff640 1 --2- 172.21.15.31:0/842997909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c06b3b0 0x7f3a5c06b7e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009d90 tx=0x7f3a5401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972f5e2370dfc762 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:03.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a5affd640 1 -- 172.21.15.31:0/842997909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540af600 con 0x7f3a5c06b3b0 2023-10-30T13:41:03.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a5affd640 1 -- 172.21.15.31:0/842997909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a540ab050 con 0x7f3a5c06b3b0 2023-10-30T13:41:03.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a5affd640 1 -- 172.21.15.31:0/842997909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540af980 con 0x7f3a5c06b3b0 2023-10-30T13:41:03.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a61b0e640 1 -- 172.21.15.31:0/842997909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c06b3b0 msgr2=0x7f3a5c06b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:03.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/842997909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c06b3b0 0x7f3a5c06b7e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009d90 tx=0x7f3a5401f130 comp rx=0 tx=0).stop 2023-10-30T13:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a61b0e640 1 -- 172.21.15.31:0/842997909 shutdown_connections 2023-10-30T13:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/842997909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c06b3b0 0x7f3a5c06b7e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a61b0e640 1 -- 172.21.15.31:0/842997909 >> 172.21.15.31:0/842997909 conn(0x7f3a5c071fb0 msgr2=0x7f3a5c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a61b0e640 1 -- 172.21.15.31:0/842997909 shutdown_connections 2023-10-30T13:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.593+0000 7f3a61b0e640 1 -- 172.21.15.31:0/842997909 wait complete. 2023-10-30T13:41:03.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 Processor -- start 2023-10-30T13:41:03.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 -- start start 2023-10-30T13:41:03.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a5c06b3b0 0x7f3a5c1ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:03.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c153aa0 0x7f3a5c14f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:03.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a5c15d840 0x7f3a5c0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:03.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c156d20 con 0x7f3a5c153aa0 2023-10-30T13:41:03.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0c5790 con 0x7f3a5c06b3b0 2023-10-30T13:41:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c1eee00 con 0x7f3a5c15d840 2023-10-30T13:41:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c153aa0 0x7f3a5c14f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c153aa0 0x7f3a5c14f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37108/0 (socket says 172.21.15.31:37108) 2023-10-30T13:41:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 -- 172.21.15.31:0/2327080023 learned_addr learned my addr 172.21.15.31:0/2327080023 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a5b7fe640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a5c06b3b0 0x7f3a5c1ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a5bfff640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a5c15d840 0x7f3a5c0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 -- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a5c15d840 msgr2=0x7f3a5c0c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a5c15d840 0x7f3a5c0c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 -- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a5c06b3b0 msgr2=0x7f3a5c1ef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a5c06b3b0 0x7f3a5c1ef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 -- 172.21.15.31:0/2327080023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54009a70 con 0x7f3a5c153aa0 2023-10-30T13:41:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a53fff640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c153aa0 0x7f3a5c14f990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c530 tx=0x7f3a4400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a58ff9640 1 -- 172.21.15.31:0/2327080023 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44005220 con 0x7f3a5c153aa0 2023-10-30T13:41:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a58ff9640 1 -- 172.21.15.31:0/2327080023 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44005380 con 0x7f3a5c153aa0 2023-10-30T13:41:03.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a58ff9640 1 -- 172.21.15.31:0/2327080023 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44007270 con 0x7f3a5c153aa0 2023-10-30T13:41:03.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c150020 con 0x7f3a5c153aa0 2023-10-30T13:41:03.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c1505e0 con 0x7f3a5c153aa0 2023-10-30T13:41:03.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.597+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5c04e170 con 0x7f3a5c153aa0 2023-10-30T13:41:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.601+0000 7f3a58ff9640 1 -- 172.21.15.31:0/2327080023 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a440073d0 con 0x7f3a5c153aa0 2023-10-30T13:41:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.601+0000 7f3a58ff9640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a3c0664a0 0x7f3a3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.601+0000 7f3a58ff9640 1 -- 172.21.15.31:0/2327080023 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f3a44083960 con 0x7f3a5c153aa0 2023-10-30T13:41:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.601+0000 7f3a5b7fe640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a3c0664a0 0x7f3a3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.601+0000 7f3a5b7fe640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a3c0664a0 0x7f3a3c068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0011f0 tx=0x7f3a4c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.605+0000 7f3a58ff9640 1 -- 172.21.15.31:0/2327080023 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a44050bd0 con 0x7f3a5c153aa0 2023-10-30T13:41:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.797+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a5c0cbc40 con 0x7f3a5c153aa0 2023-10-30T13:41:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.797+0000 7f3a58ff9640 1 -- 172.21.15.31:0/2327080023 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f3a440541f0 con 0x7f3a5c153aa0 2023-10-30T13:41:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:03.799 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:02.075781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:resolve","state_seq":23,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:03.800 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2023-10-30T13:41:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a3c0664a0 msgr2=0x7f3a3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a3c0664a0 0x7f3a3c068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0011f0 tx=0x7f3a4c009040 comp rx=0 tx=0).stop 2023-10-30T13:41:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c153aa0 msgr2=0x7f3a5c14f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c153aa0 0x7f3a5c14f990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c530 tx=0x7f3a4400c9f0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 shutdown_connections 2023-10-30T13:41:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3a3c0664a0 0x7f3a3c068960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3a5c15d840 0x7f3a5c0c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a5c153aa0 0x7f3a5c14f990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 --2- 172.21.15.31:0/2327080023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a5c06b3b0 0x7f3a5c1ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 >> 172.21.15.31:0/2327080023 conn(0x7f3a5c071fb0 msgr2=0x7f3a5c0cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:03.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 shutdown_connections 2023-10-30T13:41:03.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:03.801+0000 7f3a61b0e640 1 -- 172.21.15.31:0/2327080023 wait complete. 2023-10-30T13:41:03.820 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:05.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:05.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.973+0000 7f55ceb9d640 1 Processor -- start 2023-10-30T13:41:05.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55ceb9d640 1 -- start start 2023-10-30T13:41:05.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55ceb9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f55c8153c80 0x7f55c815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:05.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55ceb9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f55c80c42a0 0x7f55c80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:05.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55ceb9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c8159f20 con 0x7f55c80c4a10 2023-10-30T13:41:05.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55ceb9d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c815dec0 con 0x7f55c80c42a0 2023-10-30T13:41:05.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55ceb9d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c81599a0 con 0x7f55c8153c80 2023-10-30T13:41:05.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cd13a640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f55c80c42a0 0x7f55c80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:05.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cd13a640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f55c80c42a0 0x7f55c80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60822/0 (socket says 172.21.15.31:60822) 2023-10-30T13:41:05.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cd13a640 1 -- 172.21.15.31:0/997591502 learned_addr learned my addr 172.21.15.31:0/997591502 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:05.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cc939640 1 --2- 172.21.15.31:0/997591502 >> v2:172.21.15.33:3300/0 conn(0x7f55c8153c80 0x7f55c815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:05.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cd13a640 1 -- 172.21.15.31:0/997591502 >> v2:172.21.15.33:3300/0 conn(0x7f55c8153c80 msgr2=0x7f55c815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:05.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cd13a640 1 --2- 172.21.15.31:0/997591502 >> v2:172.21.15.33:3300/0 conn(0x7f55c8153c80 0x7f55c815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cd13a640 1 -- 172.21.15.31:0/997591502 >> v1:172.21.15.33:6789/0 conn(0x7f55c80c4a10 legacy=0x7f55c815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:05.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.977+0000 7f55cd13a640 1 -- 172.21.15.31:0/997591502 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c815d620 con 0x7f55c80c42a0 2023-10-30T13:41:05.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55cd13a640 1 --2- 172.21.15.31:0/997591502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c80ca1e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55c80d00f0 tx=0x7f55c001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df9f5e71317f26a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:05.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55bf7fe640 1 -- 172.21.15.31:0/997591502 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c00af400 con 0x7f55c80c42a0 2023-10-30T13:41:05.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55bf7fe640 1 -- 172.21.15.31:0/997591502 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c00ab050 con 0x7f55c80c42a0 2023-10-30T13:41:05.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55bf7fe640 1 -- 172.21.15.31:0/997591502 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c00af780 con 0x7f55c80c42a0 2023-10-30T13:41:05.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 -- 172.21.15.31:0/997591502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 msgr2=0x7f55c80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:05.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/997591502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c80ca1e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55c80d00f0 tx=0x7f55c001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 -- 172.21.15.31:0/997591502 shutdown_connections 2023-10-30T13:41:05.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/997591502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c80ca1e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/997591502 >> v2:172.21.15.33:3300/0 conn(0x7f55c8153c80 0x7f55c815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 -- 172.21.15.31:0/997591502 >> 172.21.15.31:0/997591502 conn(0x7f55c80715c0 msgr2=0x7f55c80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:05.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 -- 172.21.15.31:0/997591502 shutdown_connections 2023-10-30T13:41:05.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 -- 172.21.15.31:0/997591502 wait complete. 2023-10-30T13:41:05.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 Processor -- start 2023-10-30T13:41:05.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.981+0000 7f55ceb9d640 1 -- start start 2023-10-30T13:41:05.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c8152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:05.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c80c4a10 0x7f55c8154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:05.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f55c8153c80 0x7f55c81f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:05.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c814a4d0 con 0x7f55c80c42a0 2023-10-30T13:41:05.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c80cef00 con 0x7f55c80c4a10 2023-10-30T13:41:05.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c8150270 con 0x7f55c8153c80 2023-10-30T13:41:05.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c8152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:05.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c8152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60836/0 (socket says 172.21.15.31:60836) 2023-10-30T13:41:05.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 -- 172.21.15.31:0/3718762894 learned_addr learned my addr 172.21.15.31:0/3718762894 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55bffff640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c80c4a10 0x7f55c8154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cd13a640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f55c8153c80 0x7f55c81f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 -- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f55c8153c80 msgr2=0x7f55c81f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f55c8153c80 0x7f55c81f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 -- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c80c4a10 msgr2=0x7f55c8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c80c4a10 0x7f55c8154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 -- 172.21.15.31:0/3718762894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c0009b30 con 0x7f55c80c42a0 2023-10-30T13:41:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cd13a640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f55c8153c80 0x7f55c81f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55cc939640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c8152950 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f55c400e870 tx=0x7f55c400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55bd7fa640 1 -- 172.21.15.31:0/3718762894 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4009510 con 0x7f55c80c42a0 2023-10-30T13:41:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55bd7fa640 1 -- 172.21.15.31:0/3718762894 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c400eec0 con 0x7f55c80c42a0 2023-10-30T13:41:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c80cebd0 con 0x7f55c80c42a0 2023-10-30T13:41:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55bd7fa640 1 -- 172.21.15.31:0/3718762894 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c400cb40 con 0x7f55c80c42a0 2023-10-30T13:41:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c80cf790 con 0x7f55c80c42a0 2023-10-30T13:41:05.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.985+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c80cb660 con 0x7f55c80c42a0 2023-10-30T13:41:05.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.989+0000 7f55bd7fa640 1 -- 172.21.15.31:0/3718762894 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f55c400cca0 con 0x7f55c80c42a0 2023-10-30T13:41:05.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.989+0000 7f55bd7fa640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f55b40664a0 0x7f55b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:05.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.989+0000 7f55bd7fa640 1 -- 172.21.15.31:0/3718762894 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f55c40844e0 con 0x7f55c80c42a0 2023-10-30T13:41:05.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.989+0000 7f55bffff640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f55b40664a0 0x7f55b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:05.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.989+0000 7f55bffff640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f55b40664a0 0x7f55b4068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f55c80b19f0 tx=0x7f55b8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:05.993+0000 7f55bd7fa640 1 -- 172.21.15.31:0/3718762894 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f55c4051770 con 0x7f55c80c42a0 2023-10-30T13:41:06.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.185+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55c80cb8d0 con 0x7f55c80c42a0 2023-10-30T13:41:06.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.185+0000 7f55bd7fa640 1 -- 172.21.15.31:0/3718762894 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f55c4054d90 con 0x7f55c80c42a0 2023-10-30T13:41:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:06.189 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:02.075781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:resolve","state_seq":23,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:06.190 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2023-10-30T13:41:06.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f55b40664a0 msgr2=0x7f55b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:06.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f55b40664a0 0x7f55b4068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f55c80b19f0 tx=0x7f55b8009040 comp rx=0 tx=0).stop 2023-10-30T13:41:06.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 msgr2=0x7f55c8152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:06.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c8152950 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f55c400e870 tx=0x7f55c400ec70 comp rx=0 tx=0).stop 2023-10-30T13:41:06.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 shutdown_connections 2023-10-30T13:41:06.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f55b40664a0 0x7f55b4068960 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:06.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f55c8153c80 0x7f55c81f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:06.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c80c4a10 0x7f55c8154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:06.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 --2- 172.21.15.31:0/3718762894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55c80c42a0 0x7f55c8152950 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:06.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 >> 172.21.15.31:0/3718762894 conn(0x7f55c80715c0 msgr2=0x7f55c8158790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:06.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 shutdown_connections 2023-10-30T13:41:06.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:06.189+0000 7f55ceb9d640 1 -- 172.21.15.31:0/3718762894 wait complete. 2023-10-30T13:41:06.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:08.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2fa09640 1 Processor -- start 2023-10-30T13:41:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2fa09640 1 -- start start 2023-10-30T13:41:08.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2fa09640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fdf28153c80 0x7fdf2815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:08.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2fa09640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf28159f20 con 0x7fdf280c42a0 2023-10-30T13:41:08.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2fa09640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2815d620 con 0x7fdf280c4a10 2023-10-30T13:41:08.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2fa09640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2815dec0 con 0x7fdf28153c80 2023-10-30T13:41:08.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2d7a5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fdf28153c80 0x7fdf2815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:08.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2dfa6640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fdf280c42a0 0x7fdf280ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48646/0 (socket says 172.21.15.31:48646) 2023-10-30T13:41:08.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2dfa6640 1 -- 172.21.15.31:0/3085642456 learned_addr learned my addr 172.21.15.31:0/3085642456 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:08.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2d7a5640 1 -- 172.21.15.31:0/3085642456 >> v1:172.21.15.33:6789/0 conn(0x7fdf280c4a10 legacy=0x7fdf2815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:08.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2d7a5640 1 -- 172.21.15.31:0/3085642456 >> v1:172.21.15.31:6789/0 conn(0x7fdf280c42a0 legacy=0x7fdf280ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:08.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf1ffff640 1 -- 172.21.15.31:0/3085642456 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222607933 0 0) 0x7fdf28159f20 con 0x7fdf280c42a0 2023-10-30T13:41:08.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.361+0000 7fdf2d7a5640 1 -- 172.21.15.31:0/3085642456 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2815d900 con 0x7fdf28153c80 2023-10-30T13:41:08.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2d7a5640 1 --2- 172.21.15.31:0/3085642456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf28153c80 0x7fdf2815cf90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdf240097c0 tx=0x7fdf2401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9222095da8ca664 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:08.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf1ffff640 1 -- 172.21.15.31:0/3085642456 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf240a8970 con 0x7fdf28153c80 2023-10-30T13:41:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf1ffff640 1 -- 172.21.15.31:0/3085642456 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf240a8ad0 con 0x7fdf28153c80 2023-10-30T13:41:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf1ffff640 1 -- 172.21.15.31:0/3085642456 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf240a8e60 con 0x7fdf28153c80 2023-10-30T13:41:08.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2fa09640 1 -- 172.21.15.31:0/3085642456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf28153c80 msgr2=0x7fdf2815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:08.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/3085642456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf28153c80 0x7fdf2815cf90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdf240097c0 tx=0x7fdf2401ec70 comp rx=0 tx=0).stop 2023-10-30T13:41:08.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2fa09640 1 -- 172.21.15.31:0/3085642456 shutdown_connections 2023-10-30T13:41:08.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/3085642456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf28153c80 0x7fdf2815cf90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:08.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2fa09640 1 -- 172.21.15.31:0/3085642456 >> 172.21.15.31:0/3085642456 conn(0x7fdf280715c0 msgr2=0x7fdf280719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:08.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2fa09640 1 -- 172.21.15.31:0/3085642456 shutdown_connections 2023-10-30T13:41:08.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.365+0000 7fdf2fa09640 1 -- 172.21.15.31:0/3085642456 wait complete. 2023-10-30T13:41:08.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 Processor -- start 2023-10-30T13:41:08.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 -- start start 2023-10-30T13:41:08.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf280c42a0 0x7fdf28149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf280c4a10 0x7fdf28158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdf28153c80 0x7fdf28158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280c9670 con 0x7fdf280c4a10 2023-10-30T13:41:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280c85e0 con 0x7fdf280c42a0 2023-10-30T13:41:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280c89f0 con 0x7fdf28153c80 2023-10-30T13:41:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf280c4a10 0x7fdf28158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf280c4a10 0x7fdf28158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60854/0 (socket says 172.21.15.31:60854) 2023-10-30T13:41:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 -- 172.21.15.31:0/4032410954 learned_addr learned my addr 172.21.15.31:0/4032410954 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2d7a5640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf280c42a0 0x7fdf28149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2dfa6640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdf28153c80 0x7fdf28158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 -- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdf28153c80 msgr2=0x7fdf28158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdf28153c80 0x7fdf28158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 -- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf280c42a0 msgr2=0x7fdf28149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf280c42a0 0x7fdf28149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 -- 172.21.15.31:0/4032410954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf24009420 con 0x7fdf280c4a10 2023-10-30T13:41:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2dfa6640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdf28153c80 0x7fdf28158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2cfa4640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf280c4a10 0x7fdf28158860 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800c4d0 tx=0x7fdf1800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf1dffb640 1 -- 172.21.15.31:0/4032410954 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf180052d0 con 0x7fdf280c4a10 2023-10-30T13:41:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf1dffb640 1 -- 172.21.15.31:0/4032410954 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf18005430 con 0x7fdf280c4a10 2023-10-30T13:41:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf280c8cd0 con 0x7fdf280c4a10 2023-10-30T13:41:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf1dffb640 1 -- 172.21.15.31:0/4032410954 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18012830 con 0x7fdf280c4a10 2023-10-30T13:41:08.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf280c9290 con 0x7fdf280c4a10 2023-10-30T13:41:08.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.369+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf280c4710 con 0x7fdf280c4a10 2023-10-30T13:41:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.373+0000 7fdf1dffb640 1 -- 172.21.15.31:0/4032410954 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf18012990 con 0x7fdf280c4a10 2023-10-30T13:41:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.373+0000 7fdf1dffb640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf0c0664a0 0x7fdf0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.373+0000 7fdf1dffb640 1 -- 172.21.15.31:0/4032410954 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fdf18083a70 con 0x7fdf280c4a10 2023-10-30T13:41:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.373+0000 7fdf2d7a5640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf0c0664a0 0x7fdf0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.373+0000 7fdf2d7a5640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf0c0664a0 0x7fdf0c068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdf240b2000 tx=0x7fdf24008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.377+0000 7fdf1dffb640 1 -- 172.21.15.31:0/4032410954 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf280c4710 con 0x7fdf280c4a10 2023-10-30T13:41:08.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.565+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf280c4710 con 0x7fdf280c4a10 2023-10-30T13:41:08.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.565+0000 7fdf1dffb640 1 -- 172.21.15.31:0/4032410954 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fdf280c4710 con 0x7fdf280c4a10 2023-10-30T13:41:08.567 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:08.567 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:08.010558+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:08.567 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 33 2023-10-30T13:41:08.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf0c0664a0 msgr2=0x7fdf0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:08.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf0c0664a0 0x7fdf0c068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdf240b2000 tx=0x7fdf24008040 comp rx=0 tx=0).stop 2023-10-30T13:41:08.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf280c4a10 msgr2=0x7fdf28158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:08.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf280c4a10 0x7fdf28158860 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800c4d0 tx=0x7fdf1800c990 comp rx=0 tx=0).stop 2023-10-30T13:41:08.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 shutdown_connections 2023-10-30T13:41:08.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdf0c0664a0 0x7fdf0c068960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:08.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdf28153c80 0x7fdf28158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:08.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf280c4a10 0x7fdf28158860 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:08.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 --2- 172.21.15.31:0/4032410954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf280c42a0 0x7fdf28149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:08.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 >> 172.21.15.31:0/4032410954 conn(0x7fdf280715c0 msgr2=0x7fdf280cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:08.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 shutdown_connections 2023-10-30T13:41:08.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:08.569+0000 7fdf2fa09640 1 -- 172.21.15.31:0/4032410954 wait complete. 2023-10-30T13:41:08.588 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:10.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.741+0000 7f09a4c68640 1 Processor -- start 2023-10-30T13:41:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.741+0000 7f09a4c68640 1 -- start start 2023-10-30T13:41:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f09a4c68640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f09a006c420 0x7f09a00c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f09a4c68640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f09a00c7170 0x7f09a0159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f09a4c68640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a00c5f80 con 0x7f09a0153bb0 2023-10-30T13:41:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f09a4c68640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a00c7580 con 0x7f09a00c7170 2023-10-30T13:41:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f09a4c68640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a00c5690 con 0x7f09a006c420 2023-10-30T13:41:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f099ed9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f09a00c7170 0x7f09a0159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f099ed9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f09a00c7170 0x7f09a0159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60868/0 (socket says 172.21.15.31:60868) 2023-10-30T13:41:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f099ed9d640 1 -- 172.21.15.31:0/976337089 learned_addr learned my addr 172.21.15.31:0/976337089 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f0997fff640 1 --2- 172.21.15.31:0/976337089 >> v2:172.21.15.33:3300/0 conn(0x7f09a006c420 0x7f09a00c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f099ed9d640 1 -- 172.21.15.31:0/976337089 >> v2:172.21.15.33:3300/0 conn(0x7f09a006c420 msgr2=0x7f09a00c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f099ed9d640 1 --2- 172.21.15.31:0/976337089 >> v2:172.21.15.33:3300/0 conn(0x7f09a006c420 0x7f09a00c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f099ed9d640 1 -- 172.21.15.31:0/976337089 >> v1:172.21.15.33:6789/0 conn(0x7f09a0153bb0 legacy=0x7f09a00c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.745+0000 7f099ed9d640 1 -- 172.21.15.31:0/976337089 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a015a580 con 0x7f09a00c7170 2023-10-30T13:41:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f099ed9d640 1 --2- 172.21.15.31:0/976337089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a00c7170 0x7f09a0159980 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f09a006f800 tx=0x7f099801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb272d40257de34f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f099dd9b640 1 -- 172.21.15.31:0/976337089 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09980af400 con 0x7f09a00c7170 2023-10-30T13:41:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f099dd9b640 1 -- 172.21.15.31:0/976337089 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09980ab050 con 0x7f09a00c7170 2023-10-30T13:41:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f099dd9b640 1 -- 172.21.15.31:0/976337089 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09980af780 con 0x7f09a00c7170 2023-10-30T13:41:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 -- 172.21.15.31:0/976337089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a00c7170 msgr2=0x7f09a0159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 --2- 172.21.15.31:0/976337089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a00c7170 0x7f09a0159980 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f09a006f800 tx=0x7f099801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 -- 172.21.15.31:0/976337089 shutdown_connections 2023-10-30T13:41:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 --2- 172.21.15.31:0/976337089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a00c7170 0x7f09a0159980 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 --2- 172.21.15.31:0/976337089 >> v2:172.21.15.33:3300/0 conn(0x7f09a006c420 0x7f09a00c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 -- 172.21.15.31:0/976337089 >> 172.21.15.31:0/976337089 conn(0x7f09a00cac70 msgr2=0x7f09a00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 -- 172.21.15.31:0/976337089 shutdown_connections 2023-10-30T13:41:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 -- 172.21.15.31:0/976337089 wait complete. 2023-10-30T13:41:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 Processor -- start 2023-10-30T13:41:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.749+0000 7f09a4c68640 1 -- start start 2023-10-30T13:41:10.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a006c420 0x7f09a0070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:10.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a00c7170 0x7f09a01565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:10.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09a0153bb0 0x7f09a0154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:10.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00c4e20 con 0x7f09a006c420 2023-10-30T13:41:10.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00c3f70 con 0x7f09a00c7170 2023-10-30T13:41:10.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00c4210 con 0x7f09a0153bb0 2023-10-30T13:41:10.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a006c420 0x7f09a0070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:10.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a006c420 0x7f09a0070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60874/0 (socket says 172.21.15.31:60874) 2023-10-30T13:41:10.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 -- 172.21.15.31:0/2343479073 learned_addr learned my addr 172.21.15.31:0/2343479073 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:10.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f0997fff640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a00c7170 0x7f09a01565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:10.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099ed9d640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09a0153bb0 0x7f09a0154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:10.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 -- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09a0153bb0 msgr2=0x7f09a0154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:10.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09a0153bb0 0x7f09a0154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 -- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a00c7170 msgr2=0x7f09a01565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:10.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a00c7170 0x7f09a01565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 -- 172.21.15.31:0/2343479073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0998009b30 con 0x7f09a006c420 2023-10-30T13:41:10.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f0997fff640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a00c7170 0x7f09a01565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:10.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f099e59c640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a006c420 0x7f09a0070e90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f099000c500 tx=0x7f099000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:10.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f0996ffd640 1 -- 172.21.15.31:0/2343479073 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990005220 con 0x7f09a006c420 2023-10-30T13:41:10.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f0996ffd640 1 -- 172.21.15.31:0/2343479073 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0990005380 con 0x7f09a006c420 2023-10-30T13:41:10.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a00c44f0 con 0x7f09a006c420 2023-10-30T13:41:10.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f0996ffd640 1 -- 172.21.15.31:0/2343479073 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990007270 con 0x7f09a006c420 2023-10-30T13:41:10.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a00c4ab0 con 0x7f09a006c420 2023-10-30T13:41:10.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a004e170 con 0x7f09a006c420 2023-10-30T13:41:10.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.753+0000 7f0996ffd640 1 -- 172.21.15.31:0/2343479073 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09900073d0 con 0x7f09a006c420 2023-10-30T13:41:10.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.757+0000 7f0996ffd640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09800664d0 0x7f0980068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:10.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.757+0000 7f0996ffd640 1 -- 172.21.15.31:0/2343479073 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f0990083920 con 0x7f09a006c420 2023-10-30T13:41:10.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.757+0000 7f0997fff640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09800664d0 0x7f0980068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:10.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.757+0000 7f0997fff640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09800664d0 0x7f0980068990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f09a00c1dd0 tx=0x7f0988008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:10.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.761+0000 7f0996ffd640 1 -- 172.21.15.31:0/2343479073 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0990050ba0 con 0x7f09a006c420 2023-10-30T13:41:10.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.949+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09a006b730 con 0x7f09a006c420 2023-10-30T13:41:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.949+0000 7f0996ffd640 1 -- 172.21.15.31:0/2343479073 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f09900541c0 con 0x7f09a006c420 2023-10-30T13:41:10.952 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:10.952 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:08.010558+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:10.953 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 33 2023-10-30T13:41:10.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09800664d0 msgr2=0x7f0980068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:10.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09800664d0 0x7f0980068990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f09a00c1dd0 tx=0x7f0988008040 comp rx=0 tx=0).stop 2023-10-30T13:41:10.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a006c420 msgr2=0x7f09a0070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a006c420 0x7f09a0070e90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f099000c500 tx=0x7f099000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 shutdown_connections 2023-10-30T13:41:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f09800664d0 0x7f0980068990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f09a0153bb0 0x7f09a0154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a00c7170 0x7f09a01565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 --2- 172.21.15.31:0/2343479073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a006c420 0x7f09a0070e90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 >> 172.21.15.31:0/2343479073 conn(0x7f09a00cac70 msgr2=0x7f09a0070190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 shutdown_connections 2023-10-30T13:41:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:10.953+0000 7f09a4c68640 1 -- 172.21.15.31:0/2343479073 wait complete. 2023-10-30T13:41:10.972 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:41:08.010558+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:41:12.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:13.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe99526b640 1 Processor -- start 2023-10-30T13:41:13.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe99526b640 1 -- start start 2023-10-30T13:41:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe99526b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 0x7fe99015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe99526b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe990159f20 con 0x7fe990153c80 2023-10-30T13:41:13.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe99526b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99015d620 con 0x7fe9900c42a0 2023-10-30T13:41:13.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe99526b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99015dec0 con 0x7fe9900c4a10 2023-10-30T13:41:13.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98effd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe990153c80 0x7fe99015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48662/0 (socket says 172.21.15.31:48662) 2023-10-30T13:41:13.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98effd640 1 -- 172.21.15.31:0/2932230452 learned_addr learned my addr 172.21.15.31:0/2932230452 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe987fff640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 0x7fe99015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697835011 0 0) 0x7fe990159f20 con 0x7fe990153c80 2023-10-30T13:41:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe97c003530 con 0x7fe990153c80 2023-10-30T13:41:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890666219 0 0) 0x7fe99015d620 con 0x7fe9900c42a0 2023-10-30T13:41:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe990159f20 con 0x7fe9900c42a0 2023-10-30T13:41:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1857213638 0 0) 0x7fe97c003530 con 0x7fe990153c80 2023-10-30T13:41:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 msgr2=0x7fe99015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:13.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 0x7fe99015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.129+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 >> v1:172.21.15.33:6789/0 conn(0x7fe9900c42a0 legacy=0x7fe9900ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:13.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99015d900 con 0x7fe990153c80 2023-10-30T13:41:13.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fe980001680 con 0x7fe990153c80 2023-10-30T13:41:13.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 >> v1:172.21.15.31:6789/0 conn(0x7fe990153c80 legacy=0x7fe99015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:13.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97c003b40 0x7fe97c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 0x7fe97c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c004d10 0x7fe97c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99015dec0 con 0x7fe97c003b40 2023-10-30T13:41:13.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c00ba40 con 0x7fe97c004d10 2023-10-30T13:41:13.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c00bc00 con 0x7fe9900c4a10 2023-10-30T13:41:13.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98effd640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97c003b40 0x7fe97c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe987fff640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 0x7fe97c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98f7fe640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c004d10 0x7fe97c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98effd640 1 -- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 msgr2=0x7fe97c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:13.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98effd640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9900c4a10 0x7fe97c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98effd640 1 -- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c004d10 msgr2=0x7fe97c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:13.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98effd640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c004d10 0x7fe97c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98effd640 1 -- 172.21.15.31:0/2932230452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c003530 con 0x7fe97c003b40 2023-10-30T13:41:13.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98f7fe640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c004d10 0x7fe97c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:13.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98effd640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97c003b40 0x7fe97c003f40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe980007fa0 tx=0x7fe98001cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e78bb6e56d07cd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:13.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980005ae0 con 0x7fe97c003b40 2023-10-30T13:41:13.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe980005c40 con 0x7fe97c003b40 2023-10-30T13:41:13.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.133+0000 7fe98e7fc640 1 -- 172.21.15.31:0/2932230452 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9800a5800 con 0x7fe97c003b40 2023-10-30T13:41:13.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- 172.21.15.31:0/2932230452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97c003b40 msgr2=0x7fe97c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:13.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 --2- 172.21.15.31:0/2932230452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97c003b40 0x7fe97c003f40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe980007fa0 tx=0x7fe98001cb00 comp rx=0 tx=0).stop 2023-10-30T13:41:13.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe98f7fe640 1 -- 172.21.15.31:0/2932230452 reap_dead start 2023-10-30T13:41:13.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- 172.21.15.31:0/2932230452 shutdown_connections 2023-10-30T13:41:13.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- 172.21.15.31:0/2932230452 >> 172.21.15.31:0/2932230452 conn(0x7fe9900715c0 msgr2=0x7fe9900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:13.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- 172.21.15.31:0/2932230452 shutdown_connections 2023-10-30T13:41:13.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- 172.21.15.31:0/2932230452 wait complete. 2023-10-30T13:41:13.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 Processor -- start 2023-10-30T13:41:13.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- start start 2023-10-30T13:41:13.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c003b40 0x7fe99014f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99015bac0 0x7fe9901eef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9901542f0 0x7fe990159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99015b630 con 0x7fe9901542f0 2023-10-30T13:41:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99015b020 con 0x7fe97c003b40 2023-10-30T13:41:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe99526b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99014fb40 con 0x7fe99015bac0 2023-10-30T13:41:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe98f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9901542f0 0x7fe990159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe98f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9901542f0 0x7fe990159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60884/0 (socket says 172.21.15.31:60884) 2023-10-30T13:41:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe98effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c003b40 0x7fe99014f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe98f7fe640 1 -- 172.21.15.31:0/3352436597 learned_addr learned my addr 172.21.15.31:0/3352436597 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.137+0000 7fe987fff640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99015bac0 0x7fe9901eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe98f7fe640 1 -- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99015bac0 msgr2=0x7fe9901eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe98f7fe640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99015bac0 0x7fe9901eef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe98f7fe640 1 -- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c003b40 msgr2=0x7fe99014f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:13.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe98f7fe640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c003b40 0x7fe99014f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe98f7fe640 1 -- 172.21.15.31:0/3352436597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99014fe20 con 0x7fe9901542f0 2023-10-30T13:41:13.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe98f7fe640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9901542f0 0x7fe990159060 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe99008cc20 tx=0x7fe988011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:13.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe9877fe640 1 -- 172.21.15.31:0/3352436597 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98800fc90 con 0x7fe9901542f0 2023-10-30T13:41:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe9877fe640 1 -- 172.21.15.31:0/3352436597 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98800fe10 con 0x7fe9901542f0 2023-10-30T13:41:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe9877fe640 1 -- 172.21.15.31:0/3352436597 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988015650 con 0x7fe9901542f0 2023-10-30T13:41:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99015b2c0 con 0x7fe9901542f0 2023-10-30T13:41:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe990150100 con 0x7fe9901542f0 2023-10-30T13:41:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99015b480 con 0x7fe9901542f0 2023-10-30T13:41:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe9877fe640 1 -- 172.21.15.31:0/3352436597 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe9880157b0 con 0x7fe9901542f0 2023-10-30T13:41:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe9877fe640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9700664d0 0x7fe970068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.141+0000 7fe9877fe640 1 -- 172.21.15.31:0/3352436597 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fe988087420 con 0x7fe9901542f0 2023-10-30T13:41:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.145+0000 7fe98effd640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9700664d0 0x7fe970068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.145+0000 7fe98effd640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9700664d0 0x7fe970068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9800ab000 tx=0x7fe9800a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.149+0000 7fe9877fe640 1 -- 172.21.15.31:0/3352436597 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe9880546b0 con 0x7fe9901542f0 2023-10-30T13:41:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.337+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe99004e170 con 0x7fe9901542f0 2023-10-30T13:41:13.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.337+0000 7fe9877fe640 1 -- 172.21.15.31:0/3352436597 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fe988057cd0 con 0x7fe9901542f0 2023-10-30T13:41:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:13.341 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:13.027583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:13.341 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 34 2023-10-30T13:41:13.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9700664d0 msgr2=0x7fe970068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:13.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9700664d0 0x7fe970068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9800ab000 tx=0x7fe9800a7070 comp rx=0 tx=0).stop 2023-10-30T13:41:13.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9901542f0 msgr2=0x7fe990159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9901542f0 0x7fe990159060 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe99008cc20 tx=0x7fe988011f70 comp rx=0 tx=0).stop 2023-10-30T13:41:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 shutdown_connections 2023-10-30T13:41:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9700664d0 0x7fe970068990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9901542f0 0x7fe990159060 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe99015bac0 0x7fe9901eef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 --2- 172.21.15.31:0/3352436597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe97c003b40 0x7fe99014f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 >> 172.21.15.31:0/3352436597 conn(0x7fe9900715c0 msgr2=0x7fe990152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 shutdown_connections 2023-10-30T13:41:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:13.341+0000 7fe99526b640 1 -- 172.21.15.31:0/3352436597 wait complete. 2023-10-30T13:41:13.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:15.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:15.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f669abf8640 1 Processor -- start 2023-10-30T13:41:15.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f669abf8640 1 -- start start 2023-10-30T13:41:15.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f669abf8640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f66940c55d0 0x7f669406e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:15.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f669abf8640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f669406eb70 0x7f6694151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:15.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f669abf8640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66941578e0 con 0x7f66940c59b0 2023-10-30T13:41:15.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f669abf8640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6694158720 con 0x7f66940c55d0 2023-10-30T13:41:15.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f669abf8640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66941576f0 con 0x7f669406eb70 2023-10-30T13:41:15.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f6698994640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f66940c55d0 0x7f669406e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:15.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f6699195640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f66940c59b0 0x7f669415b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:36644/0 (socket says 172.21.15.31:36644) 2023-10-30T13:41:15.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f6699195640 1 -- 172.21.15.31:0/3610257632 learned_addr learned my addr 172.21.15.31:0/3610257632 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:15.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.517+0000 7f6693fff640 1 --2- 172.21.15.31:0/3610257632 >> v2:172.21.15.33:3300/0 conn(0x7f669406eb70 0x7f6694151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:15.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f6698994640 1 -- 172.21.15.31:0/3610257632 >> v2:172.21.15.33:3300/0 conn(0x7f669406eb70 msgr2=0x7f6694151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:15.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f6698994640 1 --2- 172.21.15.31:0/3610257632 >> v2:172.21.15.33:3300/0 conn(0x7f669406eb70 0x7f6694151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f6698994640 1 -- 172.21.15.31:0/3610257632 >> v1:172.21.15.31:6789/0 conn(0x7f66940c59b0 legacy=0x7f669415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f6698994640 1 -- 172.21.15.31:0/3610257632 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694157ba0 con 0x7f66940c55d0 2023-10-30T13:41:15.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f6693fff640 1 --2- 172.21.15.31:0/3610257632 >> v2:172.21.15.33:3300/0 conn(0x7f669406eb70 0x7f6694151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:15.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f6698994640 1 --2- 172.21.15.31:0/3610257632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669406e630 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f669406d6f0 tx=0x7f668801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0edef902695237 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:15.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f66937fe640 1 -- 172.21.15.31:0/3610257632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66880a8950 con 0x7f66940c55d0 2023-10-30T13:41:15.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f66937fe640 1 -- 172.21.15.31:0/3610257632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66880a8ab0 con 0x7f66940c55d0 2023-10-30T13:41:15.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f66937fe640 1 -- 172.21.15.31:0/3610257632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66880a8e40 con 0x7f66940c55d0 2023-10-30T13:41:15.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f669abf8640 1 -- 172.21.15.31:0/3610257632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 msgr2=0x7f669406e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f669abf8640 1 --2- 172.21.15.31:0/3610257632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669406e630 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f669406d6f0 tx=0x7f668801ec50 comp rx=0 tx=0).stop 2023-10-30T13:41:15.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f669abf8640 1 -- 172.21.15.31:0/3610257632 shutdown_connections 2023-10-30T13:41:15.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f669abf8640 1 --2- 172.21.15.31:0/3610257632 >> v2:172.21.15.33:3300/0 conn(0x7f669406eb70 0x7f6694151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f669abf8640 1 --2- 172.21.15.31:0/3610257632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669406e630 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f669abf8640 1 -- 172.21.15.31:0/3610257632 >> 172.21.15.31:0/3610257632 conn(0x7f669406afa0 msgr2=0x7f669406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:15.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.521+0000 7f669abf8640 1 -- 172.21.15.31:0/3610257632 shutdown_connections 2023-10-30T13:41:15.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 -- 172.21.15.31:0/3610257632 wait complete. 2023-10-30T13:41:15.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 Processor -- start 2023-10-30T13:41:15.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 -- start start 2023-10-30T13:41:15.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f669406eb70 0x7f66940bfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:15.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669414fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66940c59b0 0x7f6694150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669414ebd0 con 0x7f66940c55d0 2023-10-30T13:41:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66940ca5a0 con 0x7f66940c59b0 2023-10-30T13:41:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f669abf8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669414ee20 con 0x7f669406eb70 2023-10-30T13:41:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669414fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669414fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47524/0 (socket says 172.21.15.31:47524) 2023-10-30T13:41:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 -- 172.21.15.31:0/4199041992 learned_addr learned my addr 172.21.15.31:0/4199041992 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6699195640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66940c59b0 0x7f6694150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6698994640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f669406eb70 0x7f66940bfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 -- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f669406eb70 msgr2=0x7f66940bfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f669406eb70 0x7f66940bfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 -- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66940c59b0 msgr2=0x7f6694150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66940c59b0 0x7f6694150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 -- 172.21.15.31:0/4199041992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688009420 con 0x7f66940c55d0 2023-10-30T13:41:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.525+0000 7f6693fff640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669414fec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f668000e810 tx=0x7f668000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:15.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f66917fa640 1 -- 172.21.15.31:0/4199041992 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668000c8d0 con 0x7f66940c55d0 2023-10-30T13:41:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f66917fa640 1 -- 172.21.15.31:0/4199041992 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668000ca30 con 0x7f66940c55d0 2023-10-30T13:41:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f66917fa640 1 -- 172.21.15.31:0/4199041992 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668000cdc0 con 0x7f66940c55d0 2023-10-30T13:41:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669414f100 con 0x7f66940c55d0 2023-10-30T13:41:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669414f6c0 con 0x7f66940c55d0 2023-10-30T13:41:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669404e170 con 0x7f66940c55d0 2023-10-30T13:41:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f66917fa640 1 -- 172.21.15.31:0/4199041992 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6680013410 con 0x7f66940c55d0 2023-10-30T13:41:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f66917fa640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66780664d0 0x7f6678068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f66917fa640 1 -- 172.21.15.31:0/4199041992 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f6680083de0 con 0x7f66940c55d0 2023-10-30T13:41:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.529+0000 7f6698994640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66780664d0 0x7f6678068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.533+0000 7f6698994640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66780664d0 0x7f6678068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f66880b2000 tx=0x7f6688008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.533+0000 7f66917fa640 1 -- 172.21.15.31:0/4199041992 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6680051100 con 0x7f66940c55d0 2023-10-30T13:41:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.721+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f669415b440 con 0x7f66940c55d0 2023-10-30T13:41:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.721+0000 7f66917fa640 1 -- 172.21.15.31:0/4199041992 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f6680054720 con 0x7f66940c55d0 2023-10-30T13:41:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:15.724 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:13.027583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:15.724 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 34 2023-10-30T13:41:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66780664d0 msgr2=0x7f6678068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66780664d0 0x7f6678068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f66880b2000 tx=0x7f6688008040 comp rx=0 tx=0).stop 2023-10-30T13:41:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 msgr2=0x7f669414fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669414fec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f668000e810 tx=0x7f668000ecd0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 shutdown_connections 2023-10-30T13:41:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f66780664d0 0x7f6678068990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66940c59b0 0x7f6694150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66940c55d0 0x7f669414fec0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 --2- 172.21.15.31:0/4199041992 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f669406eb70 0x7f66940bfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 >> 172.21.15.31:0/4199041992 conn(0x7f669406afa0 msgr2=0x7f6694155340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 shutdown_connections 2023-10-30T13:41:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:15.725+0000 7f669abf8640 1 -- 172.21.15.31:0/4199041992 wait complete. 2023-10-30T13:41:15.745 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:17.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:17.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe928a74640 1 Processor -- start 2023-10-30T13:41:17.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe928a74640 1 -- start start 2023-10-30T13:41:17.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe928a74640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe9240c8eb0 0x7fe92415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe928a74640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92406aeb0 0x7fe924069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe928a74640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92415dcc0 con 0x7fe92415ac70 2023-10-30T13:41:17.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe928a74640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe924159800 con 0x7fe9240c8eb0 2023-10-30T13:41:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe928a74640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe924159d80 con 0x7fe92406aeb0 2023-10-30T13:41:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe9237fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe9240c8eb0 0x7fe92415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe9237fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe9240c8eb0 0x7fe92415cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47538/0 (socket says 172.21.15.31:47538) 2023-10-30T13:41:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe9237fe640 1 -- 172.21.15.31:0/4064339662 learned_addr learned my addr 172.21.15.31:0/4064339662 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:17.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe923fff640 1 --2- 172.21.15.31:0/4064339662 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92406aeb0 0x7fe924069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe9237fe640 1 -- 172.21.15.31:0/4064339662 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92406aeb0 msgr2=0x7fe924069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:17.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe9237fe640 1 --2- 172.21.15.31:0/4064339662 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92406aeb0 0x7fe924069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe9237fe640 1 -- 172.21.15.31:0/4064339662 >> v1:172.21.15.33:6789/0 conn(0x7fe92415ac70 legacy=0x7fe92415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.901+0000 7fe9237fe640 1 -- 172.21.15.31:0/4064339662 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92415d420 con 0x7fe9240c8eb0 2023-10-30T13:41:17.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe9237fe640 1 --2- 172.21.15.31:0/4064339662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9240c8eb0 0x7fe92415cd70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe914009870 tx=0x7fe91401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea053dff54412ae4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:17.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe9227fc640 1 -- 172.21.15.31:0/4064339662 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9140a8a30 con 0x7fe9240c8eb0 2023-10-30T13:41:17.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe9227fc640 1 -- 172.21.15.31:0/4064339662 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9140a8b90 con 0x7fe9240c8eb0 2023-10-30T13:41:17.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe9227fc640 1 -- 172.21.15.31:0/4064339662 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9140b0440 con 0x7fe9240c8eb0 2023-10-30T13:41:17.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 -- 172.21.15.31:0/4064339662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9240c8eb0 msgr2=0x7fe92415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 --2- 172.21.15.31:0/4064339662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9240c8eb0 0x7fe92415cd70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe914009870 tx=0x7fe91401ed30 comp rx=0 tx=0).stop 2023-10-30T13:41:17.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 -- 172.21.15.31:0/4064339662 shutdown_connections 2023-10-30T13:41:17.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 --2- 172.21.15.31:0/4064339662 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92406aeb0 0x7fe924069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 --2- 172.21.15.31:0/4064339662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9240c8eb0 0x7fe92415cd70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 -- 172.21.15.31:0/4064339662 >> 172.21.15.31:0/4064339662 conn(0x7fe9240cef00 msgr2=0x7fe9240cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:17.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 -- 172.21.15.31:0/4064339662 shutdown_connections 2023-10-30T13:41:17.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.905+0000 7fe928a74640 1 -- 172.21.15.31:0/4064339662 wait complete. 2023-10-30T13:41:17.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 Processor -- start 2023-10-30T13:41:17.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 -- start start 2023-10-30T13:41:17.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92406aeb0 0x7fe924154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9240c8eb0 0x7fe9240c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92415ac70 0x7fe9240c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415c0d0 con 0x7fe92406aeb0 2023-10-30T13:41:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415d0a0 con 0x7fe9240c8eb0 2023-10-30T13:41:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240c5a60 con 0x7fe92415ac70 2023-10-30T13:41:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92406aeb0 0x7fe924154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92406aeb0 0x7fe924154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47552/0 (socket says 172.21.15.31:47552) 2023-10-30T13:41:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 -- 172.21.15.31:0/1655108313 learned_addr learned my addr 172.21.15.31:0/1655108313 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe922ffd640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9240c8eb0 0x7fe9240c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe923fff640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92415ac70 0x7fe9240c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 -- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92415ac70 msgr2=0x7fe9240c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92415ac70 0x7fe9240c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 -- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9240c8eb0 msgr2=0x7fe9240c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9240c8eb0 0x7fe9240c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 -- 172.21.15.31:0/1655108313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140094d0 con 0x7fe92406aeb0 2023-10-30T13:41:17.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe923fff640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92415ac70 0x7fe9240c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:17.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe9237fe640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92406aeb0 0x7fe924154170 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe91401d3e0 tx=0x7fe91401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:17.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe90bfff640 1 -- 172.21.15.31:0/1655108313 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9140ba600 con 0x7fe92406aeb0 2023-10-30T13:41:17.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe90bfff640 1 -- 172.21.15.31:0/1655108313 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9140ba780 con 0x7fe92406aeb0 2023-10-30T13:41:17.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe90bfff640 1 -- 172.21.15.31:0/1655108313 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9140bab30 con 0x7fe92406aeb0 2023-10-30T13:41:17.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9240c4940 con 0x7fe92406aeb0 2023-10-30T13:41:17.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.909+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92415c2e0 con 0x7fe92406aeb0 2023-10-30T13:41:17.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.913+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92404e170 con 0x7fe92406aeb0 2023-10-30T13:41:17.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.913+0000 7fe90bfff640 1 -- 172.21.15.31:0/1655108313 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe9140bac90 con 0x7fe92406aeb0 2023-10-30T13:41:17.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.913+0000 7fe90bfff640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9100664d0 0x7fe910068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.913+0000 7fe90bfff640 1 -- 172.21.15.31:0/1655108313 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fe9141225a0 con 0x7fe92406aeb0 2023-10-30T13:41:17.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.913+0000 7fe922ffd640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9100664d0 0x7fe910068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.913+0000 7fe922ffd640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9100664d0 0x7fe910068990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe9240675b0 tx=0x7fe918009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:17.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:17.917+0000 7fe90bfff640 1 -- 172.21.15.31:0/1655108313 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe9140ef810 con 0x7fe92406aeb0 2023-10-30T13:41:18.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9240ca960 con 0x7fe92406aeb0 2023-10-30T13:41:18.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe90bfff640 1 -- 172.21.15.31:0/1655108313 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fe9140f2e30 con 0x7fe92406aeb0 2023-10-30T13:41:18.110 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:13.027583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:18.112 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 34 2023-10-30T13:41:18.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9100664d0 msgr2=0x7fe910068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:18.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9100664d0 0x7fe910068990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe9240675b0 tx=0x7fe918009040 comp rx=0 tx=0).stop 2023-10-30T13:41:18.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92406aeb0 msgr2=0x7fe924154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:18.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92406aeb0 0x7fe924154170 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe91401d3e0 tx=0x7fe91401bad0 comp rx=0 tx=0).stop 2023-10-30T13:41:18.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 shutdown_connections 2023-10-30T13:41:18.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9100664d0 0x7fe910068990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:18.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe92415ac70 0x7fe9240c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:18.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9240c8eb0 0x7fe9240c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:18.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 --2- 172.21.15.31:0/1655108313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92406aeb0 0x7fe924154170 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:18.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 >> 172.21.15.31:0/1655108313 conn(0x7fe9240cef00 msgr2=0x7fe924069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:18.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 shutdown_connections 2023-10-30T13:41:18.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:18.109+0000 7fe928a74640 1 -- 172.21.15.31:0/1655108313 wait complete. 2023-10-30T13:41:18.131 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:20.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:20.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0f29e640 1 Processor -- start 2023-10-30T13:41:20.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0f29e640 1 -- start start 2023-10-30T13:41:20.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0f29e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8b080c8f00 0x7f8b080cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:20.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0f29e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080cbc30 0x7f8b080cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:20.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0f29e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b080b51a0 con 0x7f8b080cf610 2023-10-30T13:41:20.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0f29e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0815b580 con 0x7f8b080c8f00 2023-10-30T13:41:20.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0f29e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080b4ba0 con 0x7f8b080cbc30 2023-10-30T13:41:20.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0c839640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080cbc30 0x7f8b080cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:20.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0d03a640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8b080c8f00 0x7f8b080cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:20.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0d83b640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f8b080cf610 0x7f8b0815b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:47040/0 (socket says 172.21.15.31:47040) 2023-10-30T13:41:20.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0d83b640 1 -- 172.21.15.31:0/3501190978 learned_addr learned my addr 172.21.15.31:0/3501190978 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:20.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8afffff640 1 -- 172.21.15.31:0/3501190978 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196696214 0 0) 0x7f8b080b51a0 con 0x7f8b080cf610 2023-10-30T13:41:20.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8afffff640 1 -- 172.21.15.31:0/3501190978 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af4003530 con 0x7f8b080cf610 2023-10-30T13:41:20.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0d03a640 1 -- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080cbc30 msgr2=0x7f8b080cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:20.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0d03a640 1 --2- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080cbc30 0x7f8b080cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0d03a640 1 -- 172.21.15.31:0/3501190978 >> v1:172.21.15.33:6789/0 conn(0x7f8b080cf610 legacy=0x7f8b0815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0d03a640 1 -- 172.21.15.31:0/3501190978 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b080cc150 con 0x7f8b080c8f00 2023-10-30T13:41:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.285+0000 7f8b0c839640 1 --2- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080cbc30 0x7f8b080cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0d03a640 1 --2- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080c8f00 0x7f8b080cb6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8b040097f0 tx=0x7f8b0401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea4d62f15403af22 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8afffff640 1 -- 172.21.15.31:0/3501190978 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b040ad660 con 0x7f8b080c8f00 2023-10-30T13:41:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8afffff640 1 -- 172.21.15.31:0/3501190978 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b040ad7c0 con 0x7f8b080c8f00 2023-10-30T13:41:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8afffff640 1 -- 172.21.15.31:0/3501190978 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b040adb50 con 0x7f8b080c8f00 2023-10-30T13:41:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 -- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080c8f00 msgr2=0x7f8b080cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080c8f00 0x7f8b080cb6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8b040097f0 tx=0x7f8b0401c360 comp rx=0 tx=0).stop 2023-10-30T13:41:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 -- 172.21.15.31:0/3501190978 shutdown_connections 2023-10-30T13:41:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080cbc30 0x7f8b080cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/3501190978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080c8f00 0x7f8b080cb6f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 -- 172.21.15.31:0/3501190978 >> 172.21.15.31:0/3501190978 conn(0x7f8b0806c0c0 msgr2=0x7f8b0806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 -- 172.21.15.31:0/3501190978 shutdown_connections 2023-10-30T13:41:20.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.289+0000 7f8b0f29e640 1 -- 172.21.15.31:0/3501190978 wait complete. 2023-10-30T13:41:20.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 Processor -- start 2023-10-30T13:41:20.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 -- start start 2023-10-30T13:41:20.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080c8f00 0x7f8b081560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:20.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080cbc30 0x7f8b08153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:20.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b080cf610 0x7f8b081538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:20.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08154740 con 0x7f8b080cf610 2023-10-30T13:41:20.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081f2230 con 0x7f8b080cbc30 2023-10-30T13:41:20.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081f2510 con 0x7f8b080c8f00 2023-10-30T13:41:20.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b080cf610 0x7f8b081538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:20.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b080cf610 0x7f8b081538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47562/0 (socket says 172.21.15.31:47562) 2023-10-30T13:41:20.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 -- 172.21.15.31:0/1446971421 learned_addr learned my addr 172.21.15.31:0/1446971421 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:20.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d03a640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080c8f00 0x7f8b081560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:20.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0c839640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080cbc30 0x7f8b08153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:20.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 -- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080c8f00 msgr2=0x7f8b081560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:20.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080c8f00 0x7f8b081560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 -- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080cbc30 msgr2=0x7f8b08153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:20.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080cbc30 0x7f8b08153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 -- 172.21.15.31:0/1446971421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af8009420 con 0x7f8b080cf610 2023-10-30T13:41:20.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0d83b640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b080cf610 0x7f8b081538c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000cb40 tx=0x7f8b0000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8afdffb640 1 -- 172.21.15.31:0/1446971421 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000aca0 con 0x7f8b080cf610 2023-10-30T13:41:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8afdffb640 1 -- 172.21.15.31:0/1446971421 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0000ae00 con 0x7f8b080cf610 2023-10-30T13:41:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8afdffb640 1 -- 172.21.15.31:0/1446971421 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00012620 con 0x7f8b080cf610 2023-10-30T13:41:20.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b04009420 con 0x7f8b080cf610 2023-10-30T13:41:20.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.293+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b081f2ad0 con 0x7f8b080cf610 2023-10-30T13:41:20.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.297+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b081f2cf0 con 0x7f8b080cf610 2023-10-30T13:41:20.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.297+0000 7f8afdffb640 1 -- 172.21.15.31:0/1446971421 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b00007070 con 0x7f8b080cf610 2023-10-30T13:41:20.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.297+0000 7f8afdffb640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8aec0664f0 0x7f8aec0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.297+0000 7f8afdffb640 1 -- 172.21.15.31:0/1446971421 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f8b00083ff0 con 0x7f8b080cf610 2023-10-30T13:41:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.297+0000 7f8b0d03a640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8aec0664f0 0x7f8aec0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.297+0000 7f8b0d03a640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8aec0664f0 0x7f8aec0689b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8b040af000 tx=0x7f8b04008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.301+0000 7f8afdffb640 1 -- 172.21.15.31:0/1446971421 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b000b5090 con 0x7f8b080cf610 2023-10-30T13:41:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.489+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b0804e170 con 0x7f8b080cf610 2023-10-30T13:41:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.489+0000 7f8afdffb640 1 -- 172.21.15.31:0/1446971421 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f8b00017090 con 0x7f8b080cf610 2023-10-30T13:41:20.492 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:20.492 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:18.927459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:clientreplay","state_seq":29,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:20.492 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 35 2023-10-30T13:41:20.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8aec0664f0 msgr2=0x7f8aec0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8aec0664f0 0x7f8aec0689b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8b040af000 tx=0x7f8b04008040 comp rx=0 tx=0).stop 2023-10-30T13:41:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b080cf610 msgr2=0x7f8b081538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b080cf610 0x7f8b081538c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000cb40 tx=0x7f8b0000a060 comp rx=0 tx=0).stop 2023-10-30T13:41:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 shutdown_connections 2023-10-30T13:41:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8aec0664f0 0x7f8aec0689b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b080cf610 0x7f8b081538c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b080cbc30 0x7f8b08153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 --2- 172.21.15.31:0/1446971421 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b080c8f00 0x7f8b081560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 >> 172.21.15.31:0/1446971421 conn(0x7f8b0806c0c0 msgr2=0x7f8b080c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 shutdown_connections 2023-10-30T13:41:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:20.493+0000 7f8b0f29e640 1 -- 172.21.15.31:0/1446971421 wait complete. 2023-10-30T13:41:20.512 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:22.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:22.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.665+0000 7fc63750f640 1 Processor -- start 2023-10-30T13:41:22.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc63750f640 1 -- start start 2023-10-30T13:41:22.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc63750f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c4a10 0x7fc63015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc63750f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6300c42a0 0x7fc6300ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc63750f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc630159f20 con 0x7fc630153c80 2023-10-30T13:41:22.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc63750f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63015dec0 con 0x7fc6300c42a0 2023-10-30T13:41:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc63750f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6301599a0 con 0x7fc6300c4a10 2023-10-30T13:41:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc635aac640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6300c42a0 0x7fc6300ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc635aac640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6300c42a0 0x7fc6300ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47572/0 (socket says 172.21.15.31:47572) 2023-10-30T13:41:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc635aac640 1 -- 172.21.15.31:0/2450735953 learned_addr learned my addr 172.21.15.31:0/2450735953 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc634aaa640 1 --2- 172.21.15.31:0/2450735953 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c4a10 0x7fc63015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc635aac640 1 -- 172.21.15.31:0/2450735953 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c4a10 msgr2=0x7fc63015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:22.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc635aac640 1 --2- 172.21.15.31:0/2450735953 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c4a10 0x7fc63015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc635aac640 1 -- 172.21.15.31:0/2450735953 >> v1:172.21.15.33:6789/0 conn(0x7fc630153c80 legacy=0x7fc63015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.669+0000 7fc635aac640 1 -- 172.21.15.31:0/2450735953 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63015d620 con 0x7fc6300c42a0 2023-10-30T13:41:22.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc635aac640 1 --2- 172.21.15.31:0/2450735953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6300c42a0 0x7fc6300ca1e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc628009e50 tx=0x7fc62801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cf487c140313fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:22.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc627fff640 1 -- 172.21.15.31:0/2450735953 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280af400 con 0x7fc6300c42a0 2023-10-30T13:41:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc627fff640 1 -- 172.21.15.31:0/2450735953 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6280ab050 con 0x7fc6300c42a0 2023-10-30T13:41:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc627fff640 1 -- 172.21.15.31:0/2450735953 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280af780 con 0x7fc6300c42a0 2023-10-30T13:41:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 -- 172.21.15.31:0/2450735953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6300c42a0 msgr2=0x7fc6300ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 --2- 172.21.15.31:0/2450735953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6300c42a0 0x7fc6300ca1e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc628009e50 tx=0x7fc62801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 -- 172.21.15.31:0/2450735953 shutdown_connections 2023-10-30T13:41:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 --2- 172.21.15.31:0/2450735953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6300c42a0 0x7fc6300ca1e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 --2- 172.21.15.31:0/2450735953 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c4a10 0x7fc63015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 -- 172.21.15.31:0/2450735953 >> 172.21.15.31:0/2450735953 conn(0x7fc6300715c0 msgr2=0x7fc6300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 -- 172.21.15.31:0/2450735953 shutdown_connections 2023-10-30T13:41:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 -- 172.21.15.31:0/2450735953 wait complete. 2023-10-30T13:41:22.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 Processor -- start 2023-10-30T13:41:22.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.673+0000 7fc63750f640 1 -- start start 2023-10-30T13:41:22.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c42a0 0x7fc6301f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6300c4a10 0x7fc6300b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc630153c80 0x7fc63014f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300b2610 con 0x7fc630153c80 2023-10-30T13:41:22.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300b3a60 con 0x7fc6300c4a10 2023-10-30T13:41:22.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6301f0630 con 0x7fc6300c42a0 2023-10-30T13:41:22.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc630153c80 0x7fc63014f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc630153c80 0x7fc63014f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47580/0 (socket says 172.21.15.31:47580) 2023-10-30T13:41:22.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 -- 172.21.15.31:0/676417049 learned_addr learned my addr 172.21.15.31:0/676417049 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc6352ab640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c42a0 0x7fc6301f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc634aaa640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6300c4a10 0x7fc6300b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 -- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c42a0 msgr2=0x7fc6301f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c42a0 0x7fc6301f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 -- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6300c4a10 msgr2=0x7fc6300b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6300c4a10 0x7fc6300b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 -- 172.21.15.31:0/676417049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc628009b30 con 0x7fc630153c80 2023-10-30T13:41:22.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc6352ab640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c42a0 0x7fc6301f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:22.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc635aac640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc630153c80 0x7fc63014f7b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc6280b2040 tx=0x7fc62801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:22.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc625ffb640 1 -- 172.21.15.31:0/676417049 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280afde0 con 0x7fc630153c80 2023-10-30T13:41:22.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc625ffb640 1 -- 172.21.15.31:0/676417049 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc628009510 con 0x7fc630153c80 2023-10-30T13:41:22.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc625ffb640 1 -- 172.21.15.31:0/676417049 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280b8670 con 0x7fc630153c80 2023-10-30T13:41:22.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6300b3cd0 con 0x7fc630153c80 2023-10-30T13:41:22.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6300b4290 con 0x7fc630153c80 2023-10-30T13:41:22.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.677+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63004e170 con 0x7fc630153c80 2023-10-30T13:41:22.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.681+0000 7fc625ffb640 1 -- 172.21.15.31:0/676417049 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6280b6040 con 0x7fc630153c80 2023-10-30T13:41:22.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.681+0000 7fc625ffb640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6140664a0 0x7fc614068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.681+0000 7fc625ffb640 1 -- 172.21.15.31:0/676417049 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fc628121df0 con 0x7fc630153c80 2023-10-30T13:41:22.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.681+0000 7fc6352ab640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6140664a0 0x7fc614068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.681+0000 7fc6352ab640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6140664a0 0x7fc614068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc62c000f70 tx=0x7fc62c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:22.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.685+0000 7fc625ffb640 1 -- 172.21.15.31:0/676417049 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6280ef2d0 con 0x7fc630153c80 2023-10-30T13:41:22.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.873+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc63014e460 con 0x7fc630153c80 2023-10-30T13:41:22.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.873+0000 7fc625ffb640 1 -- 172.21.15.31:0/676417049 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fc6280f28f0 con 0x7fc630153c80 2023-10-30T13:41:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:22.876 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:18.927459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:clientreplay","state_seq":29,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:22.876 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 35 2023-10-30T13:41:22.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6140664a0 msgr2=0x7fc614068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6140664a0 0x7fc614068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc62c000f70 tx=0x7fc62c009040 comp rx=0 tx=0).stop 2023-10-30T13:41:22.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc630153c80 msgr2=0x7fc63014f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc630153c80 0x7fc63014f7b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc6280b2040 tx=0x7fc62801bf90 comp rx=0 tx=0).stop 2023-10-30T13:41:22.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 shutdown_connections 2023-10-30T13:41:22.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6140664a0 0x7fc614068960 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc630153c80 0x7fc63014f7b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6300c4a10 0x7fc6300b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 --2- 172.21.15.31:0/676417049 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6300c42a0 0x7fc6301f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 >> 172.21.15.31:0/676417049 conn(0x7fc6300715c0 msgr2=0x7fc6300719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:22.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 shutdown_connections 2023-10-30T13:41:22.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:22.877+0000 7fc63750f640 1 -- 172.21.15.31:0/676417049 wait complete. 2023-10-30T13:41:22.897 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:clientreplay", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:41:18.927459+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:41:24.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:25.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.045+0000 7f2fe94ea640 1 Processor -- start 2023-10-30T13:41:25.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.045+0000 7f2fe94ea640 1 -- start start 2023-10-30T13:41:25.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.045+0000 7f2fe94ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2fe4150830 0x7f2fe406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:25.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.045+0000 7f2fe94ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe40c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:25.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.045+0000 7f2fe94ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe40c2310 con 0x7f2fe40c3e40 2023-10-30T13:41:25.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.045+0000 7f2fe94ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe4156710 con 0x7f2fe4150830 2023-10-30T13:41:25.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.045+0000 7f2fe94ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40c4360 con 0x7f2fe40cb7d0 2023-10-30T13:41:25.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fdbfff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe40c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:25.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fe2ffd640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2fe4150830 0x7f2fe406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:25.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fdbfff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe40c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53634/0 (socket says 172.21.15.31:53634) 2023-10-30T13:41:25.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fe2ffd640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2fe4150830 0x7f2fe406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:34098/0 (socket says 172.21.15.31:34098) 2023-10-30T13:41:25.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fdbfff640 1 -- 172.21.15.31:0/26414719 learned_addr learned my addr 172.21.15.31:0/26414719 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:25.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fe37fe640 1 --1- 172.21.15.31:0/26414719 >> v1:172.21.15.33:6789/0 conn(0x7f2fe40c3e40 0x7f2fe4070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:32876/0 (socket says 172.21.15.31:32876) 2023-10-30T13:41:25.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fdbfff640 1 -- 172.21.15.31:0/26414719 >> v2:172.21.15.33:3300/0 conn(0x7f2fe4150830 msgr2=0x7f2fe406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:25.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fdbfff640 1 --2- 172.21.15.31:0/26414719 >> v2:172.21.15.33:3300/0 conn(0x7f2fe4150830 0x7f2fe406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fdbfff640 1 -- 172.21.15.31:0/26414719 >> v1:172.21.15.33:6789/0 conn(0x7f2fe40c3e40 legacy=0x7f2fe4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:25.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fdbfff640 1 -- 172.21.15.31:0/26414719 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe41569b0 con 0x7f2fe40cb7d0 2023-10-30T13:41:25.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.049+0000 7f2fe2ffd640 1 --2- 172.21.15.31:0/26414719 >> v2:172.21.15.33:3300/0 conn(0x7f2fe4150830 0x7f2fe406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:25.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fdbfff640 1 --2- 172.21.15.31:0/26414719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe40c3900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00bb10 tx=0x7f2fcc021400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceb3e169cc5bcb28 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:25.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe27fc640 1 -- 172.21.15.31:0/26414719 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0b1600 con 0x7f2fe40cb7d0 2023-10-30T13:41:25.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe27fc640 1 -- 172.21.15.31:0/26414719 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc0ad050 con 0x7f2fe40cb7d0 2023-10-30T13:41:25.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe27fc640 1 -- 172.21.15.31:0/26414719 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0b1980 con 0x7f2fe40cb7d0 2023-10-30T13:41:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- 172.21.15.31:0/26414719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 msgr2=0x7f2fe40c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/26414719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe40c3900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00bb10 tx=0x7f2fcc021400 comp rx=0 tx=0).stop 2023-10-30T13:41:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- 172.21.15.31:0/26414719 shutdown_connections 2023-10-30T13:41:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/26414719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe40c3900 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/26414719 >> v2:172.21.15.33:3300/0 conn(0x7f2fe4150830 0x7f2fe406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- 172.21.15.31:0/26414719 >> 172.21.15.31:0/26414719 conn(0x7f2fe40cac70 msgr2=0x7f2fe40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- 172.21.15.31:0/26414719 shutdown_connections 2023-10-30T13:41:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- 172.21.15.31:0/26414719 wait complete. 2023-10-30T13:41:25.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 Processor -- start 2023-10-30T13:41:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- start start 2023-10-30T13:41:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe40c3e40 0x7f2fe4155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe41555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fe4150830 0x7f2fe4157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe414f800 con 0x7f2fe40c3e40 2023-10-30T13:41:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe414fdc0 con 0x7f2fe4150830 2023-10-30T13:41:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe94ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe41eec30 con 0x7f2fe40cb7d0 2023-10-30T13:41:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.053+0000 7f2fe2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe40c3e40 0x7f2fe4155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe40c3e40 0x7f2fe4155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45476/0 (socket says 172.21.15.31:45476) 2023-10-30T13:41:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 -- 172.21.15.31:0/3135838016 learned_addr learned my addr 172.21.15.31:0/3135838016 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:25.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fdbfff640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe41555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:25.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fe4150830 0x7f2fe4157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:25.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 -- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 msgr2=0x7f2fe41555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:25.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe41555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 -- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fe4150830 msgr2=0x7f2fe4157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:25.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fe4150830 0x7f2fe4157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 -- 172.21.15.31:0/3135838016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd4009420 con 0x7f2fe40c3e40 2023-10-30T13:41:25.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fe4150830 0x7f2fe4157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:25.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe2ffd640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe40c3e40 0x7f2fe4155070 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400e980 tx=0x7f2fd400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:25.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fdb7fe640 1 -- 172.21.15.31:0/3135838016 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400ca00 con 0x7f2fe40c3e40 2023-10-30T13:41:25.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fdb7fe640 1 -- 172.21.15.31:0/3135838016 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd400cb60 con 0x7f2fe40c3e40 2023-10-30T13:41:25.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc00b770 con 0x7f2fe40c3e40 2023-10-30T13:41:25.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fdb7fe640 1 -- 172.21.15.31:0/3135838016 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4013470 con 0x7f2fe40c3e40 2023-10-30T13:41:25.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe41eeed0 con 0x7f2fe40c3e40 2023-10-30T13:41:25.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe404e170 con 0x7f2fe40c3e40 2023-10-30T13:41:25.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fdb7fe640 1 -- 172.21.15.31:0/3135838016 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fd40135d0 con 0x7f2fe40c3e40 2023-10-30T13:41:25.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fdb7fe640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2fc40664a0 0x7f2fc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:25.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.057+0000 7f2fdb7fe640 1 -- 172.21.15.31:0/3135838016 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f2fd4084200 con 0x7f2fe40c3e40 2023-10-30T13:41:25.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.061+0000 7f2fdbfff640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2fc40664a0 0x7f2fc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:25.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.061+0000 7f2fdbfff640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2fc40664a0 0x7f2fc4068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0b3000 tx=0x7f2fcc0ac000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:25.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.061+0000 7f2fdb7fe640 1 -- 172.21.15.31:0/3135838016 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fd4051520 con 0x7f2fe40c3e40 2023-10-30T13:41:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.253+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe4070300 con 0x7f2fe40c3e40 2023-10-30T13:41:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.253+0000 7f2fdb7fe640 1 -- 172.21.15.31:0/3135838016 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f2fd4054b40 con 0x7f2fe40c3e40 2023-10-30T13:41:25.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:25.255 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:24.595379+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2023-10-30T13:41:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2fc40664a0 msgr2=0x7f2fc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2fc40664a0 0x7f2fc4068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0b3000 tx=0x7f2fcc0ac000 comp rx=0 tx=0).stop 2023-10-30T13:41:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe40c3e40 msgr2=0x7f2fe4155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe40c3e40 0x7f2fe4155070 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400e980 tx=0x7f2fd400ee40 comp rx=0 tx=0).stop 2023-10-30T13:41:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 shutdown_connections 2023-10-30T13:41:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2fc40664a0 0x7f2fc4068960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fe4150830 0x7f2fe4157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2fe40cb7d0 0x7f2fe41555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 --2- 172.21.15.31:0/3135838016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe40c3e40 0x7f2fe4155070 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 >> 172.21.15.31:0/3135838016 conn(0x7f2fe40cac70 msgr2=0x7f2fe40c67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 shutdown_connections 2023-10-30T13:41:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:25.257+0000 7f2fe94ea640 1 -- 172.21.15.31:0/3135838016 wait complete. 2023-10-30T13:41:25.276 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=4276 2023-10-30T13:41:25.276 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.f 2023-10-30T13:41:25.573 INFO:tasks.ceph.mds.b.smithi033.stderr:2023-10-30T13:41:25.572+0000 7f01d8981640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:41:25.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:25.648 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:29.277 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T13:41:29.277 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T13:41:29.278 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T13:41:29.281 INFO:tasks.ceph.mds.f:Started 2023-10-30T13:41:29.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:29.361 INFO:tasks.ceph.mds.f.smithi033.stdout:starting mds.f at 2023-10-30T13:41:29.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.433+0000 7f0faeb1d640 1 Processor -- start 2023-10-30T13:41:29.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.433+0000 7f0faeb1d640 1 -- start start 2023-10-30T13:41:29.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0faeb1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa8151840 0x7f0fa80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0faeb1d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0fa80cb8b0 0x7f0fa806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0faeb1d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa80c5030 con 0x7f0fa8151c20 2023-10-30T13:41:29.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0faeb1d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa8157830 con 0x7f0fa80cb8b0 2023-10-30T13:41:29.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0faeb1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa8157b10 con 0x7f0fa8151840 2023-10-30T13:41:29.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fad0ba640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0fa80cb8b0 0x7f0fa806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fa7fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f0fa8151c20 0x7f0fa8072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:32888/0 (socket says 172.21.15.31:32888) 2023-10-30T13:41:29.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fac8b9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa8151840 0x7f0fa80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fa7fff640 1 -- 172.21.15.31:0/1505304754 learned_addr learned my addr 172.21.15.31:0/1505304754 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:29.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fa77fe640 1 -- 172.21.15.31:0/1505304754 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403237844 0 0) 0x7f0fa80c5030 con 0x7f0fa8151c20 2023-10-30T13:41:29.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fa77fe640 1 -- 172.21.15.31:0/1505304754 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f98003530 con 0x7f0fa8151c20 2023-10-30T13:41:29.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fad0ba640 1 -- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa8151840 msgr2=0x7f0fa80c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:29.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fad0ba640 1 --2- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa8151840 0x7f0fa80c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fad0ba640 1 -- 172.21.15.31:0/1505304754 >> v1:172.21.15.33:6789/0 conn(0x7f0fa8151c20 legacy=0x7f0fa8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fad0ba640 1 -- 172.21.15.31:0/1505304754 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa8157df0 con 0x7f0fa80cb8b0 2023-10-30T13:41:29.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.437+0000 7f0fac8b9640 1 --2- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa8151840 0x7f0fa80c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:29.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0fad0ba640 1 --2- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa80cb8b0 0x7f0fa806b600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009d90 tx=0x7f0fa001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3f0b027cdba7b64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:29.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0fa77fe640 1 -- 172.21.15.31:0/1505304754 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00af600 con 0x7f0fa80cb8b0 2023-10-30T13:41:29.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0fa77fe640 1 -- 172.21.15.31:0/1505304754 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa00ab050 con 0x7f0fa80cb8b0 2023-10-30T13:41:29.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0fa77fe640 1 -- 172.21.15.31:0/1505304754 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00af980 con 0x7f0fa80cb8b0 2023-10-30T13:41:29.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa80cb8b0 msgr2=0x7f0fa806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa80cb8b0 0x7f0fa806b600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009d90 tx=0x7f0fa001f130 comp rx=0 tx=0).stop 2023-10-30T13:41:29.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1505304754 shutdown_connections 2023-10-30T13:41:29.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa80cb8b0 0x7f0fa806b600 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1505304754 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa8151840 0x7f0fa80c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1505304754 >> 172.21.15.31:0/1505304754 conn(0x7f0fa80cac70 msgr2=0x7f0fa80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:29.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1505304754 shutdown_connections 2023-10-30T13:41:29.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1505304754 wait complete. 2023-10-30T13:41:29.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 Processor -- start 2023-10-30T13:41:29.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.441+0000 7f0faeb1d640 1 -- start start 2023-10-30T13:41:29.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa80cb8b0 0x7f0fa80c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa8151840 0x7f0fa81eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa8151c20 0x7f0fa81ef470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa8156730 con 0x7f0fa8151c20 2023-10-30T13:41:29.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80ca2e0 con 0x7f0fa8151840 2023-10-30T13:41:29.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80ca590 con 0x7f0fa80cb8b0 2023-10-30T13:41:29.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa8151c20 0x7f0fa81ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa8151c20 0x7f0fa81ef470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45490/0 (socket says 172.21.15.31:45490) 2023-10-30T13:41:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 -- 172.21.15.31:0/1225817973 learned_addr learned my addr 172.21.15.31:0/1225817973 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fa7fff640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa8151840 0x7f0fa81eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fac8b9640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa80cb8b0 0x7f0fa80c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 -- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa80cb8b0 msgr2=0x7f0fa80c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa80cb8b0 0x7f0fa80c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 -- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa8151840 msgr2=0x7f0fa81eef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa8151840 0x7f0fa81eef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 -- 172.21.15.31:0/1225817973 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c009420 con 0x7f0fa8151c20 2023-10-30T13:41:29.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fad0ba640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa8151c20 0x7f0fa81ef470 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009d90 tx=0x7f0fa001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:29.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fa57fa640 1 -- 172.21.15.31:0/1225817973 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00afbf0 con 0x7f0fa8151c20 2023-10-30T13:41:29.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fa57fa640 1 -- 172.21.15.31:0/1225817973 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa00afd50 con 0x7f0fa8151c20 2023-10-30T13:41:29.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa0009a70 con 0x7f0fa8151c20 2023-10-30T13:41:29.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0fa57fa640 1 -- 172.21.15.31:0/1225817973 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00ba660 con 0x7f0fa8151c20 2023-10-30T13:41:29.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa80ca850 con 0x7f0fa8151c20 2023-10-30T13:41:29.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.445+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa8066510 con 0x7f0fa8151c20 2023-10-30T13:41:29.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.449+0000 7f0fa57fa640 1 -- 172.21.15.31:0/1225817973 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fa00ba7c0 con 0x7f0fa8151c20 2023-10-30T13:41:29.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.449+0000 7f0fa57fa640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f8c0664a0 0x7f0f8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.449+0000 7f0fa57fa640 1 -- 172.21.15.31:0/1225817973 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f0fa0122930 con 0x7f0fa8151c20 2023-10-30T13:41:29.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.449+0000 7f0fac8b9640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f8c0664a0 0x7f0f8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.449+0000 7f0fac8b9640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f8c0664a0 0x7f0f8c068960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009cf0 tx=0x7f0f9c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.453+0000 7f0fa57fa640 1 -- 172.21.15.31:0/1225817973 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fa00efc50 con 0x7f0fa8151c20 2023-10-30T13:41:29.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.641+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fa804e170 con 0x7f0fa8151c20 2023-10-30T13:41:29.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0fa57fa640 1 -- 172.21.15.31:0/1225817973 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f0fa00be020 con 0x7f0fa8151c20 2023-10-30T13:41:29.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:29.645 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:24.595379+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:29.645 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2023-10-30T13:41:29.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f8c0664a0 msgr2=0x7f0f8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f8c0664a0 0x7f0f8c068960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009cf0 tx=0x7f0f9c009190 comp rx=0 tx=0).stop 2023-10-30T13:41:29.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa8151c20 msgr2=0x7f0fa81ef470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa8151c20 0x7f0fa81ef470 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009d90 tx=0x7f0fa001bed0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 shutdown_connections 2023-10-30T13:41:29.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0f8c0664a0 0x7f0f8c068960 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa8151c20 0x7f0fa81ef470 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fa8151840 0x7f0fa81eef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 --2- 172.21.15.31:0/1225817973 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0fa80cb8b0 0x7f0fa80c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 >> 172.21.15.31:0/1225817973 conn(0x7f0fa80cac70 msgr2=0x7f0fa80c6390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:29.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 shutdown_connections 2023-10-30T13:41:29.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:29.645+0000 7f0faeb1d640 1 -- 172.21.15.31:0/1225817973 wait complete. 2023-10-30T13:41:29.666 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T13:41:30.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './128/37' 2023-10-30T13:41:30.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './128' 2023-10-30T13:41:30.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/19' 2023-10-30T13:41:30.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/15' 2023-10-30T13:41:30.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/31' 2023-10-30T13:41:30.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/36' 2023-10-30T13:41:30.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/20' 2023-10-30T13:41:30.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/12' 2023-10-30T13:41:30.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/39' 2023-10-30T13:41:30.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/10' 2023-10-30T13:41:30.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/34' 2023-10-30T13:41:30.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/38' 2023-10-30T13:41:30.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/11' 2023-10-30T13:41:30.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/16' 2023-10-30T13:41:30.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/32' 2023-10-30T13:41:30.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/30' 2023-10-30T13:41:30.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/14' 2023-10-30T13:41:30.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/18' 2023-10-30T13:41:30.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/13' 2023-10-30T13:41:30.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/33' 2023-10-30T13:41:30.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/17' 2023-10-30T13:41:30.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/35' 2023-10-30T13:41:30.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './129/37' 2023-10-30T13:41:30.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './129' 2023-10-30T13:41:30.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/19' 2023-10-30T13:41:30.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/15' 2023-10-30T13:41:30.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/31' 2023-10-30T13:41:30.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/36' 2023-10-30T13:41:30.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/20' 2023-10-30T13:41:30.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/12' 2023-10-30T13:41:30.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/39' 2023-10-30T13:41:30.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/10' 2023-10-30T13:41:30.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/34' 2023-10-30T13:41:30.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/38' 2023-10-30T13:41:30.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/11' 2023-10-30T13:41:30.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/16' 2023-10-30T13:41:30.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/32' 2023-10-30T13:41:30.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/30' 2023-10-30T13:41:30.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/14' 2023-10-30T13:41:30.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/18' 2023-10-30T13:41:30.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/13' 2023-10-30T13:41:30.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/33' 2023-10-30T13:41:30.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/17' 2023-10-30T13:41:30.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/35' 2023-10-30T13:41:30.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './130/37' 2023-10-30T13:41:30.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './130' 2023-10-30T13:41:30.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/19' 2023-10-30T13:41:30.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/15' 2023-10-30T13:41:30.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/31' 2023-10-30T13:41:30.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/36' 2023-10-30T13:41:30.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/20' 2023-10-30T13:41:30.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/12' 2023-10-30T13:41:30.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/39' 2023-10-30T13:41:30.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/10' 2023-10-30T13:41:30.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/34' 2023-10-30T13:41:30.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/38' 2023-10-30T13:41:30.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/11' 2023-10-30T13:41:30.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/16' 2023-10-30T13:41:30.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/32' 2023-10-30T13:41:30.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/30' 2023-10-30T13:41:30.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/14' 2023-10-30T13:41:30.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/18' 2023-10-30T13:41:30.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/13' 2023-10-30T13:41:30.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/33' 2023-10-30T13:41:30.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/17' 2023-10-30T13:41:30.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/35' 2023-10-30T13:41:30.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './131/37' 2023-10-30T13:41:30.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './131' 2023-10-30T13:41:30.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/19' 2023-10-30T13:41:30.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/15' 2023-10-30T13:41:30.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/31' 2023-10-30T13:41:30.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/36' 2023-10-30T13:41:30.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/20' 2023-10-30T13:41:30.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/12' 2023-10-30T13:41:30.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/39' 2023-10-30T13:41:30.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/10' 2023-10-30T13:41:30.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/34' 2023-10-30T13:41:30.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/38' 2023-10-30T13:41:30.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/11' 2023-10-30T13:41:30.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/16' 2023-10-30T13:41:30.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/32' 2023-10-30T13:41:30.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/30' 2023-10-30T13:41:30.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/14' 2023-10-30T13:41:30.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/18' 2023-10-30T13:41:30.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/13' 2023-10-30T13:41:30.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/33' 2023-10-30T13:41:30.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/17' 2023-10-30T13:41:30.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/35' 2023-10-30T13:41:30.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './132/37' 2023-10-30T13:41:30.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './132' 2023-10-30T13:41:30.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/19' 2023-10-30T13:41:30.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/15' 2023-10-30T13:41:30.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/31' 2023-10-30T13:41:30.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/36' 2023-10-30T13:41:30.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/20' 2023-10-30T13:41:30.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/12' 2023-10-30T13:41:30.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/39' 2023-10-30T13:41:30.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/10' 2023-10-30T13:41:30.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/34' 2023-10-30T13:41:30.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/38' 2023-10-30T13:41:30.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/11' 2023-10-30T13:41:30.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/16' 2023-10-30T13:41:30.990 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/32' 2023-10-30T13:41:30.990 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/30' 2023-10-30T13:41:30.990 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/14' 2023-10-30T13:41:30.990 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/18' 2023-10-30T13:41:30.990 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/13' 2023-10-30T13:41:30.991 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/33' 2023-10-30T13:41:30.991 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/17' 2023-10-30T13:41:30.991 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/35' 2023-10-30T13:41:30.991 INFO:tasks.workunit.client.0.smithi031.stdout:removed './133/37' 2023-10-30T13:41:30.991 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './133' 2023-10-30T13:41:30.991 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/19' 2023-10-30T13:41:30.992 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/15' 2023-10-30T13:41:30.992 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/31' 2023-10-30T13:41:30.992 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/36' 2023-10-30T13:41:30.992 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/20' 2023-10-30T13:41:30.992 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/12' 2023-10-30T13:41:30.993 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/39' 2023-10-30T13:41:30.993 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/10' 2023-10-30T13:41:30.993 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/34' 2023-10-30T13:41:30.993 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/38' 2023-10-30T13:41:30.993 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/11' 2023-10-30T13:41:30.993 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/16' 2023-10-30T13:41:30.994 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/32' 2023-10-30T13:41:30.994 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/30' 2023-10-30T13:41:30.994 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/14' 2023-10-30T13:41:30.994 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/18' 2023-10-30T13:41:30.994 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/13' 2023-10-30T13:41:30.995 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/33' 2023-10-30T13:41:30.995 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/17' 2023-10-30T13:41:30.995 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/35' 2023-10-30T13:41:30.995 INFO:tasks.workunit.client.0.smithi031.stdout:removed './134/37' 2023-10-30T13:41:30.995 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './134' 2023-10-30T13:41:30.995 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/19' 2023-10-30T13:41:30.996 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/15' 2023-10-30T13:41:30.996 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/31' 2023-10-30T13:41:30.996 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/36' 2023-10-30T13:41:30.996 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/20' 2023-10-30T13:41:30.996 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/12' 2023-10-30T13:41:30.997 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/39' 2023-10-30T13:41:30.997 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/10' 2023-10-30T13:41:30.997 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/34' 2023-10-30T13:41:30.997 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/38' 2023-10-30T13:41:30.997 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/11' 2023-10-30T13:41:30.997 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/16' 2023-10-30T13:41:30.998 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/32' 2023-10-30T13:41:30.998 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/30' 2023-10-30T13:41:30.998 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/14' 2023-10-30T13:41:30.998 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/18' 2023-10-30T13:41:30.998 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/13' 2023-10-30T13:41:30.999 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/33' 2023-10-30T13:41:30.999 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/17' 2023-10-30T13:41:30.999 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/35' 2023-10-30T13:41:30.999 INFO:tasks.workunit.client.0.smithi031.stdout:removed './135/37' 2023-10-30T13:41:30.999 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './135' 2023-10-30T13:41:30.999 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/19' 2023-10-30T13:41:31.000 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/15' 2023-10-30T13:41:31.000 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/31' 2023-10-30T13:41:31.000 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/36' 2023-10-30T13:41:31.000 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/20' 2023-10-30T13:41:31.000 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/12' 2023-10-30T13:41:31.001 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/39' 2023-10-30T13:41:31.001 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/10' 2023-10-30T13:41:31.001 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/34' 2023-10-30T13:41:31.001 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/38' 2023-10-30T13:41:31.001 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/11' 2023-10-30T13:41:31.001 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/16' 2023-10-30T13:41:31.002 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/32' 2023-10-30T13:41:31.002 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/30' 2023-10-30T13:41:31.002 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/14' 2023-10-30T13:41:31.002 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/18' 2023-10-30T13:41:31.002 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/13' 2023-10-30T13:41:31.003 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/33' 2023-10-30T13:41:31.003 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/17' 2023-10-30T13:41:31.003 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/35' 2023-10-30T13:41:31.003 INFO:tasks.workunit.client.0.smithi031.stdout:removed './136/37' 2023-10-30T13:41:31.003 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './136' 2023-10-30T13:41:31.004 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/19' 2023-10-30T13:41:31.005 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/15' 2023-10-30T13:41:31.005 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/31' 2023-10-30T13:41:31.005 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/36' 2023-10-30T13:41:31.006 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/20' 2023-10-30T13:41:31.006 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/12' 2023-10-30T13:41:31.006 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/39' 2023-10-30T13:41:31.006 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/10' 2023-10-30T13:41:31.006 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/34' 2023-10-30T13:41:31.007 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/38' 2023-10-30T13:41:31.007 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/11' 2023-10-30T13:41:31.007 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/16' 2023-10-30T13:41:31.007 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/32' 2023-10-30T13:41:31.007 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/30' 2023-10-30T13:41:31.008 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/14' 2023-10-30T13:41:31.008 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/18' 2023-10-30T13:41:31.008 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/13' 2023-10-30T13:41:31.008 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/33' 2023-10-30T13:41:31.008 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/17' 2023-10-30T13:41:31.008 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/35' 2023-10-30T13:41:31.009 INFO:tasks.workunit.client.0.smithi031.stdout:removed './137/37' 2023-10-30T13:41:31.009 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './137' 2023-10-30T13:41:31.009 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/19' 2023-10-30T13:41:31.009 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/15' 2023-10-30T13:41:31.009 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/31' 2023-10-30T13:41:31.010 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/36' 2023-10-30T13:41:31.010 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/20' 2023-10-30T13:41:31.010 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/12' 2023-10-30T13:41:31.010 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/39' 2023-10-30T13:41:31.010 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/10' 2023-10-30T13:41:31.010 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/34' 2023-10-30T13:41:31.011 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/38' 2023-10-30T13:41:31.011 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/11' 2023-10-30T13:41:31.011 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/16' 2023-10-30T13:41:31.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:31.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb7609640 1 Processor -- start 2023-10-30T13:41:31.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb7609640 1 -- start start 2023-10-30T13:41:31.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb7609640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006b760 0x7f0bb015e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb7609640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0bb006bb40 0x7f0bb0157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb7609640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb0071e00 con 0x7f0bb006c420 2023-10-30T13:41:31.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb7609640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb0070020 con 0x7f0bb006bb40 2023-10-30T13:41:31.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb7609640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb015eb90 con 0x7f0bb006b760 2023-10-30T13:41:31.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.817+0000 7f0bb53a5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006b760 0x7f0bb015e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb4ba4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0bb006bb40 0x7f0bb0157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb53a5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006b760 0x7f0bb015e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:53658/0 (socket says 172.21.15.31:53658) 2023-10-30T13:41:31.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb53a5640 1 -- 172.21.15.31:0/490895463 learned_addr learned my addr 172.21.15.31:0/490895463 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:31.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb53a5640 1 -- 172.21.15.31:0/490895463 >> v2:172.21.15.33:3300/0 conn(0x7f0bb006bb40 msgr2=0x7f0bb0157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:31.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb53a5640 1 --2- 172.21.15.31:0/490895463 >> v2:172.21.15.33:3300/0 conn(0x7f0bb006bb40 0x7f0bb0157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb53a5640 1 -- 172.21.15.31:0/490895463 >> v1:172.21.15.33:6789/0 conn(0x7f0bb006c420 legacy=0x7f0bb00c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb53a5640 1 -- 172.21.15.31:0/490895463 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb0070f10 con 0x7f0bb006b760 2023-10-30T13:41:31.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb4ba4640 1 --2- 172.21.15.31:0/490895463 >> v2:172.21.15.33:3300/0 conn(0x7f0bb006bb40 0x7f0bb0157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:31.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.821+0000 7f0bb53a5640 1 --2- 172.21.15.31:0/490895463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006b760 0x7f0bb015e650 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009870 tx=0x7f0bac01c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c8af9be9dfab635 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:31.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0ba7fff640 1 -- 172.21.15.31:0/490895463 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac0aa670 con 0x7f0bb006b760 2023-10-30T13:41:31.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0ba7fff640 1 -- 172.21.15.31:0/490895463 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac0aa7d0 con 0x7f0bb006b760 2023-10-30T13:41:31.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0ba7fff640 1 -- 172.21.15.31:0/490895463 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac0aab60 con 0x7f0bb006b760 2023-10-30T13:41:31.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- 172.21.15.31:0/490895463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006b760 msgr2=0x7f0bb015e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 --2- 172.21.15.31:0/490895463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006b760 0x7f0bb015e650 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009870 tx=0x7f0bac01c510 comp rx=0 tx=0).stop 2023-10-30T13:41:31.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- 172.21.15.31:0/490895463 shutdown_connections 2023-10-30T13:41:31.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 --2- 172.21.15.31:0/490895463 >> v2:172.21.15.33:3300/0 conn(0x7f0bb006bb40 0x7f0bb0157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 --2- 172.21.15.31:0/490895463 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006b760 0x7f0bb015e650 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- 172.21.15.31:0/490895463 >> 172.21.15.31:0/490895463 conn(0x7f0bb00cac70 msgr2=0x7f0bb00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:31.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- 172.21.15.31:0/490895463 shutdown_connections 2023-10-30T13:41:31.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- 172.21.15.31:0/490895463 wait complete. 2023-10-30T13:41:31.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 Processor -- start 2023-10-30T13:41:31.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- start start 2023-10-30T13:41:31.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bb006b760 0x7f0bb00c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006bb40 0x7f0bb014c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb006c420 0x7f0bb01fa460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb014bde0 con 0x7f0bb006c420 2023-10-30T13:41:31.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.825+0000 7f0bb7609640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb01f9fd0 con 0x7f0bb006b760 2023-10-30T13:41:31.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb7609640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb014bb00 con 0x7f0bb006bb40 2023-10-30T13:41:31.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb006c420 0x7f0bb01fa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb006c420 0x7f0bb01fa460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45506/0 (socket says 172.21.15.31:45506) 2023-10-30T13:41:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 -- 172.21.15.31:0/4188802331 learned_addr learned my addr 172.21.15.31:0/4188802331 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb53a5640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bb006b760 0x7f0bb00c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb4ba4640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006bb40 0x7f0bb014c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 -- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006bb40 msgr2=0x7f0bb014c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006bb40 0x7f0bb014c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 -- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bb006b760 msgr2=0x7f0bb00c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bb006b760 0x7f0bb00c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 -- 172.21.15.31:0/4188802331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba0009420 con 0x7f0bb006c420 2023-10-30T13:41:31.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb5ba6640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb006c420 0x7f0bb01fa460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800c730 tx=0x7f0ba800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:31.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0ba5ffb640 1 -- 172.21.15.31:0/4188802331 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8003b80 con 0x7f0bb006c420 2023-10-30T13:41:31.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0ba5ffb640 1 -- 172.21.15.31:0/4188802331 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba8003ce0 con 0x7f0bb006c420 2023-10-30T13:41:31.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0ba5ffb640 1 -- 172.21.15.31:0/4188802331 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8005310 con 0x7f0bb006c420 2023-10-30T13:41:31.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac0094d0 con 0x7f0bb006c420 2023-10-30T13:41:31.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb0157180 con 0x7f0bb006c420 2023-10-30T13:41:31.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb014a940 con 0x7f0bb006c420 2023-10-30T13:41:31.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.829+0000 7f0ba5ffb640 1 -- 172.21.15.31:0/4188802331 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ba8005680 con 0x7f0bb006c420 2023-10-30T13:41:31.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.833+0000 7f0ba5ffb640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b940664a0 0x7f0b94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.833+0000 7f0ba5ffb640 1 -- 172.21.15.31:0/4188802331 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f0ba8083ee0 con 0x7f0bb006c420 2023-10-30T13:41:31.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.833+0000 7f0bb53a5640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b940664a0 0x7f0b94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.833+0000 7f0bb53a5640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b940664a0 0x7f0b94068960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0ae000 tx=0x7f0bac0b5040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:31.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:31.837+0000 7f0ba5ffb640 1 -- 172.21.15.31:0/4188802331 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0bb014a940 con 0x7f0bb006c420 2023-10-30T13:41:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bb014a940 con 0x7f0bb006c420 2023-10-30T13:41:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0ba5ffb640 1 -- 172.21.15.31:0/4188802331 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f0bb014a940 con 0x7f0bb006c420 2023-10-30T13:41:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:32.042 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:24.595379+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:32.042 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 37 2023-10-30T13:41:32.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b940664a0 msgr2=0x7f0b94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:32.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b940664a0 0x7f0b94068960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0ae000 tx=0x7f0bac0b5040 comp rx=0 tx=0).stop 2023-10-30T13:41:32.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb006c420 msgr2=0x7f0bb01fa460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:32.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb006c420 0x7f0bb01fa460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800c730 tx=0x7f0ba800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:41:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 shutdown_connections 2023-10-30T13:41:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b940664a0 0x7f0b94068960 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb006c420 0x7f0bb01fa460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0bb006bb40 0x7f0bb014c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 --2- 172.21.15.31:0/4188802331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bb006b760 0x7f0bb00c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:32.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 >> 172.21.15.31:0/4188802331 conn(0x7f0bb00cac70 msgr2=0x7f0bb00c57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:32.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 shutdown_connections 2023-10-30T13:41:32.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:32.041+0000 7f0bb7609640 1 -- 172.21.15.31:0/4188802331 wait complete. 2023-10-30T13:41:32.064 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T13:41:32.064 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 24 secs before thrashing 2023-10-30T13:41:32.497 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/32' 2023-10-30T13:41:32.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/30' 2023-10-30T13:41:32.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/14' 2023-10-30T13:41:32.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/18' 2023-10-30T13:41:32.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/13' 2023-10-30T13:41:32.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/33' 2023-10-30T13:41:32.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/17' 2023-10-30T13:41:32.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/35' 2023-10-30T13:41:32.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './138/37' 2023-10-30T13:41:32.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './138' 2023-10-30T13:41:32.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/19' 2023-10-30T13:41:32.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/15' 2023-10-30T13:41:32.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/31' 2023-10-30T13:41:32.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/36' 2023-10-30T13:41:32.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/20' 2023-10-30T13:41:32.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/12' 2023-10-30T13:41:32.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/39' 2023-10-30T13:41:32.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/10' 2023-10-30T13:41:32.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/34' 2023-10-30T13:41:32.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/38' 2023-10-30T13:41:32.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/11' 2023-10-30T13:41:32.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/16' 2023-10-30T13:41:32.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/32' 2023-10-30T13:41:32.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/30' 2023-10-30T13:41:32.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/14' 2023-10-30T13:41:32.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/18' 2023-10-30T13:41:32.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/13' 2023-10-30T13:41:32.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/33' 2023-10-30T13:41:32.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/17' 2023-10-30T13:41:32.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/35' 2023-10-30T13:41:32.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './139/37' 2023-10-30T13:41:32.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './139' 2023-10-30T13:41:32.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/19' 2023-10-30T13:41:32.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/15' 2023-10-30T13:41:32.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/31' 2023-10-30T13:41:32.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/36' 2023-10-30T13:41:32.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/20' 2023-10-30T13:41:32.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/12' 2023-10-30T13:41:32.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/39' 2023-10-30T13:41:32.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/10' 2023-10-30T13:41:32.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/34' 2023-10-30T13:41:32.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/38' 2023-10-30T13:41:32.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/11' 2023-10-30T13:41:32.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/16' 2023-10-30T13:41:32.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/32' 2023-10-30T13:41:32.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/30' 2023-10-30T13:41:32.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/14' 2023-10-30T13:41:32.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/18' 2023-10-30T13:41:32.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/13' 2023-10-30T13:41:32.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/33' 2023-10-30T13:41:32.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/17' 2023-10-30T13:41:32.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/35' 2023-10-30T13:41:32.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './140/37' 2023-10-30T13:41:32.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './140' 2023-10-30T13:41:32.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/19' 2023-10-30T13:41:32.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/15' 2023-10-30T13:41:32.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/31' 2023-10-30T13:41:32.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/36' 2023-10-30T13:41:32.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/20' 2023-10-30T13:41:32.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/12' 2023-10-30T13:41:32.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/39' 2023-10-30T13:41:32.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/10' 2023-10-30T13:41:32.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/34' 2023-10-30T13:41:32.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/38' 2023-10-30T13:41:32.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/11' 2023-10-30T13:41:32.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/16' 2023-10-30T13:41:32.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/32' 2023-10-30T13:41:32.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/30' 2023-10-30T13:41:32.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/14' 2023-10-30T13:41:32.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/18' 2023-10-30T13:41:32.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/13' 2023-10-30T13:41:32.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/33' 2023-10-30T13:41:32.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/17' 2023-10-30T13:41:32.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/35' 2023-10-30T13:41:32.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './141/37' 2023-10-30T13:41:32.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './141' 2023-10-30T13:41:32.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/19' 2023-10-30T13:41:32.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/15' 2023-10-30T13:41:32.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/31' 2023-10-30T13:41:32.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/36' 2023-10-30T13:41:32.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/20' 2023-10-30T13:41:32.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/12' 2023-10-30T13:41:32.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/39' 2023-10-30T13:41:32.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/10' 2023-10-30T13:41:32.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/34' 2023-10-30T13:41:32.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/38' 2023-10-30T13:41:32.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/11' 2023-10-30T13:41:32.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/16' 2023-10-30T13:41:32.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/32' 2023-10-30T13:41:32.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/30' 2023-10-30T13:41:32.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/14' 2023-10-30T13:41:32.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/18' 2023-10-30T13:41:32.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/13' 2023-10-30T13:41:32.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/33' 2023-10-30T13:41:32.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/17' 2023-10-30T13:41:32.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/35' 2023-10-30T13:41:32.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './142/37' 2023-10-30T13:41:32.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './142' 2023-10-30T13:41:32.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/19' 2023-10-30T13:41:32.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/15' 2023-10-30T13:41:32.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/31' 2023-10-30T13:41:32.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/36' 2023-10-30T13:41:32.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/20' 2023-10-30T13:41:32.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/12' 2023-10-30T13:41:32.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/39' 2023-10-30T13:41:32.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/10' 2023-10-30T13:41:32.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/34' 2023-10-30T13:41:32.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/38' 2023-10-30T13:41:32.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/11' 2023-10-30T13:41:32.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/16' 2023-10-30T13:41:32.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/32' 2023-10-30T13:41:32.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/30' 2023-10-30T13:41:32.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/14' 2023-10-30T13:41:32.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/18' 2023-10-30T13:41:32.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/13' 2023-10-30T13:41:32.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/33' 2023-10-30T13:41:32.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/17' 2023-10-30T13:41:32.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/35' 2023-10-30T13:41:32.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './143/37' 2023-10-30T13:41:32.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './143' 2023-10-30T13:41:32.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/19' 2023-10-30T13:41:32.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/15' 2023-10-30T13:41:32.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/31' 2023-10-30T13:41:32.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/36' 2023-10-30T13:41:32.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/20' 2023-10-30T13:41:32.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/12' 2023-10-30T13:41:32.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/39' 2023-10-30T13:41:32.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/10' 2023-10-30T13:41:32.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/34' 2023-10-30T13:41:32.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/38' 2023-10-30T13:41:32.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/11' 2023-10-30T13:41:32.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/16' 2023-10-30T13:41:32.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/32' 2023-10-30T13:41:32.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/30' 2023-10-30T13:41:32.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/14' 2023-10-30T13:41:32.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/18' 2023-10-30T13:41:32.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/13' 2023-10-30T13:41:32.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/33' 2023-10-30T13:41:32.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/17' 2023-10-30T13:41:32.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/35' 2023-10-30T13:41:32.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './144/37' 2023-10-30T13:41:32.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './144' 2023-10-30T13:41:32.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/19' 2023-10-30T13:41:32.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/15' 2023-10-30T13:41:32.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/31' 2023-10-30T13:41:32.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/36' 2023-10-30T13:41:32.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/20' 2023-10-30T13:41:32.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/12' 2023-10-30T13:41:32.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/39' 2023-10-30T13:41:32.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/10' 2023-10-30T13:41:32.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/34' 2023-10-30T13:41:32.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/38' 2023-10-30T13:41:32.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/11' 2023-10-30T13:41:32.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/16' 2023-10-30T13:41:32.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/32' 2023-10-30T13:41:32.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/30' 2023-10-30T13:41:32.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/14' 2023-10-30T13:41:32.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/18' 2023-10-30T13:41:32.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/13' 2023-10-30T13:41:32.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/33' 2023-10-30T13:41:32.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/17' 2023-10-30T13:41:32.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/35' 2023-10-30T13:41:32.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './145/37' 2023-10-30T13:41:32.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './145' 2023-10-30T13:41:32.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/19' 2023-10-30T13:41:32.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/15' 2023-10-30T13:41:32.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/31' 2023-10-30T13:41:32.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/36' 2023-10-30T13:41:32.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/20' 2023-10-30T13:41:32.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/12' 2023-10-30T13:41:32.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/39' 2023-10-30T13:41:32.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/10' 2023-10-30T13:41:32.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/34' 2023-10-30T13:41:32.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/38' 2023-10-30T13:41:32.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/11' 2023-10-30T13:41:32.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/16' 2023-10-30T13:41:32.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/32' 2023-10-30T13:41:32.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/30' 2023-10-30T13:41:32.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/14' 2023-10-30T13:41:32.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/18' 2023-10-30T13:41:32.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/13' 2023-10-30T13:41:32.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/33' 2023-10-30T13:41:32.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/17' 2023-10-30T13:41:32.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/35' 2023-10-30T13:41:32.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './146/37' 2023-10-30T13:41:32.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './146' 2023-10-30T13:41:32.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/19' 2023-10-30T13:41:32.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/15' 2023-10-30T13:41:32.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/31' 2023-10-30T13:41:32.532 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/36' 2023-10-30T13:41:32.532 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/20' 2023-10-30T13:41:32.532 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/12' 2023-10-30T13:41:32.532 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/39' 2023-10-30T13:41:32.532 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/10' 2023-10-30T13:41:32.533 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/34' 2023-10-30T13:41:32.533 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/38' 2023-10-30T13:41:32.533 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/11' 2023-10-30T13:41:32.533 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/16' 2023-10-30T13:41:32.533 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/32' 2023-10-30T13:41:32.533 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/30' 2023-10-30T13:41:32.534 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/14' 2023-10-30T13:41:32.534 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/18' 2023-10-30T13:41:32.534 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/13' 2023-10-30T13:41:32.534 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/33' 2023-10-30T13:41:32.534 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/17' 2023-10-30T13:41:32.534 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/35' 2023-10-30T13:41:32.535 INFO:tasks.workunit.client.0.smithi031.stdout:removed './147/37' 2023-10-30T13:41:32.535 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './147' 2023-10-30T13:41:32.535 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/19' 2023-10-30T13:41:32.535 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/15' 2023-10-30T13:41:32.535 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/31' 2023-10-30T13:41:32.536 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/36' 2023-10-30T13:41:37.493 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/20' 2023-10-30T13:41:37.493 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/12' 2023-10-30T13:41:37.493 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/39' 2023-10-30T13:41:37.493 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/10' 2023-10-30T13:41:37.494 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/34' 2023-10-30T13:41:37.494 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/38' 2023-10-30T13:41:37.494 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/11' 2023-10-30T13:41:37.494 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/16' 2023-10-30T13:41:37.494 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/32' 2023-10-30T13:41:37.495 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/30' 2023-10-30T13:41:37.495 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/14' 2023-10-30T13:41:37.495 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/18' 2023-10-30T13:41:37.495 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/13' 2023-10-30T13:41:37.495 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/33' 2023-10-30T13:41:37.495 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/17' 2023-10-30T13:41:37.496 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/35' 2023-10-30T13:41:37.496 INFO:tasks.workunit.client.0.smithi031.stdout:removed './148/37' 2023-10-30T13:41:37.496 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './148' 2023-10-30T13:41:37.496 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/19' 2023-10-30T13:41:37.496 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/15' 2023-10-30T13:41:37.496 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/31' 2023-10-30T13:41:37.497 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/36' 2023-10-30T13:41:37.497 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/20' 2023-10-30T13:41:37.497 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/12' 2023-10-30T13:41:37.497 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/39' 2023-10-30T13:41:37.497 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/10' 2023-10-30T13:41:37.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/34' 2023-10-30T13:41:37.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/38' 2023-10-30T13:41:37.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/11' 2023-10-30T13:41:37.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/16' 2023-10-30T13:41:37.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/32' 2023-10-30T13:41:37.498 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/30' 2023-10-30T13:41:37.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/14' 2023-10-30T13:41:37.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/18' 2023-10-30T13:41:37.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/13' 2023-10-30T13:41:37.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/33' 2023-10-30T13:41:37.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/17' 2023-10-30T13:41:37.499 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/35' 2023-10-30T13:41:37.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './149/37' 2023-10-30T13:41:37.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './149' 2023-10-30T13:41:37.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/19' 2023-10-30T13:41:37.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/15' 2023-10-30T13:41:37.500 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/31' 2023-10-30T13:41:37.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/36' 2023-10-30T13:41:37.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/20' 2023-10-30T13:41:37.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/12' 2023-10-30T13:41:37.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/39' 2023-10-30T13:41:37.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/10' 2023-10-30T13:41:37.501 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/34' 2023-10-30T13:41:37.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/38' 2023-10-30T13:41:37.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/11' 2023-10-30T13:41:37.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/16' 2023-10-30T13:41:37.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/32' 2023-10-30T13:41:37.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/30' 2023-10-30T13:41:37.502 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/14' 2023-10-30T13:41:37.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/18' 2023-10-30T13:41:37.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/13' 2023-10-30T13:41:37.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/33' 2023-10-30T13:41:37.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/17' 2023-10-30T13:41:37.503 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/35' 2023-10-30T13:41:37.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './150/37' 2023-10-30T13:41:37.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './150' 2023-10-30T13:41:37.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/19' 2023-10-30T13:41:37.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/15' 2023-10-30T13:41:37.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/31' 2023-10-30T13:41:37.504 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/36' 2023-10-30T13:41:37.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/20' 2023-10-30T13:41:37.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/12' 2023-10-30T13:41:37.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/39' 2023-10-30T13:41:37.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/10' 2023-10-30T13:41:37.505 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/34' 2023-10-30T13:41:37.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/38' 2023-10-30T13:41:37.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/11' 2023-10-30T13:41:37.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/16' 2023-10-30T13:41:37.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/32' 2023-10-30T13:41:37.506 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/30' 2023-10-30T13:41:37.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/14' 2023-10-30T13:41:37.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/18' 2023-10-30T13:41:37.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/13' 2023-10-30T13:41:37.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/33' 2023-10-30T13:41:37.507 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/17' 2023-10-30T13:41:37.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/35' 2023-10-30T13:41:37.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './151/37' 2023-10-30T13:41:37.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './151' 2023-10-30T13:41:37.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/19' 2023-10-30T13:41:37.508 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/15' 2023-10-30T13:41:37.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/31' 2023-10-30T13:41:37.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/36' 2023-10-30T13:41:37.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/20' 2023-10-30T13:41:37.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/12' 2023-10-30T13:41:37.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/39' 2023-10-30T13:41:37.509 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/10' 2023-10-30T13:41:37.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/34' 2023-10-30T13:41:37.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/38' 2023-10-30T13:41:37.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/11' 2023-10-30T13:41:37.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/16' 2023-10-30T13:41:37.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/32' 2023-10-30T13:41:37.510 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/30' 2023-10-30T13:41:37.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/14' 2023-10-30T13:41:37.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/18' 2023-10-30T13:41:37.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/13' 2023-10-30T13:41:37.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/33' 2023-10-30T13:41:37.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/17' 2023-10-30T13:41:37.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/35' 2023-10-30T13:41:37.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './152/37' 2023-10-30T13:41:37.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './152' 2023-10-30T13:41:37.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/19' 2023-10-30T13:41:37.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/15' 2023-10-30T13:41:37.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/31' 2023-10-30T13:41:37.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/36' 2023-10-30T13:41:37.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/20' 2023-10-30T13:41:37.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/12' 2023-10-30T13:41:37.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/39' 2023-10-30T13:41:37.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/10' 2023-10-30T13:41:37.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/34' 2023-10-30T13:41:37.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/38' 2023-10-30T13:41:37.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/11' 2023-10-30T13:41:37.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/16' 2023-10-30T13:41:37.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/32' 2023-10-30T13:41:37.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/30' 2023-10-30T13:41:37.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/14' 2023-10-30T13:41:37.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/18' 2023-10-30T13:41:37.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/13' 2023-10-30T13:41:37.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/33' 2023-10-30T13:41:37.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/17' 2023-10-30T13:41:37.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/35' 2023-10-30T13:41:37.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './153/37' 2023-10-30T13:41:37.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './153' 2023-10-30T13:41:37.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/19' 2023-10-30T13:41:37.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/15' 2023-10-30T13:41:37.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/31' 2023-10-30T13:41:37.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/36' 2023-10-30T13:41:37.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/20' 2023-10-30T13:41:37.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/12' 2023-10-30T13:41:37.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/39' 2023-10-30T13:41:37.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/10' 2023-10-30T13:41:37.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/34' 2023-10-30T13:41:37.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/38' 2023-10-30T13:41:37.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/11' 2023-10-30T13:41:37.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/16' 2023-10-30T13:41:37.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/32' 2023-10-30T13:41:37.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/30' 2023-10-30T13:41:37.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/14' 2023-10-30T13:41:37.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/18' 2023-10-30T13:41:37.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/13' 2023-10-30T13:41:37.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/33' 2023-10-30T13:41:37.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/17' 2023-10-30T13:41:37.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/35' 2023-10-30T13:41:37.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './154/37' 2023-10-30T13:41:37.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './154' 2023-10-30T13:41:37.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/19' 2023-10-30T13:41:37.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/15' 2023-10-30T13:41:37.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/31' 2023-10-30T13:41:37.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/36' 2023-10-30T13:41:37.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/20' 2023-10-30T13:41:37.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/12' 2023-10-30T13:41:37.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/39' 2023-10-30T13:41:37.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/10' 2023-10-30T13:41:37.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/34' 2023-10-30T13:41:37.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/38' 2023-10-30T13:41:37.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/11' 2023-10-30T13:41:37.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/16' 2023-10-30T13:41:37.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/32' 2023-10-30T13:41:37.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/30' 2023-10-30T13:41:37.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/14' 2023-10-30T13:41:37.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/18' 2023-10-30T13:41:37.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/13' 2023-10-30T13:41:37.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/33' 2023-10-30T13:41:37.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/17' 2023-10-30T13:41:37.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/35' 2023-10-30T13:41:37.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './155/37' 2023-10-30T13:41:37.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './155' 2023-10-30T13:41:37.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/19' 2023-10-30T13:41:37.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/15' 2023-10-30T13:41:37.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/31' 2023-10-30T13:41:37.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/36' 2023-10-30T13:41:37.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/20' 2023-10-30T13:41:37.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/12' 2023-10-30T13:41:37.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/39' 2023-10-30T13:41:37.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/10' 2023-10-30T13:41:37.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/34' 2023-10-30T13:41:37.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/38' 2023-10-30T13:41:37.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/11' 2023-10-30T13:41:37.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/16' 2023-10-30T13:41:37.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/32' 2023-10-30T13:41:37.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/30' 2023-10-30T13:41:37.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/14' 2023-10-30T13:41:37.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/18' 2023-10-30T13:41:37.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/13' 2023-10-30T13:41:37.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/33' 2023-10-30T13:41:37.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/17' 2023-10-30T13:41:37.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/35' 2023-10-30T13:41:37.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './156/37' 2023-10-30T13:41:37.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './156' 2023-10-30T13:41:37.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/19' 2023-10-30T13:41:37.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/15' 2023-10-30T13:41:37.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/31' 2023-10-30T13:41:37.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/36' 2023-10-30T13:41:37.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/20' 2023-10-30T13:41:37.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/12' 2023-10-30T13:41:37.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/39' 2023-10-30T13:41:37.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/10' 2023-10-30T13:41:37.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/34' 2023-10-30T13:41:37.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/38' 2023-10-30T13:41:37.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/11' 2023-10-30T13:41:37.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/16' 2023-10-30T13:41:37.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/32' 2023-10-30T13:41:37.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/30' 2023-10-30T13:41:37.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/14' 2023-10-30T13:41:37.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/18' 2023-10-30T13:41:37.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/13' 2023-10-30T13:41:37.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/33' 2023-10-30T13:41:41.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/17' 2023-10-30T13:41:41.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/35' 2023-10-30T13:41:41.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './157/37' 2023-10-30T13:41:41.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './157' 2023-10-30T13:41:41.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/19' 2023-10-30T13:41:41.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/15' 2023-10-30T13:41:41.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/31' 2023-10-30T13:41:41.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/36' 2023-10-30T13:41:41.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/20' 2023-10-30T13:41:41.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/12' 2023-10-30T13:41:41.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/39' 2023-10-30T13:41:41.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/10' 2023-10-30T13:41:41.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/34' 2023-10-30T13:41:41.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/38' 2023-10-30T13:41:41.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/11' 2023-10-30T13:41:41.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/16' 2023-10-30T13:41:41.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/32' 2023-10-30T13:41:41.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/30' 2023-10-30T13:41:41.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/14' 2023-10-30T13:41:41.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/18' 2023-10-30T13:41:41.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/13' 2023-10-30T13:41:41.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/33' 2023-10-30T13:41:41.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/17' 2023-10-30T13:41:41.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/35' 2023-10-30T13:41:41.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './158/37' 2023-10-30T13:41:41.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './158' 2023-10-30T13:41:41.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/19' 2023-10-30T13:41:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/15' 2023-10-30T13:41:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/31' 2023-10-30T13:41:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/36' 2023-10-30T13:41:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/20' 2023-10-30T13:41:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/12' 2023-10-30T13:41:41.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/39' 2023-10-30T13:41:41.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/10' 2023-10-30T13:41:41.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/34' 2023-10-30T13:41:41.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/38' 2023-10-30T13:41:41.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/11' 2023-10-30T13:41:41.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/16' 2023-10-30T13:41:41.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/32' 2023-10-30T13:41:41.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/30' 2023-10-30T13:41:41.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/14' 2023-10-30T13:41:41.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/18' 2023-10-30T13:41:41.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/13' 2023-10-30T13:41:41.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/33' 2023-10-30T13:41:41.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/17' 2023-10-30T13:41:41.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/35' 2023-10-30T13:41:41.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './159/37' 2023-10-30T13:41:41.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './159' 2023-10-30T13:41:41.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/19' 2023-10-30T13:41:41.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/15' 2023-10-30T13:41:41.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/31' 2023-10-30T13:41:41.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/36' 2023-10-30T13:41:41.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/20' 2023-10-30T13:41:41.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/12' 2023-10-30T13:41:41.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/39' 2023-10-30T13:41:41.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/10' 2023-10-30T13:41:41.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/34' 2023-10-30T13:41:41.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/38' 2023-10-30T13:41:41.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/11' 2023-10-30T13:41:41.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/16' 2023-10-30T13:41:41.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/32' 2023-10-30T13:41:41.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/30' 2023-10-30T13:41:41.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/14' 2023-10-30T13:41:41.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/18' 2023-10-30T13:41:41.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/13' 2023-10-30T13:41:41.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/33' 2023-10-30T13:41:41.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/17' 2023-10-30T13:41:41.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/35' 2023-10-30T13:41:41.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './160/37' 2023-10-30T13:41:41.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './160' 2023-10-30T13:41:41.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/19' 2023-10-30T13:41:41.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/15' 2023-10-30T13:41:41.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/31' 2023-10-30T13:41:41.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/36' 2023-10-30T13:41:41.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/20' 2023-10-30T13:41:41.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/12' 2023-10-30T13:41:41.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/39' 2023-10-30T13:41:41.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/10' 2023-10-30T13:41:41.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/34' 2023-10-30T13:41:41.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/38' 2023-10-30T13:41:41.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/11' 2023-10-30T13:41:41.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/16' 2023-10-30T13:41:41.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/32' 2023-10-30T13:41:41.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/30' 2023-10-30T13:41:41.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/14' 2023-10-30T13:41:41.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/18' 2023-10-30T13:41:41.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/13' 2023-10-30T13:41:41.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/33' 2023-10-30T13:41:41.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/17' 2023-10-30T13:41:41.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/35' 2023-10-30T13:41:41.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './161/37' 2023-10-30T13:41:41.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './161' 2023-10-30T13:41:41.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/19' 2023-10-30T13:41:41.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/15' 2023-10-30T13:41:41.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/31' 2023-10-30T13:41:41.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/36' 2023-10-30T13:41:41.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/20' 2023-10-30T13:41:41.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/12' 2023-10-30T13:41:41.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/39' 2023-10-30T13:41:41.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/10' 2023-10-30T13:41:41.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/34' 2023-10-30T13:41:41.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/38' 2023-10-30T13:41:41.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/11' 2023-10-30T13:41:41.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/16' 2023-10-30T13:41:41.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/32' 2023-10-30T13:41:41.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/30' 2023-10-30T13:41:41.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/14' 2023-10-30T13:41:41.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/18' 2023-10-30T13:41:41.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/13' 2023-10-30T13:41:41.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/33' 2023-10-30T13:41:41.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/17' 2023-10-30T13:41:41.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/35' 2023-10-30T13:41:41.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './162/37' 2023-10-30T13:41:41.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './162' 2023-10-30T13:41:41.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/19' 2023-10-30T13:41:41.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/15' 2023-10-30T13:41:41.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/31' 2023-10-30T13:41:41.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/36' 2023-10-30T13:41:41.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/20' 2023-10-30T13:41:41.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/12' 2023-10-30T13:41:41.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/39' 2023-10-30T13:41:41.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/10' 2023-10-30T13:41:41.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/34' 2023-10-30T13:41:41.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/38' 2023-10-30T13:41:41.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/11' 2023-10-30T13:41:41.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/16' 2023-10-30T13:41:41.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/32' 2023-10-30T13:41:41.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/30' 2023-10-30T13:41:41.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/14' 2023-10-30T13:41:41.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/18' 2023-10-30T13:41:41.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/13' 2023-10-30T13:41:41.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/33' 2023-10-30T13:41:41.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/17' 2023-10-30T13:41:41.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/35' 2023-10-30T13:41:41.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './163/37' 2023-10-30T13:41:41.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './163' 2023-10-30T13:41:41.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/19' 2023-10-30T13:41:41.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/15' 2023-10-30T13:41:41.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/31' 2023-10-30T13:41:41.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/36' 2023-10-30T13:41:41.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/20' 2023-10-30T13:41:41.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/12' 2023-10-30T13:41:41.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/39' 2023-10-30T13:41:41.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/10' 2023-10-30T13:41:41.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/34' 2023-10-30T13:41:41.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/38' 2023-10-30T13:41:41.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/11' 2023-10-30T13:41:41.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/16' 2023-10-30T13:41:41.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/32' 2023-10-30T13:41:41.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/30' 2023-10-30T13:41:41.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/14' 2023-10-30T13:41:41.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/18' 2023-10-30T13:41:41.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/13' 2023-10-30T13:41:41.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/33' 2023-10-30T13:41:41.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/17' 2023-10-30T13:41:41.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/35' 2023-10-30T13:41:41.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './164/37' 2023-10-30T13:41:41.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './164' 2023-10-30T13:41:41.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/19' 2023-10-30T13:41:41.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/15' 2023-10-30T13:41:41.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/31' 2023-10-30T13:41:41.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/36' 2023-10-30T13:41:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/20' 2023-10-30T13:41:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/12' 2023-10-30T13:41:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/39' 2023-10-30T13:41:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/10' 2023-10-30T13:41:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/34' 2023-10-30T13:41:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/38' 2023-10-30T13:41:41.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/11' 2023-10-30T13:41:41.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/16' 2023-10-30T13:41:41.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/32' 2023-10-30T13:41:41.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/30' 2023-10-30T13:41:41.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/14' 2023-10-30T13:41:41.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/18' 2023-10-30T13:41:41.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/13' 2023-10-30T13:41:41.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/33' 2023-10-30T13:41:41.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/17' 2023-10-30T13:41:41.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/35' 2023-10-30T13:41:41.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './165/37' 2023-10-30T13:41:41.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './165' 2023-10-30T13:41:41.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/19' 2023-10-30T13:41:41.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/15' 2023-10-30T13:41:41.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/31' 2023-10-30T13:41:41.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/36' 2023-10-30T13:41:41.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/20' 2023-10-30T13:41:41.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/12' 2023-10-30T13:41:41.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/39' 2023-10-30T13:41:41.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/10' 2023-10-30T13:41:41.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/34' 2023-10-30T13:41:41.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/38' 2023-10-30T13:41:41.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/11' 2023-10-30T13:41:41.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/16' 2023-10-30T13:41:41.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/32' 2023-10-30T13:41:41.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/30' 2023-10-30T13:41:41.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/14' 2023-10-30T13:41:41.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/18' 2023-10-30T13:41:41.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/13' 2023-10-30T13:41:41.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/33' 2023-10-30T13:41:41.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/17' 2023-10-30T13:41:41.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/35' 2023-10-30T13:41:41.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed './166/37' 2023-10-30T13:41:41.987 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './166' 2023-10-30T13:41:41.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/19' 2023-10-30T13:41:41.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/15' 2023-10-30T13:41:41.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/31' 2023-10-30T13:41:41.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/36' 2023-10-30T13:41:41.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/20' 2023-10-30T13:41:41.988 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/12' 2023-10-30T13:41:41.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/39' 2023-10-30T13:41:41.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/10' 2023-10-30T13:41:41.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/34' 2023-10-30T13:41:41.989 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/38' 2023-10-30T13:41:45.289 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/11' 2023-10-30T13:41:45.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/16' 2023-10-30T13:41:45.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/32' 2023-10-30T13:41:45.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/30' 2023-10-30T13:41:45.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/14' 2023-10-30T13:41:45.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/18' 2023-10-30T13:41:45.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/13' 2023-10-30T13:41:45.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/33' 2023-10-30T13:41:45.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/17' 2023-10-30T13:41:45.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/35' 2023-10-30T13:41:45.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './167/37' 2023-10-30T13:41:45.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './167' 2023-10-30T13:41:45.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/19' 2023-10-30T13:41:45.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/15' 2023-10-30T13:41:45.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/31' 2023-10-30T13:41:45.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/36' 2023-10-30T13:41:45.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/20' 2023-10-30T13:41:45.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/12' 2023-10-30T13:41:45.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/39' 2023-10-30T13:41:45.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/10' 2023-10-30T13:41:45.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/34' 2023-10-30T13:41:45.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/38' 2023-10-30T13:41:45.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/11' 2023-10-30T13:41:45.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/16' 2023-10-30T13:41:45.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/32' 2023-10-30T13:41:45.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/30' 2023-10-30T13:41:45.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/14' 2023-10-30T13:41:45.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/18' 2023-10-30T13:41:45.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/13' 2023-10-30T13:41:45.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/33' 2023-10-30T13:41:45.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/17' 2023-10-30T13:41:45.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/35' 2023-10-30T13:41:45.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './168/37' 2023-10-30T13:41:45.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './168' 2023-10-30T13:41:45.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/19' 2023-10-30T13:41:45.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/15' 2023-10-30T13:41:45.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/31' 2023-10-30T13:41:45.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/36' 2023-10-30T13:41:45.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/20' 2023-10-30T13:41:45.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/12' 2023-10-30T13:41:45.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/39' 2023-10-30T13:41:45.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/10' 2023-10-30T13:41:45.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/34' 2023-10-30T13:41:45.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/38' 2023-10-30T13:41:45.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/11' 2023-10-30T13:41:45.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/16' 2023-10-30T13:41:45.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/32' 2023-10-30T13:41:45.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/30' 2023-10-30T13:41:45.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/14' 2023-10-30T13:41:45.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/18' 2023-10-30T13:41:45.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/13' 2023-10-30T13:41:45.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/33' 2023-10-30T13:41:45.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/17' 2023-10-30T13:41:45.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/35' 2023-10-30T13:41:45.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './169/37' 2023-10-30T13:41:45.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './169' 2023-10-30T13:41:45.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/19' 2023-10-30T13:41:45.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/15' 2023-10-30T13:41:45.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/31' 2023-10-30T13:41:45.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/36' 2023-10-30T13:41:45.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/20' 2023-10-30T13:41:45.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/12' 2023-10-30T13:41:45.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/39' 2023-10-30T13:41:45.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/10' 2023-10-30T13:41:45.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/34' 2023-10-30T13:41:45.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/38' 2023-10-30T13:41:45.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/11' 2023-10-30T13:41:45.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/16' 2023-10-30T13:41:45.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/32' 2023-10-30T13:41:45.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/30' 2023-10-30T13:41:45.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/14' 2023-10-30T13:41:45.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/18' 2023-10-30T13:41:45.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/13' 2023-10-30T13:41:45.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/33' 2023-10-30T13:41:45.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/17' 2023-10-30T13:41:45.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/35' 2023-10-30T13:41:45.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './170/37' 2023-10-30T13:41:45.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './170' 2023-10-30T13:41:45.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/19' 2023-10-30T13:41:45.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/15' 2023-10-30T13:41:45.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/31' 2023-10-30T13:41:45.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/36' 2023-10-30T13:41:45.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/20' 2023-10-30T13:41:45.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/12' 2023-10-30T13:41:45.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/39' 2023-10-30T13:41:45.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/10' 2023-10-30T13:41:45.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/34' 2023-10-30T13:41:45.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/38' 2023-10-30T13:41:45.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/11' 2023-10-30T13:41:45.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/16' 2023-10-30T13:41:45.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/32' 2023-10-30T13:41:45.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/30' 2023-10-30T13:41:45.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/14' 2023-10-30T13:41:45.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/18' 2023-10-30T13:41:45.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/13' 2023-10-30T13:41:45.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/33' 2023-10-30T13:41:45.313 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/17' 2023-10-30T13:41:45.313 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/35' 2023-10-30T13:41:45.313 INFO:tasks.workunit.client.0.smithi031.stdout:removed './171/37' 2023-10-30T13:41:45.313 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './171' 2023-10-30T13:41:45.314 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/19' 2023-10-30T13:41:45.314 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/15' 2023-10-30T13:41:45.314 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/31' 2023-10-30T13:41:45.314 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/36' 2023-10-30T13:41:45.315 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/20' 2023-10-30T13:41:45.315 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/12' 2023-10-30T13:41:45.315 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/39' 2023-10-30T13:41:45.315 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/10' 2023-10-30T13:41:45.316 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/34' 2023-10-30T13:41:45.316 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/38' 2023-10-30T13:41:45.316 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/11' 2023-10-30T13:41:45.316 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/16' 2023-10-30T13:41:45.316 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/32' 2023-10-30T13:41:45.317 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/30' 2023-10-30T13:41:45.317 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/14' 2023-10-30T13:41:45.317 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/18' 2023-10-30T13:41:45.317 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/13' 2023-10-30T13:41:45.317 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/33' 2023-10-30T13:41:45.318 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/17' 2023-10-30T13:41:45.318 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/35' 2023-10-30T13:41:45.318 INFO:tasks.workunit.client.0.smithi031.stdout:removed './172/37' 2023-10-30T13:41:45.318 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './172' 2023-10-30T13:41:45.319 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/19' 2023-10-30T13:41:45.319 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/15' 2023-10-30T13:41:45.319 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/31' 2023-10-30T13:41:45.319 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/36' 2023-10-30T13:41:45.320 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/20' 2023-10-30T13:41:45.320 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/12' 2023-10-30T13:41:45.320 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/39' 2023-10-30T13:41:45.320 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/10' 2023-10-30T13:41:45.320 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/34' 2023-10-30T13:41:45.321 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/38' 2023-10-30T13:41:45.321 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/11' 2023-10-30T13:41:45.321 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/16' 2023-10-30T13:41:45.321 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/32' 2023-10-30T13:41:45.321 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/30' 2023-10-30T13:41:45.322 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/14' 2023-10-30T13:41:45.322 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/18' 2023-10-30T13:41:45.322 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/13' 2023-10-30T13:41:45.322 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/33' 2023-10-30T13:41:45.322 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/17' 2023-10-30T13:41:45.323 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/35' 2023-10-30T13:41:45.323 INFO:tasks.workunit.client.0.smithi031.stdout:removed './173/37' 2023-10-30T13:41:45.323 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './173' 2023-10-30T13:41:45.323 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/19' 2023-10-30T13:41:45.323 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/15' 2023-10-30T13:41:45.324 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/31' 2023-10-30T13:41:45.324 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/36' 2023-10-30T13:41:45.324 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/20' 2023-10-30T13:41:45.324 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/12' 2023-10-30T13:41:45.324 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/39' 2023-10-30T13:41:45.325 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/10' 2023-10-30T13:41:45.325 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/34' 2023-10-30T13:41:45.325 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/38' 2023-10-30T13:41:45.325 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/11' 2023-10-30T13:41:45.326 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/16' 2023-10-30T13:41:45.326 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/32' 2023-10-30T13:41:45.326 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/30' 2023-10-30T13:41:45.326 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/14' 2023-10-30T13:41:45.326 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/18' 2023-10-30T13:41:45.327 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/13' 2023-10-30T13:41:45.327 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/33' 2023-10-30T13:41:45.327 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/17' 2023-10-30T13:41:45.327 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/35' 2023-10-30T13:41:45.328 INFO:tasks.workunit.client.0.smithi031.stdout:removed './174/37' 2023-10-30T13:41:45.328 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './174' 2023-10-30T13:41:45.328 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/19' 2023-10-30T13:41:45.328 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/15' 2023-10-30T13:41:45.328 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/31' 2023-10-30T13:41:45.329 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/36' 2023-10-30T13:41:45.329 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/20' 2023-10-30T13:41:45.329 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/12' 2023-10-30T13:41:45.329 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/39' 2023-10-30T13:41:45.330 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/10' 2023-10-30T13:41:45.330 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/34' 2023-10-30T13:41:45.330 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/38' 2023-10-30T13:41:45.330 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/11' 2023-10-30T13:41:45.330 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/16' 2023-10-30T13:41:45.331 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/32' 2023-10-30T13:41:45.331 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/30' 2023-10-30T13:41:45.331 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/14' 2023-10-30T13:41:45.331 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/18' 2023-10-30T13:41:45.331 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/13' 2023-10-30T13:41:45.332 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/33' 2023-10-30T13:41:45.332 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/17' 2023-10-30T13:41:45.332 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/35' 2023-10-30T13:41:45.332 INFO:tasks.workunit.client.0.smithi031.stdout:removed './175/37' 2023-10-30T13:41:45.332 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './175' 2023-10-30T13:41:45.333 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/19' 2023-10-30T13:41:45.333 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/15' 2023-10-30T13:41:45.333 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/31' 2023-10-30T13:41:45.333 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/36' 2023-10-30T13:41:45.334 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/20' 2023-10-30T13:41:45.334 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/12' 2023-10-30T13:41:45.334 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/39' 2023-10-30T13:41:45.334 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/10' 2023-10-30T13:41:45.334 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/34' 2023-10-30T13:41:45.335 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/38' 2023-10-30T13:41:45.335 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/11' 2023-10-30T13:41:45.335 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/16' 2023-10-30T13:41:45.335 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/32' 2023-10-30T13:41:45.335 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/30' 2023-10-30T13:41:45.336 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/14' 2023-10-30T13:41:45.336 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/18' 2023-10-30T13:41:45.336 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/13' 2023-10-30T13:41:45.336 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/33' 2023-10-30T13:41:45.336 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/17' 2023-10-30T13:41:45.337 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/35' 2023-10-30T13:41:45.337 INFO:tasks.workunit.client.0.smithi031.stdout:removed './176/37' 2023-10-30T13:41:45.337 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './176' 2023-10-30T13:41:45.337 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/19' 2023-10-30T13:41:45.337 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/15' 2023-10-30T13:41:49.588 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/31' 2023-10-30T13:41:49.588 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/36' 2023-10-30T13:41:49.589 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/20' 2023-10-30T13:41:49.589 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/12' 2023-10-30T13:41:49.589 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/39' 2023-10-30T13:41:49.589 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/10' 2023-10-30T13:41:49.589 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/34' 2023-10-30T13:41:49.590 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/38' 2023-10-30T13:41:49.590 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/11' 2023-10-30T13:41:49.590 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/16' 2023-10-30T13:41:49.590 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/32' 2023-10-30T13:41:49.590 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/30' 2023-10-30T13:41:49.590 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/14' 2023-10-30T13:41:49.591 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/18' 2023-10-30T13:41:49.591 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/13' 2023-10-30T13:41:49.591 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/33' 2023-10-30T13:41:49.591 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/17' 2023-10-30T13:41:49.591 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/35' 2023-10-30T13:41:49.592 INFO:tasks.workunit.client.0.smithi031.stdout:removed './177/37' 2023-10-30T13:41:49.592 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './177' 2023-10-30T13:41:49.592 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/19' 2023-10-30T13:41:49.592 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/15' 2023-10-30T13:41:49.592 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/31' 2023-10-30T13:41:49.592 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/36' 2023-10-30T13:41:49.593 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/20' 2023-10-30T13:41:49.593 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/12' 2023-10-30T13:41:49.593 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/39' 2023-10-30T13:41:49.593 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/10' 2023-10-30T13:41:49.593 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/34' 2023-10-30T13:41:49.593 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/38' 2023-10-30T13:41:49.594 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/11' 2023-10-30T13:41:49.594 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/16' 2023-10-30T13:41:49.594 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/32' 2023-10-30T13:41:49.594 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/30' 2023-10-30T13:41:49.594 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/14' 2023-10-30T13:41:49.595 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/18' 2023-10-30T13:41:49.595 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/13' 2023-10-30T13:41:49.595 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/33' 2023-10-30T13:41:49.595 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/17' 2023-10-30T13:41:49.595 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/35' 2023-10-30T13:41:49.595 INFO:tasks.workunit.client.0.smithi031.stdout:removed './178/37' 2023-10-30T13:41:49.596 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './178' 2023-10-30T13:41:49.596 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/19' 2023-10-30T13:41:49.596 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/15' 2023-10-30T13:41:49.596 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/31' 2023-10-30T13:41:49.596 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/36' 2023-10-30T13:41:49.597 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/20' 2023-10-30T13:41:49.597 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/12' 2023-10-30T13:41:49.597 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/39' 2023-10-30T13:41:49.597 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/10' 2023-10-30T13:41:49.597 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/34' 2023-10-30T13:41:49.598 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/38' 2023-10-30T13:41:49.598 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/11' 2023-10-30T13:41:49.598 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/16' 2023-10-30T13:41:49.598 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/32' 2023-10-30T13:41:49.598 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/30' 2023-10-30T13:41:49.598 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/14' 2023-10-30T13:41:49.599 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/18' 2023-10-30T13:41:49.599 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/13' 2023-10-30T13:41:49.599 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/33' 2023-10-30T13:41:49.599 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/17' 2023-10-30T13:41:49.599 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/35' 2023-10-30T13:41:49.599 INFO:tasks.workunit.client.0.smithi031.stdout:removed './179/37' 2023-10-30T13:41:49.600 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './179' 2023-10-30T13:41:49.600 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/19' 2023-10-30T13:41:49.600 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/15' 2023-10-30T13:41:49.600 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/31' 2023-10-30T13:41:49.600 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/36' 2023-10-30T13:41:49.600 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/20' 2023-10-30T13:41:49.601 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/12' 2023-10-30T13:41:49.601 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/39' 2023-10-30T13:41:49.601 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/10' 2023-10-30T13:41:49.601 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/34' 2023-10-30T13:41:49.601 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/38' 2023-10-30T13:41:49.601 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/11' 2023-10-30T13:41:49.602 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/16' 2023-10-30T13:41:49.602 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/32' 2023-10-30T13:41:49.602 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/30' 2023-10-30T13:41:49.602 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/14' 2023-10-30T13:41:49.602 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/18' 2023-10-30T13:41:49.603 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/13' 2023-10-30T13:41:49.603 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/33' 2023-10-30T13:41:49.603 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/17' 2023-10-30T13:41:49.603 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/35' 2023-10-30T13:41:49.603 INFO:tasks.workunit.client.0.smithi031.stdout:removed './180/37' 2023-10-30T13:41:49.604 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './180' 2023-10-30T13:41:49.604 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/19' 2023-10-30T13:41:49.604 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/15' 2023-10-30T13:41:49.604 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/31' 2023-10-30T13:41:49.604 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/36' 2023-10-30T13:41:49.604 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/20' 2023-10-30T13:41:49.605 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/12' 2023-10-30T13:41:49.605 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/39' 2023-10-30T13:41:49.605 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/10' 2023-10-30T13:41:49.605 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/34' 2023-10-30T13:41:49.605 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/38' 2023-10-30T13:41:49.605 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/11' 2023-10-30T13:41:49.606 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/16' 2023-10-30T13:41:49.606 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/32' 2023-10-30T13:41:49.606 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/30' 2023-10-30T13:41:49.606 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/14' 2023-10-30T13:41:49.606 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/18' 2023-10-30T13:41:49.607 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/13' 2023-10-30T13:41:49.607 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/33' 2023-10-30T13:41:49.607 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/17' 2023-10-30T13:41:49.607 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/35' 2023-10-30T13:41:49.607 INFO:tasks.workunit.client.0.smithi031.stdout:removed './181/37' 2023-10-30T13:41:49.608 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './181' 2023-10-30T13:41:49.608 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/19' 2023-10-30T13:41:49.608 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/15' 2023-10-30T13:41:49.608 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/31' 2023-10-30T13:41:49.608 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/36' 2023-10-30T13:41:49.609 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/20' 2023-10-30T13:41:49.609 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/12' 2023-10-30T13:41:49.609 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/39' 2023-10-30T13:41:49.609 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/10' 2023-10-30T13:41:49.609 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/34' 2023-10-30T13:41:49.609 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/38' 2023-10-30T13:41:49.610 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/11' 2023-10-30T13:41:49.610 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/16' 2023-10-30T13:41:49.610 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/32' 2023-10-30T13:41:49.610 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/30' 2023-10-30T13:41:49.610 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/14' 2023-10-30T13:41:49.611 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/18' 2023-10-30T13:41:49.611 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/13' 2023-10-30T13:41:49.611 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/33' 2023-10-30T13:41:49.611 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/17' 2023-10-30T13:41:49.611 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/35' 2023-10-30T13:41:49.611 INFO:tasks.workunit.client.0.smithi031.stdout:removed './182/37' 2023-10-30T13:41:49.612 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './182' 2023-10-30T13:41:49.612 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/19' 2023-10-30T13:41:49.612 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/15' 2023-10-30T13:41:49.612 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/31' 2023-10-30T13:41:49.612 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/36' 2023-10-30T13:41:49.613 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/20' 2023-10-30T13:41:49.613 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/12' 2023-10-30T13:41:49.613 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/39' 2023-10-30T13:41:49.613 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/10' 2023-10-30T13:41:49.613 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/34' 2023-10-30T13:41:49.613 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/38' 2023-10-30T13:41:49.614 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/11' 2023-10-30T13:41:49.614 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/16' 2023-10-30T13:41:49.614 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/32' 2023-10-30T13:41:49.614 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/30' 2023-10-30T13:41:49.614 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/14' 2023-10-30T13:41:49.615 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/18' 2023-10-30T13:41:49.615 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/13' 2023-10-30T13:41:49.615 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/33' 2023-10-30T13:41:49.615 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/17' 2023-10-30T13:41:49.615 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/35' 2023-10-30T13:41:49.616 INFO:tasks.workunit.client.0.smithi031.stdout:removed './183/37' 2023-10-30T13:41:49.616 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './183' 2023-10-30T13:41:49.616 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/19' 2023-10-30T13:41:49.616 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/15' 2023-10-30T13:41:49.616 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/31' 2023-10-30T13:41:49.616 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/36' 2023-10-30T13:41:49.617 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/20' 2023-10-30T13:41:49.617 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/12' 2023-10-30T13:41:49.617 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/39' 2023-10-30T13:41:49.617 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/10' 2023-10-30T13:41:49.617 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/34' 2023-10-30T13:41:49.618 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/38' 2023-10-30T13:41:49.618 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/11' 2023-10-30T13:41:49.618 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/16' 2023-10-30T13:41:49.618 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/32' 2023-10-30T13:41:49.618 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/30' 2023-10-30T13:41:49.618 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/14' 2023-10-30T13:41:49.619 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/18' 2023-10-30T13:41:49.619 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/13' 2023-10-30T13:41:49.619 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/33' 2023-10-30T13:41:49.619 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/17' 2023-10-30T13:41:49.619 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/35' 2023-10-30T13:41:49.620 INFO:tasks.workunit.client.0.smithi031.stdout:removed './184/37' 2023-10-30T13:41:49.620 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './184' 2023-10-30T13:41:49.620 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/19' 2023-10-30T13:41:49.620 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/15' 2023-10-30T13:41:49.620 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/31' 2023-10-30T13:41:49.621 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/36' 2023-10-30T13:41:49.621 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/20' 2023-10-30T13:41:49.621 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/12' 2023-10-30T13:41:49.621 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/39' 2023-10-30T13:41:49.621 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/10' 2023-10-30T13:41:49.621 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/34' 2023-10-30T13:41:49.622 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/38' 2023-10-30T13:41:49.622 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/11' 2023-10-30T13:41:49.622 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/16' 2023-10-30T13:41:49.622 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/32' 2023-10-30T13:41:49.622 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/30' 2023-10-30T13:41:49.623 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/14' 2023-10-30T13:41:49.623 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/18' 2023-10-30T13:41:49.623 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/13' 2023-10-30T13:41:49.623 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/33' 2023-10-30T13:41:49.623 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/17' 2023-10-30T13:41:49.624 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/35' 2023-10-30T13:41:49.624 INFO:tasks.workunit.client.0.smithi031.stdout:removed './185/37' 2023-10-30T13:41:49.625 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './185' 2023-10-30T13:41:49.625 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/19' 2023-10-30T13:41:49.625 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/15' 2023-10-30T13:41:49.626 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/31' 2023-10-30T13:41:49.626 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/36' 2023-10-30T13:41:49.626 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/20' 2023-10-30T13:41:49.626 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/12' 2023-10-30T13:41:49.626 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/39' 2023-10-30T13:41:49.626 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/10' 2023-10-30T13:41:49.627 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/34' 2023-10-30T13:41:49.627 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/38' 2023-10-30T13:41:49.627 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/11' 2023-10-30T13:41:49.627 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/16' 2023-10-30T13:41:49.627 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/32' 2023-10-30T13:41:49.628 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/30' 2023-10-30T13:41:49.628 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/14' 2023-10-30T13:41:49.628 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/18' 2023-10-30T13:41:55.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:55.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:56.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:56.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f829933e640 1 Processor -- start 2023-10-30T13:41:56.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f829933e640 1 -- start start 2023-10-30T13:41:56.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f829933e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 0x7f829415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f829933e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8294159f20 con 0x7f82940c42a0 2023-10-30T13:41:56.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f829933e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829415d620 con 0x7f82940c4a10 2023-10-30T13:41:56.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f829933e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829415dec0 con 0x7f8294153c80 2023-10-30T13:41:56.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82937fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f82940c42a0 0x7f82940ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52386/0 (socket says 172.21.15.31:52386) 2023-10-30T13:41:56.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82937fe640 1 -- 172.21.15.31:0/1469377365 learned_addr learned my addr 172.21.15.31:0/1469377365 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:56.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f8292ffd640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 0x7f829415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947544310 0 0) 0x7f8294159f20 con 0x7f82940c42a0 2023-10-30T13:41:56.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8280003530 con 0x7f82940c42a0 2023-10-30T13:41:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 607553714 0 0) 0x7f8280003530 con 0x7f82940c42a0 2023-10-30T13:41:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 msgr2=0x7f829415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 0x7f829415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 >> v1:172.21.15.33:6789/0 conn(0x7f82940c4a10 legacy=0x7f829415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829415d900 con 0x7f82940c42a0 2023-10-30T13:41:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f828c001a00 con 0x7f82940c42a0 2023-10-30T13:41:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 >> v1:172.21.15.31:6789/0 conn(0x7f82940c42a0 legacy=0x7f82940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 0x7f82800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f82800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8280004d10 0x7f8280005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829415dec0 con 0x7f8280003ef0 2023-10-30T13:41:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828000b9f0 con 0x7f8280004d10 2023-10-30T13:41:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828000bbb0 con 0x7f8294153c80 2023-10-30T13:41:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f828bfff640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f82800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f8292ffd640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 0x7f82800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.225+0000 7f82937fe640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8280004d10 0x7f8280005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f828bfff640 1 -- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 msgr2=0x7f82800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f828bfff640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8294153c80 0x7f82800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f828bfff640 1 -- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8280004d10 msgr2=0x7f8280005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f828bfff640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8280004d10 0x7f8280005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f828bfff640 1 -- 172.21.15.31:0/1469377365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280003530 con 0x7f8280003ef0 2023-10-30T13:41:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f82937fe640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8280004d10 0x7f8280005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:56.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f828bfff640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f82800047d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f82940c4e30 tx=0x7f827c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f45b1cbb422314b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c0ab600 con 0x7f8280003ef0 2023-10-30T13:41:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c0a7050 con 0x7f8280003ef0 2023-10-30T13:41:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f82927fc640 1 -- 172.21.15.31:0/1469377365 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c0ab9c0 con 0x7f8280003ef0 2023-10-30T13:41:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f829933e640 1 -- 172.21.15.31:0/1469377365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 msgr2=0x7f82800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f829933e640 1 --2- 172.21.15.31:0/1469377365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f82800047d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f82940c4e30 tx=0x7f827c01c460 comp rx=0 tx=0).stop 2023-10-30T13:41:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f82937fe640 1 -- 172.21.15.31:0/1469377365 reap_dead start 2023-10-30T13:41:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f829933e640 1 -- 172.21.15.31:0/1469377365 shutdown_connections 2023-10-30T13:41:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.229+0000 7f829933e640 1 -- 172.21.15.31:0/1469377365 >> 172.21.15.31:0/1469377365 conn(0x7f82940715c0 msgr2=0x7f82940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 -- 172.21.15.31:0/1469377365 shutdown_connections 2023-10-30T13:41:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 -- 172.21.15.31:0/1469377365 wait complete. 2023-10-30T13:41:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 Processor -- start 2023-10-30T13:41:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 -- start start 2023-10-30T13:41:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f8294158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829414fe50 0x7f829414ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f829414f2b0 0x7f82941f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829414e8e0 con 0x7f8280003ef0 2023-10-30T13:41:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829414f730 con 0x7f829414fe50 2023-10-30T13:41:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f829933e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829414e2d0 con 0x7f829414f2b0 2023-10-30T13:41:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f8294158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f8294158220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48706/0 (socket says 172.21.15.31:48706) 2023-10-30T13:41:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 -- 172.21.15.31:0/3711795420 learned_addr learned my addr 172.21.15.31:0/3711795420 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f828bfff640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829414fe50 0x7f829414ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f82937fe640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f829414f2b0 0x7f82941f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 -- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f829414f2b0 msgr2=0x7f82941f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f829414f2b0 0x7f82941f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 -- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829414fe50 msgr2=0x7f829414ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829414fe50 0x7f829414ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 -- 172.21.15.31:0/3711795420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8294158e90 con 0x7f8280003ef0 2023-10-30T13:41:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f828bfff640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829414fe50 0x7f829414ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.233+0000 7f8292ffd640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f8294158220 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f82840119d0 tx=0x7f8284011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f828b7fe640 1 -- 172.21.15.31:0/3711795420 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828400faf0 con 0x7f8280003ef0 2023-10-30T13:41:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f828b7fe640 1 -- 172.21.15.31:0/3711795420 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828400fc50 con 0x7f8280003ef0 2023-10-30T13:41:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f828b7fe640 1 -- 172.21.15.31:0/3711795420 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284014440 con 0x7f8280003ef0 2023-10-30T13:41:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82941588d0 con 0x7f8280003ef0 2023-10-30T13:41:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829414e570 con 0x7f8280003ef0 2023-10-30T13:41:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829414e130 con 0x7f8280003ef0 2023-10-30T13:41:56.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f828b7fe640 1 -- 172.21.15.31:0/3711795420 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82840145a0 con 0x7f8280003ef0 2023-10-30T13:41:56.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f828b7fe640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82740664a0 0x7f8274068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f828b7fe640 1 -- 172.21.15.31:0/3711795420 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f8284084950 con 0x7f8280003ef0 2023-10-30T13:41:56.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.237+0000 7f828bfff640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82740664a0 0x7f8274068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.241+0000 7f828bfff640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82740664a0 0x7f8274068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f827c0ad000 tx=0x7f827c001330 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:56.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.241+0000 7f828b7fe640 1 -- 172.21.15.31:0/3711795420 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f829414e130 con 0x7f8280003ef0 2023-10-30T13:41:56.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.453+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8294159b90 con 0x7f8280003ef0 2023-10-30T13:41:56.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.453+0000 7f828b7fe640 1 -- 172.21.15.31:0/3711795420 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f8294159b90 con 0x7f8280003ef0 2023-10-30T13:41:56.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:56.455 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:40.119780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:active","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:56.456 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 39 2023-10-30T13:41:56.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82740664a0 msgr2=0x7f8274068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82740664a0 0x7f8274068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f827c0ad000 tx=0x7f827c001330 comp rx=0 tx=0).stop 2023-10-30T13:41:56.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 msgr2=0x7f8294158220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f8294158220 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f82840119d0 tx=0x7f8284011dd0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 shutdown_connections 2023-10-30T13:41:56.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f82740664a0 0x7f8274068960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f829414f2b0 0x7f82941f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829414fe50 0x7f829414ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 --2- 172.21.15.31:0/3711795420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8280003ef0 0x7f8294158220 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 >> 172.21.15.31:0/3711795420 conn(0x7f82940715c0 msgr2=0x7f8294150660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:56.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 shutdown_connections 2023-10-30T13:41:56.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.457+0000 7f829933e640 1 -- 172.21.15.31:0/3711795420 wait complete. 2023-10-30T13:41:56.476 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 3 -> 1 2023-10-30T13:41:56.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2023-10-30T13:41:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.621+0000 7f95e9f63640 1 Processor -- start 2023-10-30T13:41:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.621+0000 7f95e9f63640 1 -- start start 2023-10-30T13:41:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e9f63640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f95e40718d0 0x7f95e40c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e9f63640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f95e4153300 0x7f95e4153750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e9f63640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e4153c90 con 0x7f95e4071cb0 2023-10-30T13:41:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e9f63640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e40cca10 con 0x7f95e40718d0 2023-10-30T13:41:56.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e9f63640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e415d0a0 con 0x7f95e4153300 2023-10-30T13:41:56.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e8f61640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f95e40718d0 0x7f95e40c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e3fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f95e4071cb0 0x7f95e415a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52394/0 (socket says 172.21.15.31:52394) 2023-10-30T13:41:56.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e3fff640 1 -- 172.21.15.31:0/2481494957 learned_addr learned my addr 172.21.15.31:0/2481494957 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:56.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e9762640 1 --2- 172.21.15.31:0/2481494957 >> v2:172.21.15.33:3300/0 conn(0x7f95e4153300 0x7f95e4153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e8f61640 1 -- 172.21.15.31:0/2481494957 >> v2:172.21.15.33:3300/0 conn(0x7f95e4153300 msgr2=0x7f95e4153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:56.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e8f61640 1 --2- 172.21.15.31:0/2481494957 >> v2:172.21.15.33:3300/0 conn(0x7f95e4153300 0x7f95e4153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e8f61640 1 -- 172.21.15.31:0/2481494957 >> v1:172.21.15.31:6789/0 conn(0x7f95e4071cb0 legacy=0x7f95e415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.625+0000 7f95e8f61640 1 -- 172.21.15.31:0/2481494957 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e415d380 con 0x7f95e40718d0 2023-10-30T13:41:56.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e8f61640 1 --2- 172.21.15.31:0/2481494957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e40718d0 0x7f95e40c9a10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f95e40c5b60 tx=0x7f95d801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7710ebc613b8158 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:56.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e37fe640 1 -- 172.21.15.31:0/2481494957 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80a8950 con 0x7f95e40718d0 2023-10-30T13:41:56.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e37fe640 1 -- 172.21.15.31:0/2481494957 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d80a8ab0 con 0x7f95e40718d0 2023-10-30T13:41:56.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e37fe640 1 -- 172.21.15.31:0/2481494957 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80a8e40 con 0x7f95e40718d0 2023-10-30T13:41:56.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 -- 172.21.15.31:0/2481494957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e40718d0 msgr2=0x7f95e40c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2481494957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e40718d0 0x7f95e40c9a10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f95e40c5b60 tx=0x7f95d801ec50 comp rx=0 tx=0).stop 2023-10-30T13:41:56.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 -- 172.21.15.31:0/2481494957 shutdown_connections 2023-10-30T13:41:56.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2481494957 >> v2:172.21.15.33:3300/0 conn(0x7f95e4153300 0x7f95e4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2481494957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e40718d0 0x7f95e40c9a10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 -- 172.21.15.31:0/2481494957 >> 172.21.15.31:0/2481494957 conn(0x7f95e406ab00 msgr2=0x7f95e4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 -- 172.21.15.31:0/2481494957 shutdown_connections 2023-10-30T13:41:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 -- 172.21.15.31:0/2481494957 wait complete. 2023-10-30T13:41:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 Processor -- start 2023-10-30T13:41:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.629+0000 7f95e9f63640 1 -- start start 2023-10-30T13:41:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95e40718d0 0x7f95e414acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95e4071cb0 0x7f95e41f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e4153300 0x7f95e41f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4151b90 con 0x7f95e4153300 2023-10-30T13:41:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e415c3f0 con 0x7f95e4071cb0 2023-10-30T13:41:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4151dc0 con 0x7f95e40718d0 2023-10-30T13:41:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e4153300 0x7f95e41f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e4153300 0x7f95e41f2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48738/0 (socket says 172.21.15.31:48738) 2023-10-30T13:41:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 -- 172.21.15.31:0/2523841520 learned_addr learned my addr 172.21.15.31:0/2523841520 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e8f61640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95e40718d0 0x7f95e414acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e3fff640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95e4071cb0 0x7f95e41f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 -- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95e40718d0 msgr2=0x7f95e414acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95e40718d0 0x7f95e414acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 -- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95e4071cb0 msgr2=0x7f95e41f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95e4071cb0 0x7f95e41f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 -- 172.21.15.31:0/2523841520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d8009420 con 0x7f95e4153300 2023-10-30T13:41:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e3fff640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95e4071cb0 0x7f95e41f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9762640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e4153300 0x7f95e41f2ae0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f95dc007e60 tx=0x7f95dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e17fa640 1 -- 172.21.15.31:0/2523841520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc014420 con 0x7f95e4153300 2023-10-30T13:41:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e17fa640 1 -- 172.21.15.31:0/2523841520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc0145a0 con 0x7f95e4153300 2023-10-30T13:41:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e17fa640 1 -- 172.21.15.31:0/2523841520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc014950 con 0x7f95e4153300 2023-10-30T13:41:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e415c640 con 0x7f95e4153300 2023-10-30T13:41:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e415cc00 con 0x7f95e4153300 2023-10-30T13:41:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.633+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e404e170 con 0x7f95e4153300 2023-10-30T13:41:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.637+0000 7f95e17fa640 1 -- 172.21.15.31:0/2523841520 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95dc014ab0 con 0x7f95e4153300 2023-10-30T13:41:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.637+0000 7f95e17fa640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95c80664f0 0x7f95c80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.637+0000 7f95e17fa640 1 -- 172.21.15.31:0/2523841520 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f95dc084810 con 0x7f95e4153300 2023-10-30T13:41:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.637+0000 7f95e8f61640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95c80664f0 0x7f95c80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:56.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.637+0000 7f95e8f61640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95c80664f0 0x7f95c80689b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f95d80b2000 tx=0x7f95d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:56.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.641+0000 7f95e17fa640 1 -- 172.21.15.31:0/2523841520 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95dc051b00 con 0x7f95e4153300 2023-10-30T13:41:56.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.833+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f95e40c9f80 con 0x7f95e4153300 2023-10-30T13:41:56.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e17fa640 1 -- 172.21.15.31:0/2523841520 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v40) v1 ==== 105+0+0 (secure 0 0 0) 0x7f95dc055120 con 0x7f95e4153300 2023-10-30T13:41:56.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95c80664f0 msgr2=0x7f95c80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95c80664f0 0x7f95c80689b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f95d80b2000 tx=0x7f95d8008040 comp rx=0 tx=0).stop 2023-10-30T13:41:56.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e4153300 msgr2=0x7f95e41f2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:56.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e4153300 0x7f95e41f2ae0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f95dc007e60 tx=0x7f95dc00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 shutdown_connections 2023-10-30T13:41:56.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95c80664f0 0x7f95c80689b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e4153300 0x7f95e41f2ae0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95e4071cb0 0x7f95e41f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 --2- 172.21.15.31:0/2523841520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95e40718d0 0x7f95e414acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 >> 172.21.15.31:0/2523841520 conn(0x7f95e406ab00 msgr2=0x7f95e40c3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 shutdown_connections 2023-10-30T13:41:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:56.857+0000 7f95e9f63640 1 -- 172.21.15.31:0/2523841520 wait complete. 2023-10-30T13:41:56.879 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:41:56.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.021+0000 7f9bba956640 1 Processor -- start 2023-10-30T13:41:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba956640 1 -- start start 2023-10-30T13:41:57.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba956640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9bb415cd70 0x7f9bb4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:57.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba956640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb415dcc0 con 0x7f9bb415ac70 2023-10-30T13:41:57.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba956640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb4159d80 con 0x7f9bb40c8eb0 2023-10-30T13:41:57.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba956640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb4159800 con 0x7f9bb415cd70 2023-10-30T13:41:57.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba155640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9bb415cd70 0x7f9bb4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:57.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bb9153640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f9bb415ac70 0x7f9bb406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52408/0 (socket says 172.21.15.31:52408) 2023-10-30T13:41:57.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bb9153640 1 -- 172.21.15.31:0/3751969435 learned_addr learned my addr 172.21.15.31:0/3751969435 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:57.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba155640 1 -- 172.21.15.31:0/3751969435 >> v1:172.21.15.33:6789/0 conn(0x7f9bb40c8eb0 legacy=0x7f9bb415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:57.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba155640 1 -- 172.21.15.31:0/3751969435 >> v1:172.21.15.31:6789/0 conn(0x7f9bb415ac70 legacy=0x7f9bb406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:57.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bb8952640 1 -- 172.21.15.31:0/3751969435 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587700541 0 0) 0x7f9ba4001120 con 0x7f9bb415ac70 2023-10-30T13:41:57.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.025+0000 7f9bba155640 1 -- 172.21.15.31:0/3751969435 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb415d1c0 con 0x7f9bb415cd70 2023-10-30T13:41:57.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba155640 1 --2- 172.21.15.31:0/3751969435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415cd70 0x7f9bb4069e90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009d90 tx=0x7f9bac01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19786f497dc32386 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:57.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bb8952640 1 -- 172.21.15.31:0/3751969435 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0af600 con 0x7f9bb415cd70 2023-10-30T13:41:57.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bb8952640 1 -- 172.21.15.31:0/3751969435 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac0ab050 con 0x7f9bb415cd70 2023-10-30T13:41:57.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bb8952640 1 -- 172.21.15.31:0/3751969435 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0af980 con 0x7f9bb415cd70 2023-10-30T13:41:57.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 -- 172.21.15.31:0/3751969435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415cd70 msgr2=0x7f9bb4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:57.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 --2- 172.21.15.31:0/3751969435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415cd70 0x7f9bb4069e90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009d90 tx=0x7f9bac01f130 comp rx=0 tx=0).stop 2023-10-30T13:41:57.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 -- 172.21.15.31:0/3751969435 shutdown_connections 2023-10-30T13:41:57.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 --2- 172.21.15.31:0/3751969435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415cd70 0x7f9bb4069e90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:57.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 -- 172.21.15.31:0/3751969435 >> 172.21.15.31:0/3751969435 conn(0x7f9bb40cef00 msgr2=0x7f9bb40cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:57.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 -- 172.21.15.31:0/3751969435 shutdown_connections 2023-10-30T13:41:57.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 -- 172.21.15.31:0/3751969435 wait complete. 2023-10-30T13:41:57.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 Processor -- start 2023-10-30T13:41:57.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.029+0000 7f9bba956640 1 -- start start 2023-10-30T13:41:57.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bb40c8eb0 0x7f9bb41f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:57.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415ac70 0x7f9bb41f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:57.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9bb415cd70 0x7f9bb41f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:57.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40c5480 con 0x7f9bb415ac70 2023-10-30T13:41:57.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40c6b60 con 0x7f9bb40c8eb0 2023-10-30T13:41:57.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40c4b30 con 0x7f9bb415cd70 2023-10-30T13:41:57.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415ac70 0x7f9bb41f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:57.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415ac70 0x7f9bb41f0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48756/0 (socket says 172.21.15.31:48756) 2023-10-30T13:41:57.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 -- 172.21.15.31:0/1607045449 learned_addr learned my addr 172.21.15.31:0/1607045449 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:57.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba155640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9bb415cd70 0x7f9bb41f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:57.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9954640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bb40c8eb0 0x7f9bb41f03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:57.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 -- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9bb415cd70 msgr2=0x7f9bb41f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:57.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9bb415cd70 0x7f9bb41f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:57.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 -- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bb40c8eb0 msgr2=0x7f9bb41f03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:57.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bb40c8eb0 0x7f9bb41f03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:57.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 -- 172.21.15.31:0/1607045449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac009a70 con 0x7f9bb415ac70 2023-10-30T13:41:57.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bb9153640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415ac70 0x7f9bb41f0910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c780 tx=0x7f9ba400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:57.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9baa7fc640 1 -- 172.21.15.31:0/1607045449 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba40072d0 con 0x7f9bb415ac70 2023-10-30T13:41:57.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9baa7fc640 1 -- 172.21.15.31:0/1607045449 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba4007450 con 0x7f9bb415ac70 2023-10-30T13:41:57.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb40c4e10 con 0x7f9bb415ac70 2023-10-30T13:41:57.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9baa7fc640 1 -- 172.21.15.31:0/1607045449 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba40047f0 con 0x7f9bb415ac70 2023-10-30T13:41:57.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb41e68e0 con 0x7f9bb415ac70 2023-10-30T13:41:57.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb41f00f0 con 0x7f9bb415ac70 2023-10-30T13:41:57.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.033+0000 7f9baa7fc640 1 -- 172.21.15.31:0/1607045449 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ba4004950 con 0x7f9bb415ac70 2023-10-30T13:41:57.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.037+0000 7f9baa7fc640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b980664a0 0x7f9b98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:57.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.037+0000 7f9baa7fc640 1 -- 172.21.15.31:0/1607045449 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f9ba4083400 con 0x7f9bb415ac70 2023-10-30T13:41:57.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.037+0000 7f9bb9954640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b980664a0 0x7f9b98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:57.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.037+0000 7f9bb9954640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b980664a0 0x7f9b98068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0000f70 tx=0x7f9bb0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:57.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.041+0000 7f9baa7fc640 1 -- 172.21.15.31:0/1607045449 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bb41f00f0 con 0x7f9bb415ac70 2023-10-30T13:41:57.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bb415dcc0 con 0x7f9bb415ac70 2023-10-30T13:41:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9baa7fc640 1 -- 172.21.15.31:0/1607045449 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f9bb415dcc0 con 0x7f9bb415ac70 2023-10-30T13:41:57.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:57.238 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:57.239 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:41:57.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b980664a0 msgr2=0x7f9b98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:57.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b980664a0 0x7f9b98068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0000f70 tx=0x7f9bb0009040 comp rx=0 tx=0).stop 2023-10-30T13:41:57.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415ac70 msgr2=0x7f9bb41f0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:57.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415ac70 0x7f9bb41f0910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c780 tx=0x7f9ba400cc40 comp rx=0 tx=0).stop 2023-10-30T13:41:57.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 shutdown_connections 2023-10-30T13:41:57.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b980664a0 0x7f9b98068960 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:57.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9bb415cd70 0x7f9bb41f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:57.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb415ac70 0x7f9bb41f0910 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:57.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 --2- 172.21.15.31:0/1607045449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bb40c8eb0 0x7f9bb41f03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:57.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 >> 172.21.15.31:0/1607045449 conn(0x7f9bb40cef00 msgr2=0x7f9bb40733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:57.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 shutdown_connections 2023-10-30T13:41:57.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:57.237+0000 7f9bba956640 1 -- 172.21.15.31:0/1607045449 wait complete. 2023-10-30T13:41:57.258 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:41:57.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 41 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:41:56.856155+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:41:59.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:59.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.413+0000 7fab7e980640 1 Processor -- start 2023-10-30T13:41:59.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.413+0000 7fab7e980640 1 -- start start 2023-10-30T13:41:59.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab7e980640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 0x7fab78065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab7e980640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7806e390 con 0x7fab78070c30 2023-10-30T13:41:59.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab7e980640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780660e0 con 0x7fab78159970 2023-10-30T13:41:59.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab7e980640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab78072390 con 0x7fab78153c10 2023-10-30T13:41:59.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fab78070c30 0x7fab78071070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52418/0 (socket says 172.21.15.31:52418) 2023-10-30T13:41:59.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 -- 172.21.15.31:0/3340819075 learned_addr learned my addr 172.21.15.31:0/3340819075 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:41:59.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab77fff640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 0x7fab78065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006198742 0 0) 0x7fab7806e390 con 0x7fab78070c30 2023-10-30T13:41:59.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab60003530 con 0x7fab78070c30 2023-10-30T13:41:59.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091530816 0 0) 0x7fab60003530 con 0x7fab78070c30 2023-10-30T13:41:59.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 msgr2=0x7fab78065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:59.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 0x7fab78065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 >> v1:172.21.15.33:6789/0 conn(0x7fab78159970 legacy=0x7fab78159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:59.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab780bec10 con 0x7fab78070c30 2023-10-30T13:41:59.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fab6c001680 con 0x7fab78070c30 2023-10-30T13:41:59.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 >> v1:172.21.15.31:6789/0 conn(0x7fab78070c30 legacy=0x7fab78071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:59.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600039b0 0x7fab60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab60004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 0x7fab600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab78072390 con 0x7fab600047d0 2023-10-30T13:41:59.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000bac0 con 0x7fab600039b0 2023-10-30T13:41:59.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000bc80 con 0x7fab78153c10 2023-10-30T13:41:59.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab60004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab77fff640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600039b0 0x7fab60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab7cf1d640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 0x7fab600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 -- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 msgr2=0x7fab600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:59.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab78153c10 0x7fab600051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 -- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600039b0 msgr2=0x7fab60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:59.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab600039b0 0x7fab60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.417+0000 7fab777fe640 1 -- 172.21.15.31:0/3340819075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60003530 con 0x7fab600047d0 2023-10-30T13:41:59.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab777fe640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab60004c50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c1c0 tx=0x7fab6c01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5410ca87e47f8024 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:59.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c005ae0 con 0x7fab600047d0 2023-10-30T13:41:59.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c005c60 con 0x7fab600047d0 2023-10-30T13:41:59.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab76ffd640 1 -- 172.21.15.31:0/3340819075 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c007070 con 0x7fab600047d0 2023-10-30T13:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 -- 172.21.15.31:0/3340819075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 msgr2=0x7fab60004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 --2- 172.21.15.31:0/3340819075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab60004c50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c1c0 tx=0x7fab6c01cc20 comp rx=0 tx=0).stop 2023-10-30T13:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7cf1d640 1 -- 172.21.15.31:0/3340819075 reap_dead start 2023-10-30T13:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 -- 172.21.15.31:0/3340819075 shutdown_connections 2023-10-30T13:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 -- 172.21.15.31:0/3340819075 >> 172.21.15.31:0/3340819075 conn(0x7fab780c7f60 msgr2=0x7fab780c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 -- 172.21.15.31:0/3340819075 shutdown_connections 2023-10-30T13:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 -- 172.21.15.31:0/3340819075 wait complete. 2023-10-30T13:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 Processor -- start 2023-10-30T13:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.421+0000 7fab7e980640 1 -- start start 2023-10-30T13:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab78156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab780bd990 0x7fab7814b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab7814bbd0 0x7fab781fa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7814c050 con 0x7fab600047d0 2023-10-30T13:41:59.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab78159300 con 0x7fab780bd990 2023-10-30T13:41:59.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab781fac80 con 0x7fab7814bbd0 2023-10-30T13:41:59.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab78156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab78156f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48780/0 (socket says 172.21.15.31:48780) 2023-10-30T13:41:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 -- 172.21.15.31:0/2682270072 learned_addr learned my addr 172.21.15.31:0/2682270072 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:41:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab777fe640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab780bd990 0x7fab7814b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7cf1d640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab7814bbd0 0x7fab781fa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 -- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab7814bbd0 msgr2=0x7fab781fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab7814bbd0 0x7fab781fa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 -- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab780bd990 msgr2=0x7fab7814b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab780bd990 0x7fab7814b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 -- 172.21.15.31:0/2682270072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7814ad60 con 0x7fab600047d0 2023-10-30T13:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab77fff640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab78156f80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fab680115e0 tx=0x7fab68011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab74ff9640 1 -- 172.21.15.31:0/2682270072 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800f8f0 con 0x7fab600047d0 2023-10-30T13:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab74ff9640 1 -- 172.21.15.31:0/2682270072 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6800fa50 con 0x7fab600047d0 2023-10-30T13:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab74ff9640 1 -- 172.21.15.31:0/2682270072 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800fdc0 con 0x7fab600047d0 2023-10-30T13:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7814b040 con 0x7fab600047d0 2023-10-30T13:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab780bd5a0 con 0x7fab600047d0 2023-10-30T13:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab780bd0f0 con 0x7fab600047d0 2023-10-30T13:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.425+0000 7fab74ff9640 1 -- 172.21.15.31:0/2682270072 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab68015460 con 0x7fab600047d0 2023-10-30T13:41:59.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.429+0000 7fab74ff9640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab640664d0 0x7fab64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:59.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.429+0000 7fab74ff9640 1 -- 172.21.15.31:0/2682270072 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fab680845f0 con 0x7fab600047d0 2023-10-30T13:41:59.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.429+0000 7fab777fe640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab640664d0 0x7fab64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:59.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.429+0000 7fab777fe640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab640664d0 0x7fab64068990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0ab000 tx=0x7fab6c0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:59.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.433+0000 7fab74ff9640 1 -- 172.21.15.31:0/2682270072 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab680517d0 con 0x7fab600047d0 2023-10-30T13:41:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab7804e170 con 0x7fab600047d0 2023-10-30T13:41:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab74ff9640 1 -- 172.21.15.31:0/2682270072 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fab68054df0 con 0x7fab600047d0 2023-10-30T13:41:59.622 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:41:59.622 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:59.623 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:41:59.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab640664d0 msgr2=0x7fab64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:59.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab640664d0 0x7fab64068990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0ab000 tx=0x7fab6c0a7070 comp rx=0 tx=0).stop 2023-10-30T13:41:59.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 msgr2=0x7fab78156f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:59.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab78156f80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fab680115e0 tx=0x7fab68011aa0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 shutdown_connections 2023-10-30T13:41:59.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab640664d0 0x7fab64068990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab7814bbd0 0x7fab781fa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab780bd990 0x7fab7814b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 --2- 172.21.15.31:0/2682270072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600047d0 0x7fab78156f80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:59.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 >> 172.21.15.31:0/2682270072 conn(0x7fab780c7f60 msgr2=0x7fab78071030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:59.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 shutdown_connections 2023-10-30T13:41:59.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:41:59.621+0000 7fab7e980640 1 -- 172.21.15.31:0/2682270072 wait complete. 2023-10-30T13:41:59.643 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:01.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4759e640 1 Processor -- start 2023-10-30T13:42:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4759e640 1 -- start start 2023-10-30T13:42:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4759e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fea48153b50 0x7fea481597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4759e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480c62b0 0x7fea480ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4759e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4815a770 con 0x7fea480cb180 2023-10-30T13:42:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4759e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea48159cf0 con 0x7fea48153b50 2023-10-30T13:42:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4759e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4806aa30 con 0x7fea480c62b0 2023-10-30T13:42:01.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea45d9b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480c62b0 0x7fea480ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:01.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4659c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fea48153b50 0x7fea481597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea46d9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fea480cb180 0x7fea4806ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:33106/0 (socket says 172.21.15.31:33106) 2023-10-30T13:42:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea46d9d640 1 -- 172.21.15.31:0/1378932035 learned_addr learned my addr 172.21.15.31:0/1378932035 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4559a640 1 -- 172.21.15.31:0/1378932035 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429905081 0 0) 0x7fea400014a0 con 0x7fea480cb180 2023-10-30T13:42:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4559a640 1 -- 172.21.15.31:0/1378932035 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea30003530 con 0x7fea480cb180 2023-10-30T13:42:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea45d9b640 1 -- 172.21.15.31:0/1378932035 >> v2:172.21.15.33:3300/0 conn(0x7fea48153b50 msgr2=0x7fea481597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea45d9b640 1 --2- 172.21.15.31:0/1378932035 >> v2:172.21.15.33:3300/0 conn(0x7fea48153b50 0x7fea481597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea45d9b640 1 -- 172.21.15.31:0/1378932035 >> v1:172.21.15.33:6789/0 conn(0x7fea480cb180 legacy=0x7fea4806ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea45d9b640 1 -- 172.21.15.31:0/1378932035 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea480c9d40 con 0x7fea480c62b0 2023-10-30T13:42:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.800+0000 7fea4659c640 1 --2- 172.21.15.31:0/1378932035 >> v2:172.21.15.33:3300/0 conn(0x7fea48153b50 0x7fea481597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:01.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea45d9b640 1 --2- 172.21.15.31:0/1378932035 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480c62b0 0x7fea480ca1d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0097c0 tx=0x7fea3c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe1a135ccb7606aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:01.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4559a640 1 -- 172.21.15.31:0/1378932035 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c0ab600 con 0x7fea480c62b0 2023-10-30T13:42:01.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4559a640 1 -- 172.21.15.31:0/1378932035 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3c0a7050 con 0x7fea480c62b0 2023-10-30T13:42:01.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4559a640 1 -- 172.21.15.31:0/1378932035 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c0ab980 con 0x7fea480c62b0 2023-10-30T13:42:01.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 -- 172.21.15.31:0/1378932035 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480c62b0 msgr2=0x7fea480ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:01.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 --2- 172.21.15.31:0/1378932035 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480c62b0 0x7fea480ca1d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0097c0 tx=0x7fea3c01c480 comp rx=0 tx=0).stop 2023-10-30T13:42:01.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 -- 172.21.15.31:0/1378932035 shutdown_connections 2023-10-30T13:42:01.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 --2- 172.21.15.31:0/1378932035 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480c62b0 0x7fea480ca1d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 --2- 172.21.15.31:0/1378932035 >> v2:172.21.15.33:3300/0 conn(0x7fea48153b50 0x7fea481597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 -- 172.21.15.31:0/1378932035 >> 172.21.15.31:0/1378932035 conn(0x7fea48065a80 msgr2=0x7fea48065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 -- 172.21.15.31:0/1378932035 shutdown_connections 2023-10-30T13:42:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.804+0000 7fea4759e640 1 -- 172.21.15.31:0/1378932035 wait complete. 2023-10-30T13:42:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 Processor -- start 2023-10-30T13:42:01.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 -- start start 2023-10-30T13:42:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea480c62b0 0x7fea481f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480cb180 0x7fea4814a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea48153b50 0x7fea48152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea481f15e0 con 0x7fea48153b50 2023-10-30T13:42:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea480c7a50 con 0x7fea480c62b0 2023-10-30T13:42:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4814a2c0 con 0x7fea480cb180 2023-10-30T13:42:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea48153b50 0x7fea48152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea48153b50 0x7fea48152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48786/0 (socket says 172.21.15.31:48786) 2023-10-30T13:42:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 -- 172.21.15.31:0/2174871216 learned_addr learned my addr 172.21.15.31:0/2174871216 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4659c640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea480c62b0 0x7fea481f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea45d9b640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480cb180 0x7fea4814a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 -- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480cb180 msgr2=0x7fea4814a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480cb180 0x7fea4814a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 -- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea480c62b0 msgr2=0x7fea481f1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea480c62b0 0x7fea481f1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 -- 172.21.15.31:0/2174871216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38009420 con 0x7fea48153b50 2023-10-30T13:42:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4659c640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea480c62b0 0x7fea481f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea46d9d640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea48153b50 0x7fea48152470 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fea4000cb70 tx=0x7fea4000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea36ffd640 1 -- 172.21.15.31:0/2174871216 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4000ace0 con 0x7fea48153b50 2023-10-30T13:42:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea36ffd640 1 -- 172.21.15.31:0/2174871216 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4000ae40 con 0x7fea48153b50 2023-10-30T13:42:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea36ffd640 1 -- 172.21.15.31:0/2174871216 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40012620 con 0x7fea48153b50 2023-10-30T13:42:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c009420 con 0x7fea48153b50 2023-10-30T13:42:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.808+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea481f1b60 con 0x7fea48153b50 2023-10-30T13:42:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.812+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea481f1cf0 con 0x7fea48153b50 2023-10-30T13:42:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.812+0000 7fea36ffd640 1 -- 172.21.15.31:0/2174871216 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea40007070 con 0x7fea48153b50 2023-10-30T13:42:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.812+0000 7fea36ffd640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea280664d0 0x7fea28068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.812+0000 7fea36ffd640 1 -- 172.21.15.31:0/2174871216 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fea40083d40 con 0x7fea48153b50 2023-10-30T13:42:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.812+0000 7fea4659c640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea280664d0 0x7fea28068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.812+0000 7fea4659c640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea280664d0 0x7fea28068990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea38009ca0 tx=0x7fea38008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:01.816+0000 7fea36ffd640 1 -- 172.21.15.31:0/2174871216 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea40054b80 con 0x7fea48153b50 2023-10-30T13:42:02.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea4815a770 con 0x7fea48153b50 2023-10-30T13:42:02.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea36ffd640 1 -- 172.21.15.31:0/2174871216 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fea4815a770 con 0x7fea48153b50 2023-10-30T13:42:02.017 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:02.017 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:02.017 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea280664d0 msgr2=0x7fea28068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea280664d0 0x7fea28068990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea38009ca0 tx=0x7fea38008040 comp rx=0 tx=0).stop 2023-10-30T13:42:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea48153b50 msgr2=0x7fea48152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea48153b50 0x7fea48152470 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fea4000cb70 tx=0x7fea4000a430 comp rx=0 tx=0).stop 2023-10-30T13:42:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 shutdown_connections 2023-10-30T13:42:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fea280664d0 0x7fea28068990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea48153b50 0x7fea48152470 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea480cb180 0x7fea4814a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 --2- 172.21.15.31:0/2174871216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea480c62b0 0x7fea481f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 >> 172.21.15.31:0/2174871216 conn(0x7fea48065a80 msgr2=0x7fea48069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 shutdown_connections 2023-10-30T13:42:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:02.016+0000 7fea4759e640 1 -- 172.21.15.31:0/2174871216 wait complete. 2023-10-30T13:42:02.039 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:03.889 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/13' 2023-10-30T13:42:03.889 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/33' 2023-10-30T13:42:03.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/17' 2023-10-30T13:42:03.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/35' 2023-10-30T13:42:03.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './186/37' 2023-10-30T13:42:03.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './186' 2023-10-30T13:42:03.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/19' 2023-10-30T13:42:03.891 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/15' 2023-10-30T13:42:03.891 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/31' 2023-10-30T13:42:03.891 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/36' 2023-10-30T13:42:03.891 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/20' 2023-10-30T13:42:03.891 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/12' 2023-10-30T13:42:03.892 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/39' 2023-10-30T13:42:03.892 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/10' 2023-10-30T13:42:03.892 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/34' 2023-10-30T13:42:03.892 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/38' 2023-10-30T13:42:03.892 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/11' 2023-10-30T13:42:03.892 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/16' 2023-10-30T13:42:03.893 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/32' 2023-10-30T13:42:03.893 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/30' 2023-10-30T13:42:03.893 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/14' 2023-10-30T13:42:03.893 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/18' 2023-10-30T13:42:03.893 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/13' 2023-10-30T13:42:03.894 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/33' 2023-10-30T13:42:03.894 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/17' 2023-10-30T13:42:03.894 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/35' 2023-10-30T13:42:03.894 INFO:tasks.workunit.client.0.smithi031.stdout:removed './187/37' 2023-10-30T13:42:03.894 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './187' 2023-10-30T13:42:03.895 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/19' 2023-10-30T13:42:03.895 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/15' 2023-10-30T13:42:03.895 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/31' 2023-10-30T13:42:03.895 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/36' 2023-10-30T13:42:03.895 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/20' 2023-10-30T13:42:03.896 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/12' 2023-10-30T13:42:03.896 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/39' 2023-10-30T13:42:03.896 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/10' 2023-10-30T13:42:03.896 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/34' 2023-10-30T13:42:03.896 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/38' 2023-10-30T13:42:03.896 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/11' 2023-10-30T13:42:03.897 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/16' 2023-10-30T13:42:03.897 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/32' 2023-10-30T13:42:03.897 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/30' 2023-10-30T13:42:03.897 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/14' 2023-10-30T13:42:03.897 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/18' 2023-10-30T13:42:03.897 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/13' 2023-10-30T13:42:03.898 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/33' 2023-10-30T13:42:03.898 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/17' 2023-10-30T13:42:03.898 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/35' 2023-10-30T13:42:03.898 INFO:tasks.workunit.client.0.smithi031.stdout:removed './188/37' 2023-10-30T13:42:03.898 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './188' 2023-10-30T13:42:03.899 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/19' 2023-10-30T13:42:03.899 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/15' 2023-10-30T13:42:03.899 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/31' 2023-10-30T13:42:03.899 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/36' 2023-10-30T13:42:03.899 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/20' 2023-10-30T13:42:03.899 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/12' 2023-10-30T13:42:03.900 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/39' 2023-10-30T13:42:03.900 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/10' 2023-10-30T13:42:03.900 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/34' 2023-10-30T13:42:03.900 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/38' 2023-10-30T13:42:03.900 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/11' 2023-10-30T13:42:03.900 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/16' 2023-10-30T13:42:03.901 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/32' 2023-10-30T13:42:03.901 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/30' 2023-10-30T13:42:03.901 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/14' 2023-10-30T13:42:03.901 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/18' 2023-10-30T13:42:03.901 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/13' 2023-10-30T13:42:03.901 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/33' 2023-10-30T13:42:03.902 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/17' 2023-10-30T13:42:03.902 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/35' 2023-10-30T13:42:03.902 INFO:tasks.workunit.client.0.smithi031.stdout:removed './189/37' 2023-10-30T13:42:03.902 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './189' 2023-10-30T13:42:03.902 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/19' 2023-10-30T13:42:03.903 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/15' 2023-10-30T13:42:03.903 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/31' 2023-10-30T13:42:03.903 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/36' 2023-10-30T13:42:03.903 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/20' 2023-10-30T13:42:03.903 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/12' 2023-10-30T13:42:03.903 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/39' 2023-10-30T13:42:03.904 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/10' 2023-10-30T13:42:03.904 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/34' 2023-10-30T13:42:03.904 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/38' 2023-10-30T13:42:03.904 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/11' 2023-10-30T13:42:03.904 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/16' 2023-10-30T13:42:03.904 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/32' 2023-10-30T13:42:03.905 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/30' 2023-10-30T13:42:03.905 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/14' 2023-10-30T13:42:03.905 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/18' 2023-10-30T13:42:03.905 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/13' 2023-10-30T13:42:03.905 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/33' 2023-10-30T13:42:03.905 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/17' 2023-10-30T13:42:03.906 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/35' 2023-10-30T13:42:03.906 INFO:tasks.workunit.client.0.smithi031.stdout:removed './190/37' 2023-10-30T13:42:03.906 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './190' 2023-10-30T13:42:03.906 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/19' 2023-10-30T13:42:03.906 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/15' 2023-10-30T13:42:03.906 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/31' 2023-10-30T13:42:03.907 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/36' 2023-10-30T13:42:03.907 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/20' 2023-10-30T13:42:03.907 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/12' 2023-10-30T13:42:03.907 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/39' 2023-10-30T13:42:03.907 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/10' 2023-10-30T13:42:03.907 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/34' 2023-10-30T13:42:03.908 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/38' 2023-10-30T13:42:03.908 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/11' 2023-10-30T13:42:03.908 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/16' 2023-10-30T13:42:03.908 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/32' 2023-10-30T13:42:03.908 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/30' 2023-10-30T13:42:03.908 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/14' 2023-10-30T13:42:03.909 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/18' 2023-10-30T13:42:03.909 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/13' 2023-10-30T13:42:03.909 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/33' 2023-10-30T13:42:03.909 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/17' 2023-10-30T13:42:03.909 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/35' 2023-10-30T13:42:03.910 INFO:tasks.workunit.client.0.smithi031.stdout:removed './191/37' 2023-10-30T13:42:03.910 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './191' 2023-10-30T13:42:03.910 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/19' 2023-10-30T13:42:03.910 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/15' 2023-10-30T13:42:03.910 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/31' 2023-10-30T13:42:03.910 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/36' 2023-10-30T13:42:03.911 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/20' 2023-10-30T13:42:03.911 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/12' 2023-10-30T13:42:03.911 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/39' 2023-10-30T13:42:03.911 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/10' 2023-10-30T13:42:03.911 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/34' 2023-10-30T13:42:03.911 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/38' 2023-10-30T13:42:03.912 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/11' 2023-10-30T13:42:03.912 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/16' 2023-10-30T13:42:03.912 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/32' 2023-10-30T13:42:03.912 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/30' 2023-10-30T13:42:03.912 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/14' 2023-10-30T13:42:03.913 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/18' 2023-10-30T13:42:03.913 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/13' 2023-10-30T13:42:03.913 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/33' 2023-10-30T13:42:03.913 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/17' 2023-10-30T13:42:03.913 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/35' 2023-10-30T13:42:03.913 INFO:tasks.workunit.client.0.smithi031.stdout:removed './192/37' 2023-10-30T13:42:03.914 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './192' 2023-10-30T13:42:03.914 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/19' 2023-10-30T13:42:03.914 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/15' 2023-10-30T13:42:03.914 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/31' 2023-10-30T13:42:03.914 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/36' 2023-10-30T13:42:03.914 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/20' 2023-10-30T13:42:03.915 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/12' 2023-10-30T13:42:03.915 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/39' 2023-10-30T13:42:03.915 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/10' 2023-10-30T13:42:03.915 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/34' 2023-10-30T13:42:03.915 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/38' 2023-10-30T13:42:03.915 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/11' 2023-10-30T13:42:03.916 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/16' 2023-10-30T13:42:03.916 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/32' 2023-10-30T13:42:03.916 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/30' 2023-10-30T13:42:03.916 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/14' 2023-10-30T13:42:03.916 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/18' 2023-10-30T13:42:03.917 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/13' 2023-10-30T13:42:03.917 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/33' 2023-10-30T13:42:03.917 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/17' 2023-10-30T13:42:03.917 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/35' 2023-10-30T13:42:03.917 INFO:tasks.workunit.client.0.smithi031.stdout:removed './193/37' 2023-10-30T13:42:03.917 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './193' 2023-10-30T13:42:03.918 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/19' 2023-10-30T13:42:03.918 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/15' 2023-10-30T13:42:03.918 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/31' 2023-10-30T13:42:03.918 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/36' 2023-10-30T13:42:03.918 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/20' 2023-10-30T13:42:03.919 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/12' 2023-10-30T13:42:03.919 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/39' 2023-10-30T13:42:03.919 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/10' 2023-10-30T13:42:03.919 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/34' 2023-10-30T13:42:03.919 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/38' 2023-10-30T13:42:03.919 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/11' 2023-10-30T13:42:03.920 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/16' 2023-10-30T13:42:03.920 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/32' 2023-10-30T13:42:03.920 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/30' 2023-10-30T13:42:03.920 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/14' 2023-10-30T13:42:03.920 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/18' 2023-10-30T13:42:03.920 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/13' 2023-10-30T13:42:03.921 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/33' 2023-10-30T13:42:03.921 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/17' 2023-10-30T13:42:03.921 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/35' 2023-10-30T13:42:03.921 INFO:tasks.workunit.client.0.smithi031.stdout:removed './194/37' 2023-10-30T13:42:03.921 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './194' 2023-10-30T13:42:03.922 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/19' 2023-10-30T13:42:03.922 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/15' 2023-10-30T13:42:03.922 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/31' 2023-10-30T13:42:03.922 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/36' 2023-10-30T13:42:03.922 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/20' 2023-10-30T13:42:03.922 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/12' 2023-10-30T13:42:03.923 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/39' 2023-10-30T13:42:03.923 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/10' 2023-10-30T13:42:03.923 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/34' 2023-10-30T13:42:03.923 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/38' 2023-10-30T13:42:03.923 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/11' 2023-10-30T13:42:03.923 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/16' 2023-10-30T13:42:03.924 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/32' 2023-10-30T13:42:03.924 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/30' 2023-10-30T13:42:03.924 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/14' 2023-10-30T13:42:03.924 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/18' 2023-10-30T13:42:03.924 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/13' 2023-10-30T13:42:03.925 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/33' 2023-10-30T13:42:03.925 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/17' 2023-10-30T13:42:03.925 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/35' 2023-10-30T13:42:03.925 INFO:tasks.workunit.client.0.smithi031.stdout:removed './195/37' 2023-10-30T13:42:03.925 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './195' 2023-10-30T13:42:03.925 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/19' 2023-10-30T13:42:03.926 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/15' 2023-10-30T13:42:03.926 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/31' 2023-10-30T13:42:03.926 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/36' 2023-10-30T13:42:03.926 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/20' 2023-10-30T13:42:03.926 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/12' 2023-10-30T13:42:03.927 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/39' 2023-10-30T13:42:03.927 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/10' 2023-10-30T13:42:04.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.188+0000 7f154473e640 1 Processor -- start 2023-10-30T13:42:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.188+0000 7f154473e640 1 -- start start 2023-10-30T13:42:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f154473e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f154473e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f153c06c480 0x7f153c159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f154473e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0c9fd0 con 0x7f153c153b10 2023-10-30T13:42:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f154473e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c159900 con 0x7f153c06c480 2023-10-30T13:42:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f154473e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c15d070 con 0x7f153c0c9b90 2023-10-30T13:42:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f1542cdb640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f153c06c480 0x7f153c159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f1541cd9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f15424da640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f153c153b10 0x7f153c0ca8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:33120/0 (socket says 172.21.15.31:33120) 2023-10-30T13:42:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f15424da640 1 -- 172.21.15.31:0/876427490 learned_addr learned my addr 172.21.15.31:0/876427490 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f1542cdb640 1 -- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 msgr2=0x7f153c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f1542cdb640 1 --2- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f1542cdb640 1 -- 172.21.15.31:0/876427490 >> v1:172.21.15.33:6789/0 conn(0x7f153c153b10 legacy=0x7f153c0ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f1542cdb640 1 -- 172.21.15.31:0/876427490 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c15d350 con 0x7f153c06c480 2023-10-30T13:42:04.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.192+0000 7f1541cd9640 1 --2- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f1542cdb640 1 --2- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c06c480 0x7f153c159d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1534009d90 tx=0x7f153401c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27c4c90231beccab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f15414d8640 1 -- 172.21.15.31:0/876427490 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15340a5890 con 0x7f153c06c480 2023-10-30T13:42:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f15414d8640 1 -- 172.21.15.31:0/876427490 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15340a59f0 con 0x7f153c06c480 2023-10-30T13:42:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f15414d8640 1 -- 172.21.15.31:0/876427490 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15340a5d80 con 0x7f153c06c480 2023-10-30T13:42:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 -- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c06c480 msgr2=0x7f153c159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 --2- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c06c480 0x7f153c159d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1534009d90 tx=0x7f153401c8c0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 -- 172.21.15.31:0/876427490 shutdown_connections 2023-10-30T13:42:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 --2- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 --2- 172.21.15.31:0/876427490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c06c480 0x7f153c159d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 -- 172.21.15.31:0/876427490 >> 172.21.15.31:0/876427490 conn(0x7f153c06a240 msgr2=0x7f153c06a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 -- 172.21.15.31:0/876427490 shutdown_connections 2023-10-30T13:42:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.196+0000 7f154473e640 1 -- 172.21.15.31:0/876427490 wait complete. 2023-10-30T13:42:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 Processor -- start 2023-10-30T13:42:04.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 -- start start 2023-10-30T13:42:04.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c06c480 0x7f153c140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c153b10 0x7f153c147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0b33c0 con 0x7f153c06c480 2023-10-30T13:42:04.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c146420 con 0x7f153c153b10 2023-10-30T13:42:04.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0b2740 con 0x7f153c0c9b90 2023-10-30T13:42:04.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c06c480 0x7f153c140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c06c480 0x7f153c140de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48808/0 (socket says 172.21.15.31:48808) 2023-10-30T13:42:04.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 -- 172.21.15.31:0/3115620003 learned_addr learned my addr 172.21.15.31:0/3115620003 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:04.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f1542cdb640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c153b10 0x7f153c147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f1541cd9640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 -- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 msgr2=0x7f153c147980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c147980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 -- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c153b10 msgr2=0x7f153c147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c153b10 0x7f153c147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 -- 172.21.15.31:0/3115620003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c009420 con 0x7f153c06c480 2023-10-30T13:42:04.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f1542cdb640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c153b10 0x7f153c147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:04.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f1541cd9640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c147980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:04.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f15424da640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c06c480 0x7f153c140de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f153800c400 tx=0x7f153800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:04.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f1532ffd640 1 -- 172.21.15.31:0/3115620003 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538005cf0 con 0x7f153c06c480 2023-10-30T13:42:04.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f1532ffd640 1 -- 172.21.15.31:0/3115620003 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1538005e50 con 0x7f153c06c480 2023-10-30T13:42:04.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1534009a70 con 0x7f153c06c480 2023-10-30T13:42:04.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f1532ffd640 1 -- 172.21.15.31:0/3115620003 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15380072d0 con 0x7f153c06c480 2023-10-30T13:42:04.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c0b2d00 con 0x7f153c06c480 2023-10-30T13:42:04.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.200+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c067540 con 0x7f153c06c480 2023-10-30T13:42:04.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.204+0000 7f1532ffd640 1 -- 172.21.15.31:0/3115620003 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1538007430 con 0x7f153c06c480 2023-10-30T13:42:04.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.204+0000 7f1532ffd640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15200664d0 0x7f1520068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.204+0000 7f1532ffd640 1 -- 172.21.15.31:0/3115620003 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f15380833e0 con 0x7f153c06c480 2023-10-30T13:42:04.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.204+0000 7f1541cd9640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15200664d0 0x7f1520068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.204+0000 7f1541cd9640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15200664d0 0x7f1520068990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f153c1efa10 tx=0x7f152c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:04.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.208+0000 7f1532ffd640 1 -- 172.21.15.31:0/3115620003 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1538050670 con 0x7f153c06c480 2023-10-30T13:42:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.408+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f153c0cbf40 con 0x7f153c06c480 2023-10-30T13:42:04.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.408+0000 7f1532ffd640 1 -- 172.21.15.31:0/3115620003 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f1538053c90 con 0x7f153c06c480 2023-10-30T13:42:04.412 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:04.412 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:04.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15200664d0 msgr2=0x7f1520068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15200664d0 0x7f1520068990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f153c1efa10 tx=0x7f152c008040 comp rx=0 tx=0).stop 2023-10-30T13:42:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c06c480 msgr2=0x7f153c140de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c06c480 0x7f153c140de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f153800c400 tx=0x7f153800c8c0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 shutdown_connections 2023-10-30T13:42:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15200664d0 0x7f1520068990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f153c153b10 0x7f153c147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f153c0c9b90 0x7f153c147980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 --2- 172.21.15.31:0/3115620003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c06c480 0x7f153c140de0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 >> 172.21.15.31:0/3115620003 conn(0x7f153c06a240 msgr2=0x7f153c0c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 shutdown_connections 2023-10-30T13:42:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:04.412+0000 7f154473e640 1 -- 172.21.15.31:0/3115620003 wait complete. 2023-10-30T13:42:04.439 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:06.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:06.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.592+0000 7f85ded80640 1 Processor -- start 2023-10-30T13:42:06.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.592+0000 7f85ded80640 1 -- start start 2023-10-30T13:42:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85ded80640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f85d8153c10 0x7f85d8065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85ded80640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d806e390 con 0x7f85d8070c30 2023-10-30T13:42:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85ded80640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d80660e0 con 0x7f85d8159970 2023-10-30T13:42:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85ded80640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d8072390 con 0x7f85d8153c10 2023-10-30T13:42:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d7fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f85d8070c30 0x7f85d8071070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55052/0 (socket says 172.21.15.31:55052) 2023-10-30T13:42:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d7fff640 1 -- 172.21.15.31:0/877488179 learned_addr learned my addr 172.21.15.31:0/877488179 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/877488179 >> v2:172.21.15.33:3300/0 conn(0x7f85d8153c10 0x7f85d8065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309176641 0 0) 0x7f85d806e390 con 0x7f85d8070c30 2023-10-30T13:42:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c8003530 con 0x7f85d8070c30 2023-10-30T13:42:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279285383 0 0) 0x7f85c8003530 con 0x7f85d8070c30 2023-10-30T13:42:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 >> v2:172.21.15.33:3300/0 conn(0x7f85d8153c10 msgr2=0x7f85d8065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 --2- 172.21.15.31:0/877488179 >> v2:172.21.15.33:3300/0 conn(0x7f85d8153c10 0x7f85d8065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 >> v1:172.21.15.33:6789/0 conn(0x7f85d8159970 legacy=0x7f85d8159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80bec10 con 0x7f85d8070c30 2023-10-30T13:42:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f85c4001680 con 0x7f85d8070c30 2023-10-30T13:42:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 >> v1:172.21.15.31:6789/0 conn(0x7f85d8070c30 legacy=0x7f85d8071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85c8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85c80047d0 0x7f85c8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85c8005190 0x7f85c800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d8072390 con 0x7f85c8003980 2023-10-30T13:42:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c800c130 con 0x7f85c80047d0 2023-10-30T13:42:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c800c2f0 con 0x7f85c8005190 2023-10-30T13:42:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85c8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d7fff640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85c80047d0 0x7f85c8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dd31d640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85c8005190 0x7f85c800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dcb1c640 1 -- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85c8005190 msgr2=0x7f85c800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:06.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85c8005190 0x7f85c800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dcb1c640 1 -- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85c80047d0 msgr2=0x7f85c8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:06.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85c80047d0 0x7f85c8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dcb1c640 1 -- 172.21.15.31:0/877488179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8003530 con 0x7f85c8003980 2023-10-30T13:42:06.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dd31d640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85c8005190 0x7f85c800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:42:06.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85d7fff640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85c80047d0 0x7f85c8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:06.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.596+0000 7f85dd31d640 1 -- 172.21.15.31:0/877488179 reap_dead start 2023-10-30T13:42:06.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85c8003da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009ae0 tx=0x7f85cc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24adce918b3853 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:06.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc0ab450 con 0x7f85c8003980 2023-10-30T13:42:06.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc0ab5b0 con 0x7f85c8003980 2023-10-30T13:42:06.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85d77fe640 1 -- 172.21.15.31:0/877488179 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc0ab960 con 0x7f85c8003980 2023-10-30T13:42:06.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 -- 172.21.15.31:0/877488179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 msgr2=0x7f85c8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85c8003da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009ae0 tx=0x7f85cc01ca50 comp rx=0 tx=0).stop 2023-10-30T13:42:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 -- 172.21.15.31:0/877488179 shutdown_connections 2023-10-30T13:42:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 --2- 172.21.15.31:0/877488179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85c8003da0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 -- 172.21.15.31:0/877488179 >> 172.21.15.31:0/877488179 conn(0x7f85d80c7f60 msgr2=0x7f85d80c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 -- 172.21.15.31:0/877488179 shutdown_connections 2023-10-30T13:42:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 -- 172.21.15.31:0/877488179 wait complete. 2023-10-30T13:42:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 Processor -- start 2023-10-30T13:42:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.600+0000 7f85ded80640 1 -- start start 2023-10-30T13:42:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85d81e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85d80be3a0 0x7f85d81f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d814b9c0 0x7f85d81f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d814f060 con 0x7f85c8003980 2023-10-30T13:42:06.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80c9c20 con 0x7f85d814b9c0 2023-10-30T13:42:06.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d814d6d0 con 0x7f85d80be3a0 2023-10-30T13:42:06.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85d81e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85d81e8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35648/0 (socket says 172.21.15.31:35648) 2023-10-30T13:42:06.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 -- 172.21.15.31:0/1464575380 learned_addr learned my addr 172.21.15.31:0/1464575380 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:06.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dd31d640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d814b9c0 0x7f85d81f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85d7fff640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85d80be3a0 0x7f85d81f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 -- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85d80be3a0 msgr2=0x7f85d81f5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:06.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85d80be3a0 0x7f85d81f5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 -- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d814b9c0 msgr2=0x7f85d81f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:06.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d814b9c0 0x7f85d81f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 -- 172.21.15.31:0/1464575380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d814c0a0 con 0x7f85c8003980 2023-10-30T13:42:06.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85dcb1c640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85d81e8420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f85cc01c880 tx=0x7f85c8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:06.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85d57fa640 1 -- 172.21.15.31:0/1464575380 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc005ce0 con 0x7f85c8003980 2023-10-30T13:42:06.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85d57fa640 1 -- 172.21.15.31:0/1464575380 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc005e40 con 0x7f85c8003980 2023-10-30T13:42:06.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d814edb0 con 0x7f85c8003980 2023-10-30T13:42:06.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85d57fa640 1 -- 172.21.15.31:0/1464575380 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc0ab920 con 0x7f85c8003980 2023-10-30T13:42:06.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d80c98b0 con 0x7f85c8003980 2023-10-30T13:42:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d804e170 con 0x7f85c8003980 2023-10-30T13:42:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.604+0000 7f85d57fa640 1 -- 172.21.15.31:0/1464575380 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85cc0aba80 con 0x7f85c8003980 2023-10-30T13:42:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.608+0000 7f85d57fa640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85bc0664f0 0x7f85bc0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.608+0000 7f85d57fa640 1 -- 172.21.15.31:0/1464575380 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f85cc11e060 con 0x7f85c8003980 2023-10-30T13:42:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.608+0000 7f85d7fff640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85bc0664f0 0x7f85bc0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.608+0000 7f85d7fff640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85bc0664f0 0x7f85bc0689b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f85c4001ec0 tx=0x7f85c4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.612+0000 7f85d57fa640 1 -- 172.21.15.31:0/1464575380 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85cc0eb600 con 0x7f85c8003980 2023-10-30T13:42:06.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85d806e300 con 0x7f85c8003980 2023-10-30T13:42:06.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85d57fa640 1 -- 172.21.15.31:0/1464575380 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f85cc0c1020 con 0x7f85c8003980 2023-10-30T13:42:06.818 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:06.818 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:06.818 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:06.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85bc0664f0 msgr2=0x7f85bc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:06.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85bc0664f0 0x7f85bc0689b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f85c4001ec0 tx=0x7f85c4009040 comp rx=0 tx=0).stop 2023-10-30T13:42:06.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 msgr2=0x7f85d81e8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:06.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85d81e8420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f85cc01c880 tx=0x7f85c8005800 comp rx=0 tx=0).stop 2023-10-30T13:42:06.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 shutdown_connections 2023-10-30T13:42:06.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f85bc0664f0 0x7f85bc0689b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d814b9c0 0x7f85d81f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f85d80be3a0 0x7f85d81f5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 --2- 172.21.15.31:0/1464575380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85c8003980 0x7f85d81e8420 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 >> 172.21.15.31:0/1464575380 conn(0x7f85d80c7f60 msgr2=0x7f85d80bd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 shutdown_connections 2023-10-30T13:42:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:06.816+0000 7f85ded80640 1 -- 172.21.15.31:0/1464575380 wait complete. 2023-10-30T13:42:06.839 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:07.639 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/34' 2023-10-30T13:42:07.639 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/38' 2023-10-30T13:42:07.639 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/11' 2023-10-30T13:42:07.639 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/16' 2023-10-30T13:42:07.640 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/32' 2023-10-30T13:42:07.640 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/30' 2023-10-30T13:42:07.640 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/14' 2023-10-30T13:42:07.640 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/18' 2023-10-30T13:42:07.640 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/13' 2023-10-30T13:42:07.640 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/33' 2023-10-30T13:42:07.641 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/17' 2023-10-30T13:42:07.641 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/35' 2023-10-30T13:42:07.641 INFO:tasks.workunit.client.0.smithi031.stdout:removed './196/37' 2023-10-30T13:42:07.641 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './196' 2023-10-30T13:42:07.641 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/19' 2023-10-30T13:42:07.641 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/15' 2023-10-30T13:42:07.642 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/31' 2023-10-30T13:42:07.642 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/36' 2023-10-30T13:42:07.642 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/20' 2023-10-30T13:42:07.642 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/12' 2023-10-30T13:42:07.642 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/39' 2023-10-30T13:42:07.642 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/10' 2023-10-30T13:42:07.643 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/34' 2023-10-30T13:42:07.643 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/38' 2023-10-30T13:42:07.643 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/11' 2023-10-30T13:42:07.643 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/16' 2023-10-30T13:42:07.643 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/32' 2023-10-30T13:42:07.643 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/30' 2023-10-30T13:42:07.644 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/14' 2023-10-30T13:42:07.644 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/18' 2023-10-30T13:42:07.644 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/13' 2023-10-30T13:42:07.644 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/33' 2023-10-30T13:42:07.644 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/17' 2023-10-30T13:42:07.644 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/35' 2023-10-30T13:42:07.645 INFO:tasks.workunit.client.0.smithi031.stdout:removed './197/37' 2023-10-30T13:42:07.645 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './197' 2023-10-30T13:42:07.645 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/19' 2023-10-30T13:42:07.645 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/15' 2023-10-30T13:42:07.645 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/31' 2023-10-30T13:42:07.645 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/36' 2023-10-30T13:42:07.646 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/20' 2023-10-30T13:42:07.646 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/12' 2023-10-30T13:42:07.646 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/39' 2023-10-30T13:42:07.646 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/10' 2023-10-30T13:42:07.646 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/34' 2023-10-30T13:42:07.647 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/38' 2023-10-30T13:42:07.647 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/11' 2023-10-30T13:42:07.647 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/16' 2023-10-30T13:42:07.647 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/32' 2023-10-30T13:42:07.647 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/30' 2023-10-30T13:42:07.647 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/14' 2023-10-30T13:42:07.648 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/18' 2023-10-30T13:42:07.648 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/13' 2023-10-30T13:42:07.648 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/33' 2023-10-30T13:42:07.648 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/17' 2023-10-30T13:42:07.648 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/35' 2023-10-30T13:42:07.648 INFO:tasks.workunit.client.0.smithi031.stdout:removed './198/37' 2023-10-30T13:42:07.649 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './198' 2023-10-30T13:42:07.649 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/19' 2023-10-30T13:42:07.649 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/15' 2023-10-30T13:42:07.649 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/31' 2023-10-30T13:42:07.649 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/36' 2023-10-30T13:42:07.649 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/20' 2023-10-30T13:42:07.649 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/12' 2023-10-30T13:42:07.650 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/39' 2023-10-30T13:42:07.650 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/10' 2023-10-30T13:42:07.650 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/34' 2023-10-30T13:42:07.650 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/38' 2023-10-30T13:42:07.650 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/11' 2023-10-30T13:42:07.651 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/16' 2023-10-30T13:42:07.651 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/32' 2023-10-30T13:42:07.651 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/30' 2023-10-30T13:42:07.651 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/14' 2023-10-30T13:42:07.651 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/18' 2023-10-30T13:42:07.651 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/13' 2023-10-30T13:42:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/33' 2023-10-30T13:42:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/17' 2023-10-30T13:42:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/35' 2023-10-30T13:42:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:removed './199/37' 2023-10-30T13:42:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './199' 2023-10-30T13:42:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/21' 2023-10-30T13:42:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/27' 2023-10-30T13:42:07.653 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/31' 2023-10-30T13:42:07.653 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/36' 2023-10-30T13:42:07.653 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/20' 2023-10-30T13:42:07.653 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/39' 2023-10-30T13:42:07.653 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/34' 2023-10-30T13:42:07.653 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/38' 2023-10-30T13:42:07.654 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/28' 2023-10-30T13:42:07.654 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/24' 2023-10-30T13:42:07.654 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/29' 2023-10-30T13:42:07.654 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/32' 2023-10-30T13:42:07.654 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/22' 2023-10-30T13:42:07.654 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/25' 2023-10-30T13:42:07.655 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/30' 2023-10-30T13:42:07.655 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/33' 2023-10-30T13:42:07.655 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/23' 2023-10-30T13:42:07.655 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/35' 2023-10-30T13:42:07.655 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/26' 2023-10-30T13:42:07.655 INFO:tasks.workunit.client.0.smithi031.stdout:removed './200/37' 2023-10-30T13:42:07.656 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './200' 2023-10-30T13:42:07.656 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/21' 2023-10-30T13:42:07.656 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/27' 2023-10-30T13:42:07.656 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/31' 2023-10-30T13:42:07.656 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/36' 2023-10-30T13:42:07.656 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/20' 2023-10-30T13:42:07.657 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/39' 2023-10-30T13:42:07.657 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/34' 2023-10-30T13:42:07.657 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/38' 2023-10-30T13:42:07.657 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/28' 2023-10-30T13:42:07.657 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/24' 2023-10-30T13:42:07.658 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/29' 2023-10-30T13:42:07.658 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/32' 2023-10-30T13:42:07.658 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/22' 2023-10-30T13:42:07.658 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/25' 2023-10-30T13:42:07.658 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/30' 2023-10-30T13:42:07.658 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/33' 2023-10-30T13:42:07.659 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/23' 2023-10-30T13:42:07.659 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/35' 2023-10-30T13:42:07.659 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/26' 2023-10-30T13:42:07.659 INFO:tasks.workunit.client.0.smithi031.stdout:removed './201/37' 2023-10-30T13:42:07.659 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './201' 2023-10-30T13:42:07.659 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/21' 2023-10-30T13:42:07.660 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/27' 2023-10-30T13:42:07.660 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/31' 2023-10-30T13:42:07.660 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/36' 2023-10-30T13:42:07.660 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/20' 2023-10-30T13:42:07.660 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/39' 2023-10-30T13:42:07.660 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/34' 2023-10-30T13:42:07.661 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/38' 2023-10-30T13:42:07.661 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/28' 2023-10-30T13:42:07.661 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/24' 2023-10-30T13:42:07.661 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/29' 2023-10-30T13:42:07.661 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/32' 2023-10-30T13:42:07.661 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/22' 2023-10-30T13:42:07.662 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/25' 2023-10-30T13:42:07.662 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/30' 2023-10-30T13:42:07.662 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/33' 2023-10-30T13:42:07.662 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/23' 2023-10-30T13:42:07.662 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/35' 2023-10-30T13:42:07.663 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/26' 2023-10-30T13:42:07.663 INFO:tasks.workunit.client.0.smithi031.stdout:removed './202/37' 2023-10-30T13:42:07.663 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './202' 2023-10-30T13:42:07.663 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/21' 2023-10-30T13:42:07.663 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/27' 2023-10-30T13:42:07.663 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/31' 2023-10-30T13:42:07.664 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/36' 2023-10-30T13:42:07.664 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/20' 2023-10-30T13:42:07.664 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/39' 2023-10-30T13:42:07.664 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/34' 2023-10-30T13:42:07.664 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/38' 2023-10-30T13:42:07.664 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/28' 2023-10-30T13:42:07.665 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/24' 2023-10-30T13:42:07.665 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/29' 2023-10-30T13:42:07.665 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/32' 2023-10-30T13:42:07.665 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/22' 2023-10-30T13:42:07.665 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/25' 2023-10-30T13:42:07.665 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/30' 2023-10-30T13:42:07.666 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/33' 2023-10-30T13:42:07.666 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/23' 2023-10-30T13:42:07.666 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/35' 2023-10-30T13:42:07.666 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/26' 2023-10-30T13:42:07.666 INFO:tasks.workunit.client.0.smithi031.stdout:removed './203/37' 2023-10-30T13:42:07.666 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './203' 2023-10-30T13:42:07.667 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/21' 2023-10-30T13:42:07.667 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/27' 2023-10-30T13:42:07.667 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/31' 2023-10-30T13:42:07.667 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/36' 2023-10-30T13:42:07.667 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/20' 2023-10-30T13:42:07.667 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/39' 2023-10-30T13:42:07.668 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/34' 2023-10-30T13:42:07.668 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/38' 2023-10-30T13:42:07.668 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/28' 2023-10-30T13:42:07.668 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/24' 2023-10-30T13:42:07.668 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/29' 2023-10-30T13:42:07.668 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/32' 2023-10-30T13:42:07.669 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/22' 2023-10-30T13:42:07.669 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/25' 2023-10-30T13:42:07.669 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/30' 2023-10-30T13:42:07.669 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/33' 2023-10-30T13:42:07.669 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/23' 2023-10-30T13:42:07.669 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/35' 2023-10-30T13:42:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/26' 2023-10-30T13:42:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:removed './204/37' 2023-10-30T13:42:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './204' 2023-10-30T13:42:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/21' 2023-10-30T13:42:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/27' 2023-10-30T13:42:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/31' 2023-10-30T13:42:07.671 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/36' 2023-10-30T13:42:07.671 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/20' 2023-10-30T13:42:07.671 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/39' 2023-10-30T13:42:07.671 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/34' 2023-10-30T13:42:07.671 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/38' 2023-10-30T13:42:07.671 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/28' 2023-10-30T13:42:07.672 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/24' 2023-10-30T13:42:07.672 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/29' 2023-10-30T13:42:07.672 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/32' 2023-10-30T13:42:07.672 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/22' 2023-10-30T13:42:07.672 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/25' 2023-10-30T13:42:07.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/30' 2023-10-30T13:42:07.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/33' 2023-10-30T13:42:07.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/23' 2023-10-30T13:42:07.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/35' 2023-10-30T13:42:07.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/26' 2023-10-30T13:42:07.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed './205/37' 2023-10-30T13:42:07.673 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './205' 2023-10-30T13:42:07.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/21' 2023-10-30T13:42:07.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/27' 2023-10-30T13:42:07.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/31' 2023-10-30T13:42:07.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/36' 2023-10-30T13:42:07.674 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/20' 2023-10-30T13:42:07.675 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/39' 2023-10-30T13:42:08.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd659f5640 1 Processor -- start 2023-10-30T13:42:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd659f5640 1 -- start start 2023-10-30T13:42:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd659f5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5f80 0x7fcd600c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd659f5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6015cda0 con 0x7fcd600c9b40 2023-10-30T13:42:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd659f5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6006f490 con 0x7fcd600c5ba0 2023-10-30T13:42:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd659f5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6015d990 con 0x7fcd600c5f80 2023-10-30T13:42:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5f7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fcd600c9b40 0x7fcd601597b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55068/0 (socket says 172.21.15.31:55068) 2023-10-30T13:42:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5f7fe640 1 -- 172.21.15.31:0/150972334 learned_addr learned my addr 172.21.15.31:0/150972334 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5e7fc640 1 --2- 172.21.15.31:0/150972334 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5f80 0x7fcd600c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5dffb640 1 -- 172.21.15.31:0/150972334 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015771646 0 0) 0x7fcd6015cda0 con 0x7fcd600c9b40 2023-10-30T13:42:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5dffb640 1 -- 172.21.15.31:0/150972334 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd48003530 con 0x7fcd600c9b40 2023-10-30T13:42:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5e7fc640 1 -- 172.21.15.31:0/150972334 >> v1:172.21.15.33:6789/0 conn(0x7fcd600c5ba0 legacy=0x7fcd60069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5e7fc640 1 -- 172.21.15.31:0/150972334 >> v1:172.21.15.31:6789/0 conn(0x7fcd600c9b40 legacy=0x7fcd601597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.000+0000 7fcd5e7fc640 1 -- 172.21.15.31:0/150972334 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6015cff0 con 0x7fcd600c5f80 2023-10-30T13:42:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd5e7fc640 1 --2- 172.21.15.31:0/150972334 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5f80 0x7fcd600c70f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd540097f0 tx=0x7fcd5401c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad987bc7b56d9d6c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd5dffb640 1 -- 172.21.15.31:0/150972334 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540ab600 con 0x7fcd600c5f80 2023-10-30T13:42:09.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd5dffb640 1 -- 172.21.15.31:0/150972334 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd540a7050 con 0x7fcd600c5f80 2023-10-30T13:42:09.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd5dffb640 1 -- 172.21.15.31:0/150972334 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540ab960 con 0x7fcd600c5f80 2023-10-30T13:42:09.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd659f5640 1 -- 172.21.15.31:0/150972334 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5f80 msgr2=0x7fcd600c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:09.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd659f5640 1 --2- 172.21.15.31:0/150972334 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5f80 0x7fcd600c70f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd540097f0 tx=0x7fcd5401c430 comp rx=0 tx=0).stop 2023-10-30T13:42:09.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd659f5640 1 -- 172.21.15.31:0/150972334 shutdown_connections 2023-10-30T13:42:09.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd659f5640 1 --2- 172.21.15.31:0/150972334 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5f80 0x7fcd600c70f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd659f5640 1 -- 172.21.15.31:0/150972334 >> 172.21.15.31:0/150972334 conn(0x7fcd6006afa0 msgr2=0x7fcd6006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd659f5640 1 -- 172.21.15.31:0/150972334 shutdown_connections 2023-10-30T13:42:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.004+0000 7fcd659f5640 1 -- 172.21.15.31:0/150972334 wait complete. 2023-10-30T13:42:09.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 Processor -- start 2023-10-30T13:42:09.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 -- start start 2023-10-30T13:42:09.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5ba0 0x7fcd601502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:09.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd600c5f80 0x7fcd60145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:09.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd600c9b40 0x7fcd60146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:09.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd60153aa0 con 0x7fcd600c9b40 2023-10-30T13:42:09.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd60146c80 con 0x7fcd600c5f80 2023-10-30T13:42:09.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd60150e80 con 0x7fcd600c5ba0 2023-10-30T13:42:09.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd600c9b40 0x7fcd60146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:09.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5effd640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5ba0 0x7fcd601502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:09.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd600c9b40 0x7fcd60146530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35654/0 (socket says 172.21.15.31:35654) 2023-10-30T13:42:09.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 -- 172.21.15.31:0/3261067625 learned_addr learned my addr 172.21.15.31:0/3261067625 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:09.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5e7fc640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd600c5f80 0x7fcd60145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:09.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 -- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5ba0 msgr2=0x7fcd601502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:09.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5ba0 0x7fcd601502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 -- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd600c5f80 msgr2=0x7fcd60145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:09.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd600c5f80 0x7fcd60145ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 -- 172.21.15.31:0/3261067625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54009420 con 0x7fcd600c9b40 2023-10-30T13:42:09.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5e7fc640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd600c5f80 0x7fcd60145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:09.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd5f7fe640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd600c9b40 0x7fcd60146530 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c9f0 tx=0x7fcd5800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:09.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd47fff640 1 -- 172.21.15.31:0/3261067625 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5800ab70 con 0x7fcd600c9b40 2023-10-30T13:42:09.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd47fff640 1 -- 172.21.15.31:0/3261067625 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5800acd0 con 0x7fcd600c9b40 2023-10-30T13:42:09.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd60153550 con 0x7fcd600c9b40 2023-10-30T13:42:09.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd47fff640 1 -- 172.21.15.31:0/3261067625 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58012600 con 0x7fcd600c9b40 2023-10-30T13:42:09.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.008+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6014f4b0 con 0x7fcd600c9b40 2023-10-30T13:42:09.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.012+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6014f310 con 0x7fcd600c9b40 2023-10-30T13:42:09.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.012+0000 7fcd47fff640 1 -- 172.21.15.31:0/3261067625 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd58007070 con 0x7fcd600c9b40 2023-10-30T13:42:09.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.012+0000 7fcd47fff640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcd4c0664a0 0x7fcd4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.012+0000 7fcd47fff640 1 -- 172.21.15.31:0/3261067625 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fcd58083eb0 con 0x7fcd600c9b40 2023-10-30T13:42:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.012+0000 7fcd5effd640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcd4c0664a0 0x7fcd4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.012+0000 7fcd5effd640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcd4c0664a0 0x7fcd4c068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd50001770 tx=0x7fcd50009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:09.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.016+0000 7fcd47fff640 1 -- 172.21.15.31:0/3261067625 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd6014f310 con 0x7fcd600c9b40 2023-10-30T13:42:09.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd60150ff0 con 0x7fcd600c9b40 2023-10-30T13:42:09.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd47fff640 1 -- 172.21.15.31:0/3261067625 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fcd60150ff0 con 0x7fcd600c9b40 2023-10-30T13:42:09.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:09.210 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:09.210 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:09.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcd4c0664a0 msgr2=0x7fcd4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:09.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcd4c0664a0 0x7fcd4c068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd50001770 tx=0x7fcd50009040 comp rx=0 tx=0).stop 2023-10-30T13:42:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd600c9b40 msgr2=0x7fcd60146530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd600c9b40 0x7fcd60146530 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c9f0 tx=0x7fcd5800ceb0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 shutdown_connections 2023-10-30T13:42:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcd4c0664a0 0x7fcd4c068960 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd600c9b40 0x7fcd60146530 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd600c5f80 0x7fcd60145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 --2- 172.21.15.31:0/3261067625 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcd600c5ba0 0x7fcd601502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:09.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 >> 172.21.15.31:0/3261067625 conn(0x7fcd6006afa0 msgr2=0x7fcd60070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:09.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 shutdown_connections 2023-10-30T13:42:09.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:09.208+0000 7fcd659f5640 1 -- 172.21.15.31:0/3261067625 wait complete. 2023-10-30T13:42:09.231 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:09.232 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 41 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:41:56.856155+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:11.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.384+0000 7f2758279640 1 Processor -- start 2023-10-30T13:42:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.384+0000 7f2758279640 1 -- start start 2023-10-30T13:42:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2758279640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2750150830 0x7f275006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2758279640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2750070520 0x7f27500709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2758279640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500c2310 con 0x7f27500cb7d0 2023-10-30T13:42:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2758279640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2750156710 con 0x7f2750150830 2023-10-30T13:42:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2758279640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27500c4360 con 0x7f2750070520 2023-10-30T13:42:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756015640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2750150830 0x7f275006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756015640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2750150830 0x7f275006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60170/0 (socket says 172.21.15.31:60170) 2023-10-30T13:42:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756015640 1 -- 172.21.15.31:0/2295459239 learned_addr learned my addr 172.21.15.31:0/2295459239 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756816640 1 --2- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2750070520 0x7f27500709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756015640 1 -- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2750070520 msgr2=0x7f27500709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756015640 1 --2- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2750070520 0x7f27500709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756015640 1 -- 172.21.15.31:0/2295459239 >> v1:172.21.15.33:6789/0 conn(0x7f27500cb7d0 legacy=0x7f27500c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.388+0000 7f2756015640 1 -- 172.21.15.31:0/2295459239 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27501569b0 con 0x7f2750150830 2023-10-30T13:42:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2756015640 1 --2- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750150830 0x7f275006b500 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f274c00bb10 tx=0x7f274c021400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=655af33ed6c4fbc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:11.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2755013640 1 -- 172.21.15.31:0/2295459239 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0b1600 con 0x7f2750150830 2023-10-30T13:42:11.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2755013640 1 -- 172.21.15.31:0/2295459239 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c0ad050 con 0x7f2750150830 2023-10-30T13:42:11.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2755013640 1 -- 172.21.15.31:0/2295459239 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0b1980 con 0x7f2750150830 2023-10-30T13:42:11.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 -- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750150830 msgr2=0x7f275006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:11.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 --2- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750150830 0x7f275006b500 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f274c00bb10 tx=0x7f274c021400 comp rx=0 tx=0).stop 2023-10-30T13:42:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 -- 172.21.15.31:0/2295459239 shutdown_connections 2023-10-30T13:42:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 --2- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2750070520 0x7f27500709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 --2- 172.21.15.31:0/2295459239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750150830 0x7f275006b500 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 -- 172.21.15.31:0/2295459239 >> 172.21.15.31:0/2295459239 conn(0x7f27500cac70 msgr2=0x7f27500cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 -- 172.21.15.31:0/2295459239 shutdown_connections 2023-10-30T13:42:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 -- 172.21.15.31:0/2295459239 wait complete. 2023-10-30T13:42:11.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 Processor -- start 2023-10-30T13:42:11.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.392+0000 7f2758279640 1 -- start start 2023-10-30T13:42:11.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750070520 0x7f2750154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:11.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27500cb7d0 0x7f2750147610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:11.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2750150830 0x7f275014e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:11.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27501548b0 con 0x7f2750150830 2023-10-30T13:42:11.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2750158ec0 con 0x7f2750070520 2023-10-30T13:42:11.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2750159a80 con 0x7f27500cb7d0 2023-10-30T13:42:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2750150830 0x7f275014e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2750150830 0x7f275014e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35658/0 (socket says 172.21.15.31:35658) 2023-10-30T13:42:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 -- 172.21.15.31:0/1060493306 learned_addr learned my addr 172.21.15.31:0/1060493306 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756015640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750070520 0x7f2750154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2755814640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27500cb7d0 0x7f2750147610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:11.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 -- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27500cb7d0 msgr2=0x7f2750147610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:11.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27500cb7d0 0x7f2750147610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 -- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750070520 msgr2=0x7f2750154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:11.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750070520 0x7f2750154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 -- 172.21.15.31:0/1060493306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c00b770 con 0x7f2750150830 2023-10-30T13:42:11.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756015640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750070520 0x7f2750154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:11.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2756816640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2750150830 0x7f275014e710 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f274800efc0 tx=0x7f274800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:11.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2746ffd640 1 -- 172.21.15.31:0/1060493306 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2748014480 con 0x7f2750150830 2023-10-30T13:42:11.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2746ffd640 1 -- 172.21.15.31:0/1060493306 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274800f040 con 0x7f2750150830 2023-10-30T13:42:11.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2746ffd640 1 -- 172.21.15.31:0/1060493306 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2748014820 con 0x7f2750150830 2023-10-30T13:42:11.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2750147b50 con 0x7f2750150830 2023-10-30T13:42:11.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2750157c40 con 0x7f2750150830 2023-10-30T13:42:11.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275004e170 con 0x7f2750150830 2023-10-30T13:42:11.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.396+0000 7f2746ffd640 1 -- 172.21.15.31:0/1060493306 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2748014980 con 0x7f2750150830 2023-10-30T13:42:11.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.400+0000 7f2746ffd640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27340664d0 0x7f2734068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:11.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.400+0000 7f2746ffd640 1 -- 172.21.15.31:0/1060493306 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f2748084a90 con 0x7f2750150830 2023-10-30T13:42:11.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.400+0000 7f2756015640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27340664d0 0x7f2734068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:11.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.400+0000 7f2756015640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27340664d0 0x7f2734068990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f274c0b3000 tx=0x7f274c0ac000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:11.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.404+0000 7f2746ffd640 1 -- 172.21.15.31:0/1060493306 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2748051d90 con 0x7f2750150830 2023-10-30T13:42:11.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.592+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2750153fd0 con 0x7f2750150830 2023-10-30T13:42:11.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.592+0000 7f2746ffd640 1 -- 172.21.15.31:0/1060493306 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f2748018020 con 0x7f2750150830 2023-10-30T13:42:11.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:11.595 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:11.595 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:11.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27340664d0 msgr2=0x7f2734068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:11.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27340664d0 0x7f2734068990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f274c0b3000 tx=0x7f274c0ac000 comp rx=0 tx=0).stop 2023-10-30T13:42:11.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2750150830 msgr2=0x7f275014e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:11.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2750150830 0x7f275014e710 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f274800efc0 tx=0x7f274800c490 comp rx=0 tx=0).stop 2023-10-30T13:42:11.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 shutdown_connections 2023-10-30T13:42:11.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27340664d0 0x7f2734068990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2750150830 0x7f275014e710 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27500cb7d0 0x7f2750147610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 --2- 172.21.15.31:0/1060493306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2750070520 0x7f2750154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:11.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 >> 172.21.15.31:0/1060493306 conn(0x7f27500cac70 msgr2=0x7f27500733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:11.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 shutdown_connections 2023-10-30T13:42:11.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:11.596+0000 7f2758279640 1 -- 172.21.15.31:0/1060493306 wait complete. 2023-10-30T13:42:11.616 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:13.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff6a1aea640 1 Processor -- start 2023-10-30T13:42:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff6a1aea640 1 -- start start 2023-10-30T13:42:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff6a1aea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff69c15cf90 0x7ff69c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff6a1aea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69c159f20 con 0x7ff69c153c80 2023-10-30T13:42:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff6a1aea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69c0c47e0 con 0x7ff69c0c4a10 2023-10-30T13:42:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff6a1aea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69c15dec0 con 0x7ff69c15cf90 2023-10-30T13:42:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff69c153c80 0x7ff69c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55074/0 (socket says 172.21.15.31:55074) 2023-10-30T13:42:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69b7fe640 1 -- 172.21.15.31:0/1972241096 learned_addr learned my addr 172.21.15.31:0/1972241096 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:13.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69bfff640 1 --2- 172.21.15.31:0/1972241096 >> v2:172.21.15.33:3300/0 conn(0x7ff69c15cf90 0x7ff69c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793744756 0 0) 0x7ff69c159f20 con 0x7ff69c153c80 2023-10-30T13:42:13.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff684003530 con 0x7ff69c153c80 2023-10-30T13:42:13.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 539582221 0 0) 0x7ff684003530 con 0x7ff69c153c80 2023-10-30T13:42:13.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 >> v2:172.21.15.33:3300/0 conn(0x7ff69c15cf90 msgr2=0x7ff69c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:13.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 --2- 172.21.15.31:0/1972241096 >> v2:172.21.15.33:3300/0 conn(0x7ff69c15cf90 0x7ff69c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 >> v1:172.21.15.33:6789/0 conn(0x7ff69c0c4a10 legacy=0x7ff69c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:13.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c15d620 con 0x7ff69c153c80 2023-10-30T13:42:13.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7ff68c001680 con 0x7ff69c153c80 2023-10-30T13:42:13.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 >> v1:172.21.15.31:6789/0 conn(0x7ff69c153c80 legacy=0x7ff69c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:13.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff684003980 0x7ff684003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6840047d0 0x7ff684004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff68400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c15dec0 con 0x7ff684005190 2023-10-30T13:42:13.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68400c040 con 0x7ff6840047d0 2023-10-30T13:42:13.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68400c200 con 0x7ff684003980 2023-10-30T13:42:13.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69bfff640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff68400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69b7fe640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff684003980 0x7ff684003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69affd640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6840047d0 0x7ff684004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69bfff640 1 -- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff684003980 msgr2=0x7ff684003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:13.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69bfff640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff684003980 0x7ff684003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69bfff640 1 -- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6840047d0 msgr2=0x7ff684004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:13.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69bfff640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6840047d0 0x7ff684004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69bfff640 1 -- 172.21.15.31:0/1972241096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff684003530 con 0x7ff684005190 2023-10-30T13:42:13.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.772+0000 7ff69affd640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6840047d0 0x7ff684004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:13.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff69bfff640 1 -- 172.21.15.31:0/1972241096 reap_dead start 2023-10-30T13:42:13.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff69bfff640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff68400b990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff694009fc0 tx=0x7ff69401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba1089783f2b258 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:13.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940a5cb0 con 0x7ff684005190 2023-10-30T13:42:13.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69400f040 con 0x7ff684005190 2023-10-30T13:42:13.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff69a7fc640 1 -- 172.21.15.31:0/1972241096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694003be0 con 0x7ff684005190 2023-10-30T13:42:13.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff6a1aea640 1 -- 172.21.15.31:0/1972241096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 msgr2=0x7ff68400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:13.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff68400b990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff694009fc0 tx=0x7ff69401cca0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff6a1aea640 1 -- 172.21.15.31:0/1972241096 shutdown_connections 2023-10-30T13:42:13.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/1972241096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff68400b990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff6a1aea640 1 -- 172.21.15.31:0/1972241096 >> 172.21.15.31:0/1972241096 conn(0x7ff69c0715c0 msgr2=0x7ff69c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:13.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff6a1aea640 1 -- 172.21.15.31:0/1972241096 shutdown_connections 2023-10-30T13:42:13.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.776+0000 7ff6a1aea640 1 -- 172.21.15.31:0/1972241096 wait complete. 2023-10-30T13:42:13.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 Processor -- start 2023-10-30T13:42:13.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 -- start start 2023-10-30T13:42:13.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff69c0c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff69c1f1d40 0x7ff69c15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff69c15ab60 0x7ff69c15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c14f270 con 0x7ff684005190 2023-10-30T13:42:13.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c157af0 con 0x7ff69c1f1d40 2023-10-30T13:42:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c14df30 con 0x7ff69c15ab60 2023-10-30T13:42:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff69c0c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff69c0c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35680/0 (socket says 172.21.15.31:35680) 2023-10-30T13:42:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 -- 172.21.15.31:0/501034217 learned_addr learned my addr 172.21.15.31:0/501034217 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69affd640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff69c1f1d40 0x7ff69c15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69bfff640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff69c15ab60 0x7ff69c15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 -- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff69c15ab60 msgr2=0x7ff69c15b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:13.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff69c15ab60 0x7ff69c15b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 -- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff69c1f1d40 msgr2=0x7ff69c15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:13.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff69c1f1d40 0x7ff69c15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 -- 172.21.15.31:0/501034217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c14e1d0 con 0x7ff684005190 2023-10-30T13:42:13.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff69b7fe640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff69c0c5630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00e780 tx=0x7ff68c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:13.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff683fff640 1 -- 172.21.15.31:0/501034217 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00ee90 con 0x7ff684005190 2023-10-30T13:42:13.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff683fff640 1 -- 172.21.15.31:0/501034217 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c00c870 con 0x7ff684005190 2023-10-30T13:42:13.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff683fff640 1 -- 172.21.15.31:0/501034217 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00cc20 con 0x7ff684005190 2023-10-30T13:42:13.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c14e4b0 con 0x7ff684005190 2023-10-30T13:42:13.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.780+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c14ea70 con 0x7ff684005190 2023-10-30T13:42:13.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.784+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c04e170 con 0x7ff684005190 2023-10-30T13:42:13.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.784+0000 7ff683fff640 1 -- 172.21.15.31:0/501034217 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff68c01d460 con 0x7ff684005190 2023-10-30T13:42:13.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.784+0000 7ff683fff640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6880664a0 0x7ff688068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:13.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.784+0000 7ff683fff640 1 -- 172.21.15.31:0/501034217 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ff68c083fd0 con 0x7ff684005190 2023-10-30T13:42:13.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.784+0000 7ff69affd640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6880664a0 0x7ff688068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:13.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.784+0000 7ff69affd640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6880664a0 0x7ff688068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff69c1f2140 tx=0x7ff690009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:13.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.788+0000 7ff683fff640 1 -- 172.21.15.31:0/501034217 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff68c051240 con 0x7ff684005190 2023-10-30T13:42:13.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.976+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69c0b5510 con 0x7ff684005190 2023-10-30T13:42:13.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff683fff640 1 -- 172.21.15.31:0/501034217 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7ff68c054860 con 0x7ff684005190 2023-10-30T13:42:13.980 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:13.980 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:13.981 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:13.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6880664a0 msgr2=0x7ff688068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:13.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6880664a0 0x7ff688068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff69c1f2140 tx=0x7ff690009040 comp rx=0 tx=0).stop 2023-10-30T13:42:13.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 msgr2=0x7ff69c0c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:13.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff69c0c5630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00e780 tx=0x7ff68c00ec40 comp rx=0 tx=0).stop 2023-10-30T13:42:13.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 shutdown_connections 2023-10-30T13:42:13.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6880664a0 0x7ff688068960 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff69c15ab60 0x7ff69c15b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff69c1f1d40 0x7ff69c15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 --2- 172.21.15.31:0/501034217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff684005190 0x7ff69c0c5630 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:13.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 >> 172.21.15.31:0/501034217 conn(0x7ff69c0715c0 msgr2=0x7ff69c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:13.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 shutdown_connections 2023-10-30T13:42:13.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:13.980+0000 7ff6a1aea640 1 -- 172.21.15.31:0/501034217 wait complete. 2023-10-30T13:42:14.001 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:16.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:16.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f346206b640 1 Processor -- start 2023-10-30T13:42:16.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f346206b640 1 -- start start 2023-10-30T13:42:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f346206b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 0x7f345c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f346206b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c0c2310 con 0x7f345c0cb7d0 2023-10-30T13:42:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f346206b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c156710 con 0x7f345c150830 2023-10-30T13:42:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f346206b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c06bc10 con 0x7f345c070520 2023-10-30T13:42:16.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f345c0cb7d0 0x7f345c0c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54652/0 (socket says 172.21.15.31:54652) 2023-10-30T13:42:16.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345affd640 1 -- 172.21.15.31:0/997408207 learned_addr learned my addr 172.21.15.31:0/997408207 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:16.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345bfff640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 0x7f345c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050291507 0 0) 0x7f345c0c2310 con 0x7f345c0cb7d0 2023-10-30T13:42:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3444005740 con 0x7f345c0cb7d0 2023-10-30T13:42:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1701772785 0 0) 0x7f3444005740 con 0x7f345c0cb7d0 2023-10-30T13:42:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 msgr2=0x7f345c0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345a7fc640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 0x7f345c0709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 >> v1:172.21.15.33:6789/0 conn(0x7f345c150830 legacy=0x7f345c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.156+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c0c4360 con 0x7f345c0cb7d0 2023-10-30T13:42:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f3450001680 con 0x7f345c0cb7d0 2023-10-30T13:42:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 >> v1:172.21.15.31:6789/0 conn(0x7f345c0cb7d0 legacy=0x7f345c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444005bc0 0x7f3444005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 0x7f34440069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444006f20 0x7f34440073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c06bc10 con 0x7f3444005bc0 2023-10-30T13:42:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344400dce0 con 0x7f3444006f20 2023-10-30T13:42:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344400dea0 con 0x7f345c070520 2023-10-30T13:42:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345b7fe640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444005bc0 0x7f3444005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345affd640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 0x7f34440069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345bfff640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444006f20 0x7f34440073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345b7fe640 1 -- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 msgr2=0x7f34440069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345b7fe640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f345c070520 0x7f34440069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345b7fe640 1 -- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444006f20 msgr2=0x7f34440073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345b7fe640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3444006f20 0x7f34440073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345b7fe640 1 -- 172.21.15.31:0/997408207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444005740 con 0x7f3444005bc0 2023-10-30T13:42:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.160+0000 7f345b7fe640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444005bc0 0x7f3444005fe0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f344c001750 tx=0x7f344c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a66cfaf3e350e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0052d0 con 0x7f3444005bc0 2023-10-30T13:42:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c0a7050 con 0x7f3444005bc0 2023-10-30T13:42:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f345a7fc640 1 -- 172.21.15.31:0/997408207 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c003600 con 0x7f3444005bc0 2023-10-30T13:42:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- 172.21.15.31:0/997408207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444005bc0 msgr2=0x7f3444005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 --2- 172.21.15.31:0/997408207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3444005bc0 0x7f3444005fe0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f344c001750 tx=0x7f344c01c3c0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f345bfff640 1 -- 172.21.15.31:0/997408207 reap_dead start 2023-10-30T13:42:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- 172.21.15.31:0/997408207 shutdown_connections 2023-10-30T13:42:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- 172.21.15.31:0/997408207 >> 172.21.15.31:0/997408207 conn(0x7f345c0cac70 msgr2=0x7f345c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- 172.21.15.31:0/997408207 shutdown_connections 2023-10-30T13:42:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- 172.21.15.31:0/997408207 wait complete. 2023-10-30T13:42:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 Processor -- start 2023-10-30T13:42:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- start start 2023-10-30T13:42:16.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3444005bc0 0x7f345c15a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f345c14bbe0 0x7f345c1ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f345c1ee750 0x7f345c1474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c157b20 con 0x7f345c14bbe0 2023-10-30T13:42:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c1edd80 con 0x7f345c1ee750 2023-10-30T13:42:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f346206b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c1eebd0 con 0x7f3444005bc0 2023-10-30T13:42:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f345affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f345c14bbe0 0x7f345c1ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f345affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f345c14bbe0 0x7f345c1ee210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33070/0 (socket says 172.21.15.31:33070) 2023-10-30T13:42:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.164+0000 7f345affd640 1 -- 172.21.15.31:0/2263774564 learned_addr learned my addr 172.21.15.31:0/2263774564 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345b7fe640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3444005bc0 0x7f345c15a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345bfff640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f345c1ee750 0x7f345c1474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345affd640 1 -- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3444005bc0 msgr2=0x7f345c15a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345affd640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3444005bc0 0x7f345c15a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345affd640 1 -- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f345c1ee750 msgr2=0x7f345c1474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345affd640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f345c1ee750 0x7f345c1474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345affd640 1 -- 172.21.15.31:0/2263774564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c157530 con 0x7f345c14bbe0 2023-10-30T13:42:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f345affd640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f345c14bbe0 0x7f345c1ee210 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3450012490 tx=0x7f3450012950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f3443fff640 1 -- 172.21.15.31:0/2263774564 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450014810 con 0x7f345c14bbe0 2023-10-30T13:42:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f3443fff640 1 -- 172.21.15.31:0/2263774564 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3450014970 con 0x7f345c14bbe0 2023-10-30T13:42:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f3443fff640 1 -- 172.21.15.31:0/2263774564 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450014d20 con 0x7f345c14bbe0 2023-10-30T13:42:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c157860 con 0x7f345c14bbe0 2023-10-30T13:42:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c0c9fe0 con 0x7f345c14bbe0 2023-10-30T13:42:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c066510 con 0x7f345c14bbe0 2023-10-30T13:42:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.168+0000 7f3443fff640 1 -- 172.21.15.31:0/2263774564 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34500153e0 con 0x7f345c14bbe0 2023-10-30T13:42:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.172+0000 7f3443fff640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3448066460 0x7f3448068920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.172+0000 7f3443fff640 1 -- 172.21.15.31:0/2263774564 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f34500869c0 con 0x7f345c14bbe0 2023-10-30T13:42:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.172+0000 7f345b7fe640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3448066460 0x7f3448068920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:16.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.172+0000 7f345b7fe640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3448066460 0x7f3448068920 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f34440079c0 tx=0x7f344c0011a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:16.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.176+0000 7f3443fff640 1 -- 172.21.15.31:0/2263774564 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3450053e20 con 0x7f345c14bbe0 2023-10-30T13:42:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f345c156f40 con 0x7f345c14bbe0 2023-10-30T13:42:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f3443fff640 1 -- 172.21.15.31:0/2263774564 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f345001c070 con 0x7f345c14bbe0 2023-10-30T13:42:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:16.366 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3448066460 msgr2=0x7f3448068920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3448066460 0x7f3448068920 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f34440079c0 tx=0x7f344c0011a0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f345c14bbe0 msgr2=0x7f345c1ee210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f345c14bbe0 0x7f345c1ee210 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3450012490 tx=0x7f3450012950 comp rx=0 tx=0).stop 2023-10-30T13:42:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 shutdown_connections 2023-10-30T13:42:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3448066460 0x7f3448068920 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f345c1ee750 0x7f345c1474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f345c14bbe0 0x7f345c1ee210 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 --2- 172.21.15.31:0/2263774564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3444005bc0 0x7f345c15a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 >> 172.21.15.31:0/2263774564 conn(0x7f345c0cac70 msgr2=0x7f345c14ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 shutdown_connections 2023-10-30T13:42:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:16.364+0000 7f346206b640 1 -- 172.21.15.31:0/2263774564 wait complete. 2023-10-30T13:42:16.387 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:18.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f745d3d3640 1 Processor -- start 2023-10-30T13:42:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f745d3d3640 1 -- start start 2023-10-30T13:42:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f745d3d3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7458153c80 0x7f745815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f745d3d3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f74580c42a0 0x7f74580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f745d3d3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7458159f20 con 0x7f74580c4a10 2023-10-30T13:42:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f745d3d3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745815dec0 con 0x7f7458153c80 2023-10-30T13:42:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f745d3d3640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74581599a0 con 0x7f74580c42a0 2023-10-30T13:42:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f7456ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7458153c80 0x7f745815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f744ffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f74580c4a10 0x7f745815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54666/0 (socket says 172.21.15.31:54666) 2023-10-30T13:42:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.540+0000 7f744ffff640 1 -- 172.21.15.31:0/1261644486 learned_addr learned my addr 172.21.15.31:0/1261644486 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:18.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f74577fe640 1 --2- 172.21.15.31:0/1261644486 >> v2:172.21.15.33:3300/0 conn(0x7f74580c42a0 0x7f74580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:18.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f7456ffd640 1 -- 172.21.15.31:0/1261644486 >> v2:172.21.15.33:3300/0 conn(0x7f74580c42a0 msgr2=0x7f74580ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:18.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f7456ffd640 1 --2- 172.21.15.31:0/1261644486 >> v2:172.21.15.33:3300/0 conn(0x7f74580c42a0 0x7f74580ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f7456ffd640 1 -- 172.21.15.31:0/1261644486 >> v1:172.21.15.31:6789/0 conn(0x7f74580c4a10 legacy=0x7f745815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:18.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f7456ffd640 1 -- 172.21.15.31:0/1261644486 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745815d620 con 0x7f7458153c80 2023-10-30T13:42:18.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f74567fc640 1 -- 172.21.15.31:0/1261644486 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358077165 0 0) 0x7f7458159f20 con 0x7f74580c4a10 2023-10-30T13:42:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f7456ffd640 1 --2- 172.21.15.31:0/1261644486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7458153c80 0x7f745815cf90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f74580b1d40 tx=0x7f744801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d44abb199be8d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f74567fc640 1 -- 172.21.15.31:0/1261644486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480a8950 con 0x7f7458153c80 2023-10-30T13:42:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f74567fc640 1 -- 172.21.15.31:0/1261644486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74480a8ab0 con 0x7f7458153c80 2023-10-30T13:42:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.544+0000 7f74567fc640 1 -- 172.21.15.31:0/1261644486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480a8e40 con 0x7f7458153c80 2023-10-30T13:42:18.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- 172.21.15.31:0/1261644486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7458153c80 msgr2=0x7f745815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:18.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 --2- 172.21.15.31:0/1261644486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7458153c80 0x7f745815cf90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f74580b1d40 tx=0x7f744801ec50 comp rx=0 tx=0).stop 2023-10-30T13:42:18.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- 172.21.15.31:0/1261644486 shutdown_connections 2023-10-30T13:42:18.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 --2- 172.21.15.31:0/1261644486 >> v2:172.21.15.33:3300/0 conn(0x7f74580c42a0 0x7f74580ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 --2- 172.21.15.31:0/1261644486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7458153c80 0x7f745815cf90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- 172.21.15.31:0/1261644486 >> 172.21.15.31:0/1261644486 conn(0x7f74580715c0 msgr2=0x7f74580719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:18.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- 172.21.15.31:0/1261644486 shutdown_connections 2023-10-30T13:42:18.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- 172.21.15.31:0/1261644486 wait complete. 2023-10-30T13:42:18.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 Processor -- start 2023-10-30T13:42:18.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- start start 2023-10-30T13:42:18.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74580c42a0 0x7f745806c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:18.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74580c4a10 0x7f7458151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:18.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7458153c80 0x7f7458151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:18.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745815cae0 con 0x7f74580c42a0 2023-10-30T13:42:18.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745814b6e0 con 0x7f74580c4a10 2023-10-30T13:42:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f745d3d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745815b930 con 0x7f7458153c80 2023-10-30T13:42:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74580c42a0 0x7f745806c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74580c42a0 0x7f745806c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33088/0 (socket says 172.21.15.31:33088) 2023-10-30T13:42:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 -- 172.21.15.31:0/60040520 learned_addr learned my addr 172.21.15.31:0/60040520 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f744ffff640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74580c4a10 0x7f7458151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:18.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f74577fe640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7458153c80 0x7f7458151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:18.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 -- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7458153c80 msgr2=0x7f7458151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:18.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7458153c80 0x7f7458151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 -- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74580c4a10 msgr2=0x7f7458151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:18.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74580c4a10 0x7f7458151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 -- 172.21.15.31:0/60040520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7448009420 con 0x7f74580c42a0 2023-10-30T13:42:18.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.548+0000 7f7456ffd640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74580c42a0 0x7f745806c240 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f744801b860 tx=0x7f744801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:18.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f744f7fe640 1 -- 172.21.15.31:0/60040520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480b0ea0 con 0x7f74580c42a0 2023-10-30T13:42:18.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f744f7fe640 1 -- 172.21.15.31:0/60040520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7448006e20 con 0x7f74580c42a0 2023-10-30T13:42:18.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f744f7fe640 1 -- 172.21.15.31:0/60040520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448004230 con 0x7f74580c42a0 2023-10-30T13:42:18.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745815bb80 con 0x7f74580c42a0 2023-10-30T13:42:18.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745815c140 con 0x7f74580c42a0 2023-10-30T13:42:18.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745804e170 con 0x7f74580c42a0 2023-10-30T13:42:18.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f744f7fe640 1 -- 172.21.15.31:0/60040520 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74480b7040 con 0x7f74580c42a0 2023-10-30T13:42:18.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f744f7fe640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f74380664a0 0x7f7438068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:18.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f744f7fe640 1 -- 172.21.15.31:0/60040520 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f7448120f10 con 0x7f74580c42a0 2023-10-30T13:42:18.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.552+0000 7f744ffff640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f74380664a0 0x7f7438068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:18.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.556+0000 7f744ffff640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f74380664a0 0x7f7438068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7458151920 tx=0x7f7440007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:18.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.556+0000 7f744f7fe640 1 -- 172.21.15.31:0/60040520 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74480ee210 con 0x7f74580c42a0 2023-10-30T13:42:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.744+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74581594f0 con 0x7f74580c42a0 2023-10-30T13:42:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.744+0000 7f744f7fe640 1 -- 172.21.15.31:0/60040520 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f74480b5090 con 0x7f74580c42a0 2023-10-30T13:42:18.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:18.747 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:41:56.856155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2023-10-30T13:42:18.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f74380664a0 msgr2=0x7f7438068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:18.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f74380664a0 0x7f7438068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7458151920 tx=0x7f7440007b80 comp rx=0 tx=0).stop 2023-10-30T13:42:18.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74580c42a0 msgr2=0x7f745806c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:18.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74580c42a0 0x7f745806c240 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f744801b860 tx=0x7f744801b9f0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 shutdown_connections 2023-10-30T13:42:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f74380664a0 0x7f7438068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7458153c80 0x7f7458151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74580c4a10 0x7f7458151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 --2- 172.21.15.31:0/60040520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74580c42a0 0x7f745806c240 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 >> 172.21.15.31:0/60040520 conn(0x7f74580715c0 msgr2=0x7f74580c5650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 shutdown_connections 2023-10-30T13:42:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:18.748+0000 7f745d3d3640 1 -- 172.21.15.31:0/60040520 wait complete. 2023-10-30T13:42:18.768 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:20.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:20.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.920+0000 7fbc069b8640 1 Processor -- start 2023-10-30T13:42:20.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc069b8640 1 -- start start 2023-10-30T13:42:20.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc069b8640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fbc00150830 0x7fbc0006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc069b8640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc000c2310 con 0x7fbc00070520 2023-10-30T13:42:20.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc069b8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc00156710 con 0x7fbc000cb7d0 2023-10-30T13:42:20.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc069b8640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc000c4360 con 0x7fbc00150830 2023-10-30T13:42:20.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fbc00070520 0x7fbc00160f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54668/0 (socket says 172.21.15.31:54668) 2023-10-30T13:42:20.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 -- 172.21.15.31:0/3219146287 learned_addr learned my addr 172.21.15.31:0/3219146287 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:20.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfffff640 1 --2- 172.21.15.31:0/3219146287 >> v2:172.21.15.33:3300/0 conn(0x7fbc00150830 0x7fbc0006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677632863 0 0) 0x7fbc000c2310 con 0x7fbc00070520 2023-10-30T13:42:20.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe8005740 con 0x7fbc00070520 2023-10-30T13:42:20.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811440871 0 0) 0x7fbbe8005740 con 0x7fbc00070520 2023-10-30T13:42:20.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 >> v2:172.21.15.33:3300/0 conn(0x7fbc00150830 msgr2=0x7fbc0006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:20.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 --2- 172.21.15.31:0/3219146287 >> v2:172.21.15.33:3300/0 conn(0x7fbc00150830 0x7fbc0006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 >> v1:172.21.15.33:6789/0 conn(0x7fbc000cb7d0 legacy=0x7fbc000c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc001569b0 con 0x7fbc00070520 2023-10-30T13:42:20.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fbbf8001a20 con 0x7fbc00070520 2023-10-30T13:42:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 >> v1:172.21.15.31:6789/0 conn(0x7fbc00070520 legacy=0x7fbc00160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbe8005b90 0x7fbbe8005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbbe80069e0 0x7fbbe8006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbbe800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc000c4360 con 0x7fbbe80073a0 2023-10-30T13:42:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe800e250 con 0x7fbbe8005b90 2023-10-30T13:42:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe800e410 con 0x7fbbe80069e0 2023-10-30T13:42:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbbe800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbfffff640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbe8005b90 0x7fbbe8005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbbff7fe640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbbe80069e0 0x7fbbe8006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 -- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbbe80069e0 msgr2=0x7fbbe8006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:20.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbbe80069e0 0x7fbbe8006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 -- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbe8005b90 msgr2=0x7fbbe8005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:20.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbe8005b90 0x7fbbe8005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.924+0000 7fbc04f55640 1 -- 172.21.15.31:0/3219146287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8005740 con 0x7fbbe80073a0 2023-10-30T13:42:20.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc04f55640 1 -- 172.21.15.31:0/3219146287 reap_dead start 2023-10-30T13:42:20.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc04f55640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbbe800dba0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc000c24a0 tx=0x7fbbf801d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71fd9803b8469ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:20.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf80ad630 con 0x7fbbe80073a0 2023-10-30T13:42:20.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf80a8070 con 0x7fbbe80073a0 2023-10-30T13:42:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbbfeffd640 1 -- 172.21.15.31:0/3219146287 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf80ad990 con 0x7fbbe80073a0 2023-10-30T13:42:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc069b8640 1 -- 172.21.15.31:0/3219146287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 msgr2=0x7fbbe800dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc069b8640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbbe800dba0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc000c24a0 tx=0x7fbbf801d300 comp rx=0 tx=0).stop 2023-10-30T13:42:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc069b8640 1 -- 172.21.15.31:0/3219146287 shutdown_connections 2023-10-30T13:42:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc069b8640 1 --2- 172.21.15.31:0/3219146287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbbe800dba0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc069b8640 1 -- 172.21.15.31:0/3219146287 >> 172.21.15.31:0/3219146287 conn(0x7fbc000cac70 msgr2=0x7fbc000cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:20.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc069b8640 1 -- 172.21.15.31:0/3219146287 shutdown_connections 2023-10-30T13:42:20.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.928+0000 7fbc069b8640 1 -- 172.21.15.31:0/3219146287 wait complete. 2023-10-30T13:42:20.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 Processor -- start 2023-10-30T13:42:20.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 -- start start 2023-10-30T13:42:20.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbc000ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbc00147c30 0x7fbc000c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc00155650 0x7fbc001edcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00147810 con 0x7fbbe80073a0 2023-10-30T13:42:20.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00145d90 con 0x7fbc00155650 2023-10-30T13:42:20.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00147450 con 0x7fbc00147c30 2023-10-30T13:42:20.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbc000ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbc000ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33104/0 (socket says 172.21.15.31:33104) 2023-10-30T13:42:20.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 -- 172.21.15.31:0/1724655707 learned_addr learned my addr 172.21.15.31:0/1724655707 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:20.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc04f55640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc00155650 0x7fbc001edcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbff7fe640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbc00147c30 0x7fbc000c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 -- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbc00147c30 msgr2=0x7fbc000c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbc00147c30 0x7fbc000c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 -- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc00155650 msgr2=0x7fbc001edcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc00155650 0x7fbc001edcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 -- 172.21.15.31:0/1724655707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc001457d0 con 0x7fbbe80073a0 2023-10-30T13:42:20.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfffff640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbc000ca600 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0011960 tx=0x7fbbf0011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:20.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfcff9640 1 -- 172.21.15.31:0/1724655707 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000faf0 con 0x7fbbe80073a0 2023-10-30T13:42:20.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfcff9640 1 -- 172.21.15.31:0/1724655707 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf000fc50 con 0x7fbbe80073a0 2023-10-30T13:42:20.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbbfcff9640 1 -- 172.21.15.31:0/1724655707 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0014440 con 0x7fbbe80073a0 2023-10-30T13:42:20.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc00145ab0 con 0x7fbbe80073a0 2023-10-30T13:42:20.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.932+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc00144930 con 0x7fbbe80073a0 2023-10-30T13:42:20.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.936+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc00147a80 con 0x7fbbe80073a0 2023-10-30T13:42:20.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.936+0000 7fbbfcff9640 1 -- 172.21.15.31:0/1724655707 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbbf00145a0 con 0x7fbbe80073a0 2023-10-30T13:42:20.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.936+0000 7fbbfcff9640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbbec0664d0 0x7fbbec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:20.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.936+0000 7fbbfcff9640 1 -- 172.21.15.31:0/1724655707 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fbbf00848d0 con 0x7fbbe80073a0 2023-10-30T13:42:20.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.936+0000 7fbbff7fe640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbbec0664d0 0x7fbbec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:20.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.936+0000 7fbbff7fe640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbbec0664d0 0x7fbbec068990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbc00156640 tx=0x7fbbf4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:20.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:20.940+0000 7fbbfcff9640 1 -- 172.21.15.31:0/1724655707 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbbf0051b40 con 0x7fbbe80073a0 2023-10-30T13:42:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc001472b0 con 0x7fbbe80073a0 2023-10-30T13:42:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbbfcff9640 1 -- 172.21.15.31:0/1724655707 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fbc001472b0 con 0x7fbbe80073a0 2023-10-30T13:42:21.133 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:21.133 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:19.345575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:21.133 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2023-10-30T13:42:21.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbbec0664d0 msgr2=0x7fbbec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:21.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbbec0664d0 0x7fbbec068990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbc00156640 tx=0x7fbbf4009040 comp rx=0 tx=0).stop 2023-10-30T13:42:21.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 msgr2=0x7fbc000ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:21.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbc000ca600 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0011960 tx=0x7fbbf0011e20 comp rx=0 tx=0).stop 2023-10-30T13:42:21.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 shutdown_connections 2023-10-30T13:42:21.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbbec0664d0 0x7fbbec068990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:21.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc00155650 0x7fbc001edcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:21.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbc00147c30 0x7fbc000c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:21.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 --2- 172.21.15.31:0/1724655707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe80073a0 0x7fbc000ca600 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:21.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 >> 172.21.15.31:0/1724655707 conn(0x7fbc000cac70 msgr2=0x7fbc0014f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:21.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 shutdown_connections 2023-10-30T13:42:21.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:21.132+0000 7fbc069b8640 1 -- 172.21.15.31:0/1724655707 wait complete. 2023-10-30T13:42:21.154 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:21.154 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 41, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 41, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 41 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:19.345575+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:23.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:23.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26f4d6640 1 Processor -- start 2023-10-30T13:42:23.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26f4d6640 1 -- start start 2023-10-30T13:42:23.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26f4d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe2681516c0 0x7fe2680c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:23.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26f4d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:23.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26f4d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe268151aa0 con 0x7fe2680c82d0 2023-10-30T13:42:23.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26f4d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2680c74a0 con 0x7fe2681516c0 2023-10-30T13:42:23.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26f4d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680c2e70 con 0x7fe2680c2500 2023-10-30T13:42:23.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26d272640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe2681516c0 0x7fe2680c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:23.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.308+0000 7fe26da73640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:23.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26ca71640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fe2680c82d0 0x7fe2680ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:49178/0 (socket says 172.21.15.31:49178) 2023-10-30T13:42:23.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26d272640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe2681516c0 0x7fe2680c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:47274/0 (socket says 172.21.15.31:47274) 2023-10-30T13:42:23.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26ca71640 1 -- 172.21.15.31:0/678437027 learned_addr learned my addr 172.21.15.31:0/678437027 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:23.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26da73640 1 -- 172.21.15.31:0/678437027 >> v2:172.21.15.33:3300/0 conn(0x7fe2681516c0 msgr2=0x7fe2680c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:23.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26da73640 1 --2- 172.21.15.31:0/678437027 >> v2:172.21.15.33:3300/0 conn(0x7fe2681516c0 0x7fe2680c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26da73640 1 -- 172.21.15.31:0/678437027 >> v1:172.21.15.33:6789/0 conn(0x7fe2680c82d0 legacy=0x7fe2680ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:23.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe25ffff640 1 -- 172.21.15.31:0/678437027 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676317854 0 0) 0x7fe268151aa0 con 0x7fe2680c82d0 2023-10-30T13:42:23.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26da73640 1 -- 172.21.15.31:0/678437027 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe268158af0 con 0x7fe2680c2500 2023-10-30T13:42:23.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26d272640 1 --2- 172.21.15.31:0/678437027 >> v2:172.21.15.33:3300/0 conn(0x7fe2681516c0 0x7fe2680c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:23.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.312+0000 7fe26da73640 1 --2- 172.21.15.31:0/678437027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c2930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe260009d90 tx=0x7fe26001c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eba3911cd08d4b94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:23.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe25ffff640 1 -- 172.21.15.31:0/678437027 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2600a5810 con 0x7fe2680c2500 2023-10-30T13:42:23.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe25ffff640 1 -- 172.21.15.31:0/678437027 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2600a5970 con 0x7fe2680c2500 2023-10-30T13:42:23.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe25ffff640 1 -- 172.21.15.31:0/678437027 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2600a5d00 con 0x7fe2680c2500 2023-10-30T13:42:23.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- 172.21.15.31:0/678437027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 msgr2=0x7fe2680c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:23.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/678437027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c2930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe260009d90 tx=0x7fe26001c950 comp rx=0 tx=0).stop 2023-10-30T13:42:23.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- 172.21.15.31:0/678437027 shutdown_connections 2023-10-30T13:42:23.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/678437027 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c2930 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/678437027 >> v2:172.21.15.33:3300/0 conn(0x7fe2681516c0 0x7fe2680c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- 172.21.15.31:0/678437027 >> 172.21.15.31:0/678437027 conn(0x7fe268071fb0 msgr2=0x7fe2680723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:23.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- 172.21.15.31:0/678437027 shutdown_connections 2023-10-30T13:42:23.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- 172.21.15.31:0/678437027 wait complete. 2023-10-30T13:42:23.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 Processor -- start 2023-10-30T13:42:23.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- start start 2023-10-30T13:42:23.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:23.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe2680c82d0 0x7fe268140670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:23.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2681516c0 0x7fe2681edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:23.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681edb80 con 0x7fe2681516c0 2023-10-30T13:42:23.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26806bd30 con 0x7fe2680c82d0 2023-10-30T13:42:23.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.316+0000 7fe26f4d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681f1c90 con 0x7fe2680c2500 2023-10-30T13:42:23.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2681516c0 0x7fe2681edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:23.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2681516c0 0x7fe2681edfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33112/0 (socket says 172.21.15.31:33112) 2023-10-30T13:42:23.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 -- 172.21.15.31:0/3478429888 learned_addr learned my addr 172.21.15.31:0/3478429888 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:23.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26d272640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:23.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26ca71640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe2680c82d0 0x7fe268140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:23.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 -- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 msgr2=0x7fe2680c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:23.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 -- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe2680c82d0 msgr2=0x7fe268140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:23.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe2680c82d0 0x7fe268140670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 -- 172.21.15.31:0/3478429888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264009420 con 0x7fe2681516c0 2023-10-30T13:42:23.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26ca71640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe2680c82d0 0x7fe268140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:23.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26da73640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2681516c0 0x7fe2681edfa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe26001b000 tx=0x7fe2600196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:23.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe25dffb640 1 -- 172.21.15.31:0/3478429888 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2600b9860 con 0x7fe2681516c0 2023-10-30T13:42:23.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe25dffb640 1 -- 172.21.15.31:0/3478429888 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2600b99c0 con 0x7fe2681516c0 2023-10-30T13:42:23.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe25dffb640 1 -- 172.21.15.31:0/3478429888 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2600b9d50 con 0x7fe2681516c0 2023-10-30T13:42:23.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe260009a70 con 0x7fe2681516c0 2023-10-30T13:42:23.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26806b490 con 0x7fe2681516c0 2023-10-30T13:42:23.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2681ed8e0 con 0x7fe2681516c0 2023-10-30T13:42:23.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.320+0000 7fe25dffb640 1 -- 172.21.15.31:0/3478429888 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe2600ad440 con 0x7fe2681516c0 2023-10-30T13:42:23.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.324+0000 7fe25dffb640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe24c0664a0 0x7fe24c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:23.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.324+0000 7fe25dffb640 1 -- 172.21.15.31:0/3478429888 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fe26011f490 con 0x7fe2681516c0 2023-10-30T13:42:23.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.324+0000 7fe26d272640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe24c0664a0 0x7fe24c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:23.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.324+0000 7fe26d272640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe24c0664a0 0x7fe24c068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe264009cf0 tx=0x7fe264009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:23.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.328+0000 7fe25dffb640 1 -- 172.21.15.31:0/3478429888 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2681ed8e0 con 0x7fe2681516c0 2023-10-30T13:42:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2681ed8e0 con 0x7fe2681516c0 2023-10-30T13:42:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe25dffb640 1 -- 172.21.15.31:0/3478429888 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fe2681ed8e0 con 0x7fe2681516c0 2023-10-30T13:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:19.345575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2023-10-30T13:42:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe24c0664a0 msgr2=0x7fe24c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe24c0664a0 0x7fe24c068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe264009cf0 tx=0x7fe264009190 comp rx=0 tx=0).stop 2023-10-30T13:42:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2681516c0 msgr2=0x7fe2681edfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2681516c0 0x7fe2681edfa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe26001b000 tx=0x7fe2600196f0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 shutdown_connections 2023-10-30T13:42:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe24c0664a0 0x7fe24c068960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2681516c0 0x7fe2681edfa0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe2680c82d0 0x7fe268140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 --2- 172.21.15.31:0/3478429888 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2680c2500 0x7fe2680c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.520+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 >> 172.21.15.31:0/3478429888 conn(0x7fe268071fb0 msgr2=0x7fe26806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.524+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 shutdown_connections 2023-10-30T13:42:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:23.524+0000 7fe26f4d6640 1 -- 172.21.15.31:0/3478429888 wait complete. 2023-10-30T13:42:23.543 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:25.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:25.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.696+0000 7fd04e71d640 1 Processor -- start 2023-10-30T13:42:25.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.696+0000 7fd04e71d640 1 -- start start 2023-10-30T13:42:25.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd04e71d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd048153aa0 0x7fd04815d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:25.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd04e71d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480701e0 0x7fd0480ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:25.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd04e71d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0480c7b00 con 0x7fd0480cd420 2023-10-30T13:42:25.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd04e71d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd048070a50 con 0x7fd048153aa0 2023-10-30T13:42:25.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd04e71d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04815dd80 con 0x7fd0480701e0 2023-10-30T13:42:25.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd047fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd048153aa0 0x7fd04815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:25.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd0477fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480701e0 0x7fd0480ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:25.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd04ccba640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fd0480cd420 0x7fd04806b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:56348/0 (socket says 172.21.15.31:56348) 2023-10-30T13:42:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd0477fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480701e0 0x7fd0480ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:43482/0 (socket says 172.21.15.31:43482) 2023-10-30T13:42:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd04ccba640 1 -- 172.21.15.31:0/3997223411 learned_addr learned my addr 172.21.15.31:0/3997223411 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd046ffd640 1 -- 172.21.15.31:0/3997223411 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258217680 0 0) 0x7fd0480c7b00 con 0x7fd0480cd420 2023-10-30T13:42:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd046ffd640 1 -- 172.21.15.31:0/3997223411 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd030003530 con 0x7fd0480cd420 2023-10-30T13:42:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd0477fe640 1 -- 172.21.15.31:0/3997223411 >> v2:172.21.15.33:3300/0 conn(0x7fd048153aa0 msgr2=0x7fd04815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd0477fe640 1 --2- 172.21.15.31:0/3997223411 >> v2:172.21.15.33:3300/0 conn(0x7fd048153aa0 0x7fd04815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd0477fe640 1 -- 172.21.15.31:0/3997223411 >> v1:172.21.15.33:6789/0 conn(0x7fd0480cd420 legacy=0x7fd04806b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd0477fe640 1 -- 172.21.15.31:0/3997223411 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0480c2050 con 0x7fd0480701e0 2023-10-30T13:42:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.700+0000 7fd047fff640 1 --2- 172.21.15.31:0/3997223411 >> v2:172.21.15.33:3300/0 conn(0x7fd048153aa0 0x7fd04815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:25.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd0477fe640 1 --2- 172.21.15.31:0/3997223411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480701e0 0x7fd0480ccee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0097c0 tx=0x7fd03c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a25ffb380f842c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:25.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd046ffd640 1 -- 172.21.15.31:0/3997223411 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0a8970 con 0x7fd0480701e0 2023-10-30T13:42:25.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd046ffd640 1 -- 172.21.15.31:0/3997223411 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c0a8ad0 con 0x7fd0480701e0 2023-10-30T13:42:25.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd046ffd640 1 -- 172.21.15.31:0/3997223411 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0a8e60 con 0x7fd0480701e0 2023-10-30T13:42:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 -- 172.21.15.31:0/3997223411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480701e0 msgr2=0x7fd0480ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 --2- 172.21.15.31:0/3997223411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480701e0 0x7fd0480ccee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0097c0 tx=0x7fd03c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:42:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 -- 172.21.15.31:0/3997223411 shutdown_connections 2023-10-30T13:42:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 --2- 172.21.15.31:0/3997223411 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480701e0 0x7fd0480ccee0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 --2- 172.21.15.31:0/3997223411 >> v2:172.21.15.33:3300/0 conn(0x7fd048153aa0 0x7fd04815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 -- 172.21.15.31:0/3997223411 >> 172.21.15.31:0/3997223411 conn(0x7fd048071fb0 msgr2=0x7fd0480723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 -- 172.21.15.31:0/3997223411 shutdown_connections 2023-10-30T13:42:25.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 -- 172.21.15.31:0/3997223411 wait complete. 2023-10-30T13:42:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 Processor -- start 2023-10-30T13:42:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.704+0000 7fd04e71d640 1 -- start start 2023-10-30T13:42:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0480701e0 0x7fd048149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480cd420 0x7fd0481576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd048153aa0 0x7fd048157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04814a390 con 0x7fd0480701e0 2023-10-30T13:42:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd048155890 con 0x7fd048153aa0 2023-10-30T13:42:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd048155b50 con 0x7fd0480cd420 2023-10-30T13:42:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0480701e0 0x7fd048149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0480701e0 0x7fd048149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47694/0 (socket says 172.21.15.31:47694) 2023-10-30T13:42:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 -- 172.21.15.31:0/328393826 learned_addr learned my addr 172.21.15.31:0/328393826 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04ccba640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd048153aa0 0x7fd048157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd0477fe640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480cd420 0x7fd0481576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 -- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480cd420 msgr2=0x7fd0481576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480cd420 0x7fd0481576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 -- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd048153aa0 msgr2=0x7fd048157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd048153aa0 0x7fd048157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 -- 172.21.15.31:0/328393826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd038009420 con 0x7fd0480701e0 2023-10-30T13:42:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd047fff640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0480701e0 0x7fd048149e50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd03800e980 tx=0x7fd03800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd044ff9640 1 -- 172.21.15.31:0/328393826 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03800ca00 con 0x7fd0480701e0 2023-10-30T13:42:25.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd044ff9640 1 -- 172.21.15.31:0/328393826 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03800cb60 con 0x7fd0480701e0 2023-10-30T13:42:25.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c009420 con 0x7fd0480701e0 2023-10-30T13:42:25.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd044ff9640 1 -- 172.21.15.31:0/328393826 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038013470 con 0x7fd0480701e0 2023-10-30T13:42:25.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd048159d60 con 0x7fd0480701e0 2023-10-30T13:42:25.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.708+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04804e170 con 0x7fd0480701e0 2023-10-30T13:42:25.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.712+0000 7fd044ff9640 1 -- 172.21.15.31:0/328393826 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0380135d0 con 0x7fd0480701e0 2023-10-30T13:42:25.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.712+0000 7fd044ff9640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd0340664d0 0x7fd034068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:25.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.712+0000 7fd044ff9640 1 -- 172.21.15.31:0/328393826 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fd038084280 con 0x7fd0480701e0 2023-10-30T13:42:25.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.712+0000 7fd0477fe640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd0340664d0 0x7fd034068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:25.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.712+0000 7fd0477fe640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd0340664d0 0x7fd034068990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0b2000 tx=0x7fd03c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.716+0000 7fd044ff9640 1 -- 172.21.15.31:0/328393826 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd038051500 con 0x7fd0480701e0 2023-10-30T13:42:25.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:25.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:25.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.908+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0480c7a70 con 0x7fd0480701e0 2023-10-30T13:42:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.908+0000 7fd044ff9640 1 -- 172.21.15.31:0/328393826 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fd038054b20 con 0x7fd0480701e0 2023-10-30T13:42:25.910 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:25.910 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:24.344953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:25.911 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 42 2023-10-30T13:42:25.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd0340664d0 msgr2=0x7fd034068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:25.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd0340664d0 0x7fd034068990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0b2000 tx=0x7fd03c008040 comp rx=0 tx=0).stop 2023-10-30T13:42:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0480701e0 msgr2=0x7fd048149e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0480701e0 0x7fd048149e50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd03800e980 tx=0x7fd03800ee40 comp rx=0 tx=0).stop 2023-10-30T13:42:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 shutdown_connections 2023-10-30T13:42:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd0340664d0 0x7fd034068990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd048153aa0 0x7fd048157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd0480cd420 0x7fd0481576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 --2- 172.21.15.31:0/328393826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0480701e0 0x7fd048149e50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 >> 172.21.15.31:0/328393826 conn(0x7fd048071fb0 msgr2=0x7fd0480ca260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 shutdown_connections 2023-10-30T13:42:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:25.912+0000 7fd04e71d640 1 -- 172.21.15.31:0/328393826 wait complete. 2023-10-30T13:42:25.931 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:27.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 Processor -- start 2023-10-30T13:42:28.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 -- start start 2023-10-30T13:42:28.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:28.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1fd4070c30 0x7f1fd4071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:28.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f1fd4159970 0x7f1fd4159e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd4072390 con 0x7f1fd4070c30 2023-10-30T13:42:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd406e390 con 0x7f1fd4159970 2023-10-30T13:42:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fda417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd40660e0 con 0x7f1fd4153c10 2023-10-30T13:42:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1fd4070c30 0x7f1fd4071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1fd4070c30 0x7f1fd4071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47698/0 (socket says 172.21.15.31:47698) 2023-10-30T13:42:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 -- 172.21.15.31:0/1127185190 learned_addr learned my addr 172.21.15.31:0/1127185190 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd89b4640 1 --2- 172.21.15.31:0/1127185190 >> v2:172.21.15.33:3300/0 conn(0x7f1fd4159970 0x7f1fd4159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd3fff640 1 --2- 172.21.15.31:0/1127185190 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 -- 172.21.15.31:0/1127185190 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 msgr2=0x7f1fd4065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 --2- 172.21.15.31:0/1127185190 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 -- 172.21.15.31:0/1127185190 >> v2:172.21.15.33:3300/0 conn(0x7f1fd4159970 msgr2=0x7f1fd4159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 --2- 172.21.15.31:0/1127185190 >> v2:172.21.15.33:3300/0 conn(0x7f1fd4159970 0x7f1fd4159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd37fe640 1 -- 172.21.15.31:0/1127185190 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd40715b0 con 0x7f1fd4070c30 2023-10-30T13:42:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd3fff640 1 --2- 172.21.15.31:0/1127185190 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.088+0000 7f1fd89b4640 1 --2- 172.21.15.31:0/1127185190 >> v2:172.21.15.33:3300/0 conn(0x7f1fd4159970 0x7f1fd4159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.092+0000 7f1fd37fe640 1 --2- 172.21.15.31:0/1127185190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd4071070 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4072de0 tx=0x7f1fc801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd20a76aa11f993e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.092+0000 7f1fd2ffd640 1 -- 172.21.15.31:0/1127185190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80ab600 con 0x7f1fd4070c30 2023-10-30T13:42:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.092+0000 7f1fd2ffd640 1 -- 172.21.15.31:0/1127185190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc80a7050 con 0x7f1fd4070c30 2023-10-30T13:42:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.092+0000 7f1fd2ffd640 1 -- 172.21.15.31:0/1127185190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80ab980 con 0x7f1fd4070c30 2023-10-30T13:42:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- 172.21.15.31:0/1127185190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 msgr2=0x7f1fd4071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 --2- 172.21.15.31:0/1127185190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd4071070 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4072de0 tx=0x7f1fc801c470 comp rx=0 tx=0).stop 2023-10-30T13:42:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- 172.21.15.31:0/1127185190 shutdown_connections 2023-10-30T13:42:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 --2- 172.21.15.31:0/1127185190 >> v2:172.21.15.33:3300/0 conn(0x7f1fd4159970 0x7f1fd4159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 --2- 172.21.15.31:0/1127185190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd4071070 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 --2- 172.21.15.31:0/1127185190 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- 172.21.15.31:0/1127185190 >> 172.21.15.31:0/1127185190 conn(0x7f1fd40c7f60 msgr2=0x7f1fd40c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:28.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- 172.21.15.31:0/1127185190 shutdown_connections 2023-10-30T13:42:28.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- 172.21.15.31:0/1127185190 wait complete. 2023-10-30T13:42:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 Processor -- start 2023-10-30T13:42:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- start start 2023-10-30T13:42:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd41460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1fd4159970 0x7f1fd41f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd414d760 con 0x7f1fd4070c30 2023-10-30T13:42:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd4146770 con 0x7f1fd4159970 2023-10-30T13:42:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fda417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd414cbe0 con 0x7f1fd4153c10 2023-10-30T13:42:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd41460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd41460e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47706/0 (socket says 172.21.15.31:47706) 2023-10-30T13:42:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fd3fff640 1 -- 172.21.15.31:0/419712450 learned_addr learned my addr 172.21.15.31:0/419712450 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fd37fe640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.096+0000 7f1fd89b4640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1fd4159970 0x7f1fd41f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd3fff640 1 -- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 msgr2=0x7f1fd4144490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd3fff640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4144490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd3fff640 1 -- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1fd4159970 msgr2=0x7f1fd41f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd3fff640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1fd4159970 0x7f1fd41f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd3fff640 1 -- 172.21.15.31:0/419712450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80094d0 con 0x7f1fd4070c30 2023-10-30T13:42:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd3fff640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd41460e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400e810 tx=0x7f1fc400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd0ff9640 1 -- 172.21.15.31:0/419712450 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4004030 con 0x7f1fd4070c30 2023-10-30T13:42:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd0ff9640 1 -- 172.21.15.31:0/419712450 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4004190 con 0x7f1fd4070c30 2023-10-30T13:42:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd0ff9640 1 -- 172.21.15.31:0/419712450 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4004520 con 0x7f1fd4070c30 2023-10-30T13:42:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd414cec0 con 0x7f1fd4070c30 2023-10-30T13:42:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd414d480 con 0x7f1fd4070c30 2023-10-30T13:42:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd404e170 con 0x7f1fd4070c30 2023-10-30T13:42:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.100+0000 7f1fd0ff9640 1 -- 172.21.15.31:0/419712450 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1fc400c6d0 con 0x7f1fd4070c30 2023-10-30T13:42:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.104+0000 7f1fd0ff9640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1fc00664a0 0x7f1fc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.104+0000 7f1fd0ff9640 1 -- 172.21.15.31:0/419712450 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f1fc4083c90 con 0x7f1fd4070c30 2023-10-30T13:42:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.104+0000 7f1fd37fe640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1fc00664a0 0x7f1fc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.104+0000 7f1fd37fe640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1fc00664a0 0x7f1fc0068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80ad000 tx=0x7f1fc80a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:28.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.108+0000 7f1fd0ff9640 1 -- 172.21.15.31:0/419712450 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1fc4050f00 con 0x7f1fd4070c30 2023-10-30T13:42:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fd41f6120 con 0x7f1fd4070c30 2023-10-30T13:42:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fd0ff9640 1 -- 172.21.15.31:0/419712450 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f1fc4054520 con 0x7f1fd4070c30 2023-10-30T13:42:28.297 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:28.297 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:24.344953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 42 2023-10-30T13:42:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1fc00664a0 msgr2=0x7f1fc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1fc00664a0 0x7f1fc0068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80ad000 tx=0x7f1fc80a6000 comp rx=0 tx=0).stop 2023-10-30T13:42:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 msgr2=0x7f1fd41460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd41460e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400e810 tx=0x7f1fc400ecd0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 shutdown_connections 2023-10-30T13:42:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1fc00664a0 0x7f1fc0068960 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1fd4159970 0x7f1fd41f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1fd4153c10 0x7f1fd4144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 --2- 172.21.15.31:0/419712450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd4070c30 0x7f1fd41460e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 >> 172.21.15.31:0/419712450 conn(0x7f1fd40c7f60 msgr2=0x7f1fd4150940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 shutdown_connections 2023-10-30T13:42:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:28.296+0000 7f1fda417640 1 -- 172.21.15.31:0/419712450 wait complete. 2023-10-30T13:42:28.318 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:30.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:30.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e3fd3640 1 Processor -- start 2023-10-30T13:42:30.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e3fd3640 1 -- start start 2023-10-30T13:42:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e3fd3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb1dc153c10 0x7fb1dc065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e3fd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e3fd3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc06e390 con 0x7fb1dc159970 2023-10-30T13:42:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e3fd3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc072390 con 0x7fb1dc153c10 2023-10-30T13:42:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e3fd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0660e0 con 0x7fb1dc070c30 2023-10-30T13:42:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e2570640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb1dc159970 0x7fb1dc159e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37300/0 (socket says 172.21.15.31:37300) 2023-10-30T13:42:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e1d6f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb1dc153c10 0x7fb1dc065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.472+0000 7fb1e2570640 1 -- 172.21.15.31:0/1857189546 learned_addr learned my addr 172.21.15.31:0/1857189546 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e156e640 1 --2- 172.21.15.31:0/1857189546 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e1d6f640 1 -- 172.21.15.31:0/1857189546 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 msgr2=0x7fb1dc071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e1d6f640 1 --2- 172.21.15.31:0/1857189546 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e1d6f640 1 -- 172.21.15.31:0/1857189546 >> v1:172.21.15.31:6789/0 conn(0x7fb1dc159970 legacy=0x7fb1dc159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e1d6f640 1 -- 172.21.15.31:0/1857189546 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0715b0 con 0x7fb1dc153c10 2023-10-30T13:42:30.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e1d6f640 1 --2- 172.21.15.31:0/1857189546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc065ba0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80097c0 tx=0x7fb1d801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c303d9935797491 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:30.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e0d6d640 1 -- 172.21.15.31:0/1857189546 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80ab600 con 0x7fb1dc153c10 2023-10-30T13:42:30.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e0d6d640 1 -- 172.21.15.31:0/1857189546 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d80a7050 con 0x7fb1dc153c10 2023-10-30T13:42:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e0d6d640 1 -- 172.21.15.31:0/1857189546 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80ab980 con 0x7fb1dc153c10 2023-10-30T13:42:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/1857189546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 msgr2=0x7fb1dc065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.476+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/1857189546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc065ba0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80097c0 tx=0x7fb1d801c3f0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/1857189546 shutdown_connections 2023-10-30T13:42:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/1857189546 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/1857189546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc065ba0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/1857189546 >> 172.21.15.31:0/1857189546 conn(0x7fb1dc0c7f60 msgr2=0x7fb1dc0c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/1857189546 shutdown_connections 2023-10-30T13:42:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/1857189546 wait complete. 2023-10-30T13:42:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 Processor -- start 2023-10-30T13:42:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- start start 2023-10-30T13:42:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc1f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc159970 0x7fb1dc0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc147590 con 0x7fb1dc153c10 2023-10-30T13:42:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1f64f0 con 0x7fb1dc159970 2023-10-30T13:42:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e3fd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1f2030 con 0x7fb1dc070c30 2023-10-30T13:42:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47724/0 (socket says 172.21.15.31:47724) 2023-10-30T13:42:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 -- 172.21.15.31:0/3659533275 learned_addr learned my addr 172.21.15.31:0/3659533275 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e1d6f640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc1f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e2570640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc159970 0x7fb1dc0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 -- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 msgr2=0x7fb1dc1f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc1f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 -- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc159970 msgr2=0x7fb1dc0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc159970 0x7fb1dc0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 -- 172.21.15.31:0/3659533275 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d8009420 con 0x7fb1dc153c10 2023-10-30T13:42:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.480+0000 7fb1e156e640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc158f60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00d800 tx=0x7fb1cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:30.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1d27fc640 1 -- 172.21.15.31:0/3659533275 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc00b6c0 con 0x7fb1dc153c10 2023-10-30T13:42:30.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1d27fc640 1 -- 172.21.15.31:0/3659533275 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc00b820 con 0x7fb1dc153c10 2023-10-30T13:42:30.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1d27fc640 1 -- 172.21.15.31:0/3659533275 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc00bb90 con 0x7fb1dc153c10 2023-10-30T13:42:30.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc071ad0 con 0x7fb1dc153c10 2023-10-30T13:42:30.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc1513b0 con 0x7fb1dc153c10 2023-10-30T13:42:30.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc1f2ab0 con 0x7fb1dc153c10 2023-10-30T13:42:30.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1d27fc640 1 -- 172.21.15.31:0/3659533275 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1cc00bcf0 con 0x7fb1dc153c10 2023-10-30T13:42:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1d27fc640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1c00664d0 0x7fb1c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1d27fc640 1 -- 172.21.15.31:0/3659533275 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fb1cc084470 con 0x7fb1dc153c10 2023-10-30T13:42:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.484+0000 7fb1e1d6f640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1c00664d0 0x7fb1c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.488+0000 7fb1e1d6f640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1c00664d0 0x7fb1c0068990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80ad000 tx=0x7fb1d80a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.488+0000 7fb1d27fc640 1 -- 172.21.15.31:0/3659533275 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1dc1f2ab0 con 0x7fb1dc153c10 2023-10-30T13:42:30.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.676+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1dc1f2ab0 con 0x7fb1dc153c10 2023-10-30T13:42:30.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1d27fc640 1 -- 172.21.15.31:0/3659533275 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fb1dc1f2ab0 con 0x7fb1dc153c10 2023-10-30T13:42:30.681 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:30.681 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:24.344953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 42 2023-10-30T13:42:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1c00664d0 msgr2=0x7fb1c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1c00664d0 0x7fb1c0068990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80ad000 tx=0x7fb1d80a6000 comp rx=0 tx=0).stop 2023-10-30T13:42:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 msgr2=0x7fb1dc158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc158f60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00d800 tx=0x7fb1cc00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 shutdown_connections 2023-10-30T13:42:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1c00664d0 0x7fb1c0068990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc159970 0x7fb1dc0ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1dc153c10 0x7fb1dc158f60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 --2- 172.21.15.31:0/3659533275 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb1dc070c30 0x7fb1dc1f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 >> 172.21.15.31:0/3659533275 conn(0x7fb1dc0c7f60 msgr2=0x7fb1dc159de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 shutdown_connections 2023-10-30T13:42:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:30.680+0000 7fb1e3fd3640 1 -- 172.21.15.31:0/3659533275 wait complete. 2023-10-30T13:42:30.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:32.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:32.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f813753e640 1 Processor -- start 2023-10-30T13:42:32.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f813753e640 1 -- start start 2023-10-30T13:42:32.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f813753e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 0x7f813015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f813753e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8130159f20 con 0x7f8130153c80 2023-10-30T13:42:32.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f813753e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813015d620 con 0x7f81300c42a0 2023-10-30T13:42:32.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f813753e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813015dec0 con 0x7f81300c4a10 2023-10-30T13:42:32.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f81352da640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8130153c80 0x7f813015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37312/0 (socket says 172.21.15.31:37312) 2023-10-30T13:42:32.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f81352da640 1 -- 172.21.15.31:0/4216261279 learned_addr learned my addr 172.21.15.31:0/4216261279 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:32.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8134ad9640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 0x7f813015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597844091 0 0) 0x7f8130159f20 con 0x7f8130153c80 2023-10-30T13:42:32.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f811c003530 con 0x7f8130153c80 2023-10-30T13:42:32.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276503713 0 0) 0x7f811c003530 con 0x7f8130153c80 2023-10-30T13:42:32.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 msgr2=0x7f813015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:32.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 0x7f813015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 >> v1:172.21.15.33:6789/0 conn(0x7f81300c42a0 legacy=0x7f81300ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813015d900 con 0x7f8130153c80 2023-10-30T13:42:32.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f812c001680 con 0x7f8130153c80 2023-10-30T13:42:32.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 >> v1:172.21.15.31:6789/0 conn(0x7f8130153c80 legacy=0x7f813015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f811c0039b0 0x7f811c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 0x7f811c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f811c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813015dec0 con 0x7f811c004d10 2023-10-30T13:42:32.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c00b990 con 0x7f811c0039b0 2023-10-30T13:42:32.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811c00bb50 con 0x7f81300c4a10 2023-10-30T13:42:32.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8135adb640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f811c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f8134ad9640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 0x7f811c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.856+0000 7f81352da640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f811c0039b0 0x7f811c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8135adb640 1 -- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 msgr2=0x7f811c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:32.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8135adb640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81300c4a10 0x7f811c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8135adb640 1 -- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f811c0039b0 msgr2=0x7f811c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:32.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8135adb640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f811c0039b0 0x7f811c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8135adb640 1 -- 172.21.15.31:0/4216261279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c003530 con 0x7f811c004d10 2023-10-30T13:42:32.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8135adb640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f811c005190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8128002010 tx=0x7f812801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2302daf514df5108 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:32.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81280a57b0 con 0x7f811c004d10 2023-10-30T13:42:32.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81280a5930 con 0x7f811c004d10 2023-10-30T13:42:32.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8127fff640 1 -- 172.21.15.31:0/4216261279 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81280a5ce0 con 0x7f811c004d10 2023-10-30T13:42:32.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f813753e640 1 -- 172.21.15.31:0/4216261279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 msgr2=0x7f811c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f813753e640 1 --2- 172.21.15.31:0/4216261279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f811c005190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8128002010 tx=0x7f812801c8c0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f8135adb640 1 -- 172.21.15.31:0/4216261279 reap_dead start 2023-10-30T13:42:32.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f813753e640 1 -- 172.21.15.31:0/4216261279 shutdown_connections 2023-10-30T13:42:32.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f813753e640 1 -- 172.21.15.31:0/4216261279 >> 172.21.15.31:0/4216261279 conn(0x7f81300715c0 msgr2=0x7f81300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:32.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.860+0000 7f813753e640 1 -- 172.21.15.31:0/4216261279 shutdown_connections 2023-10-30T13:42:32.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 -- 172.21.15.31:0/4216261279 wait complete. 2023-10-30T13:42:32.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 Processor -- start 2023-10-30T13:42:32.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 -- start start 2023-10-30T13:42:32.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f813014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81301584e0 0x7f813014eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81300c9290 0x7f81300c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8130159420 con 0x7f811c004d10 2023-10-30T13:42:32.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813014e620 con 0x7f81300c9290 2023-10-30T13:42:32.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8130159910 con 0x7f81301584e0 2023-10-30T13:42:32.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f813014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f813014eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47732/0 (socket says 172.21.15.31:47732) 2023-10-30T13:42:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 -- 172.21.15.31:0/2479949755 learned_addr learned my addr 172.21.15.31:0/2479949755 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f8135adb640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81300c9290 0x7f81300c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f8134ad9640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81301584e0 0x7f813014eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 -- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81301584e0 msgr2=0x7f813014eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81301584e0 0x7f813014eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 -- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81300c9290 msgr2=0x7f81300c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81300c9290 0x7f81300c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 -- 172.21.15.31:0/2479949755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8130159630 con 0x7f811c004d10 2023-10-30T13:42:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f81352da640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f813014eab0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f812c00e850 tx=0x7f812c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f8125ffb640 1 -- 172.21.15.31:0/2479949755 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00c8d0 con 0x7f811c004d10 2023-10-30T13:42:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f8125ffb640 1 -- 172.21.15.31:0/2479949755 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c00ca30 con 0x7f811c004d10 2023-10-30T13:42:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813014dfd0 con 0x7f811c004d10 2023-10-30T13:42:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.864+0000 7f8125ffb640 1 -- 172.21.15.31:0/2479949755 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00cde0 con 0x7f811c004d10 2023-10-30T13:42:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.868+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81300cb960 con 0x7f811c004d10 2023-10-30T13:42:32.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.868+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813004e170 con 0x7f811c004d10 2023-10-30T13:42:32.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.868+0000 7f8125ffb640 1 -- 172.21.15.31:0/2479949755 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f812c01d460 con 0x7f811c004d10 2023-10-30T13:42:32.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.868+0000 7f8125ffb640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81140664d0 0x7f8114068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.868+0000 7f8125ffb640 1 -- 172.21.15.31:0/2479949755 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f812c084160 con 0x7f811c004d10 2023-10-30T13:42:32.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.868+0000 7f8134ad9640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81140664d0 0x7f8114068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.868+0000 7f8134ad9640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81140664d0 0x7f8114068990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f81301f3530 tx=0x7f8120008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:32.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:32.872+0000 7f8125ffb640 1 -- 172.21.15.31:0/2479949755 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f812c0513c0 con 0x7f811c004d10 2023-10-30T13:42:33.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81300cad40 con 0x7f811c004d10 2023-10-30T13:42:33.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f8125ffb640 1 -- 172.21.15.31:0/2479949755 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f812c0549e0 con 0x7f811c004d10 2023-10-30T13:42:33.066 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:33.066 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:24.344953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:33.067 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 42 2023-10-30T13:42:33.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81140664d0 msgr2=0x7f8114068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81140664d0 0x7f8114068990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f81301f3530 tx=0x7f8120008040 comp rx=0 tx=0).stop 2023-10-30T13:42:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 msgr2=0x7f813014eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f813014eab0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f812c00e850 tx=0x7f812c00ed10 comp rx=0 tx=0).stop 2023-10-30T13:42:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 shutdown_connections 2023-10-30T13:42:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f81140664d0 0x7f8114068990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81300c9290 0x7f81300c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f81301584e0 0x7f813014eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 --2- 172.21.15.31:0/2479949755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811c004d10 0x7f813014eab0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 >> 172.21.15.31:0/2479949755 conn(0x7f81300715c0 msgr2=0x7f81300cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.064+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 shutdown_connections 2023-10-30T13:42:33.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:33.068+0000 7f813753e640 1 -- 172.21.15.31:0/2479949755 wait complete. 2023-10-30T13:42:33.087 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:33.088 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4739": { "addr": "172.21.15.33:6836/3073449389", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3073449389, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3073449389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4739, "incarnation": 22, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 41 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:24.344953+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082, "mds_2": 4739 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:35.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:35.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.244+0000 7f833d8f7640 1 Processor -- start 2023-10-30T13:42:35.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.244+0000 7f833d8f7640 1 -- start start 2023-10-30T13:42:35.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.244+0000 7f833d8f7640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c4a10 0x7f833815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:35.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.244+0000 7f833d8f7640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f83380c42a0 0x7f83380ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:35.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f833d8f7640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8338159f20 con 0x7f8338153c80 2023-10-30T13:42:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f833d8f7640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833815dec0 con 0x7f83380c42a0 2023-10-30T13:42:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f833d8f7640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83381599a0 con 0x7f83380c4a10 2023-10-30T13:42:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f83380c42a0 0x7f83380ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f83380c42a0 0x7f83380ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44950/0 (socket says 172.21.15.31:44950) 2023-10-30T13:42:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 -- 172.21.15.31:0/3441907134 learned_addr learned my addr 172.21.15.31:0/3441907134 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f832ffff640 1 --2- 172.21.15.31:0/3441907134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c4a10 0x7f833815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 -- 172.21.15.31:0/3441907134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c4a10 msgr2=0x7f833815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 --2- 172.21.15.31:0/3441907134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c4a10 0x7f833815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 -- 172.21.15.31:0/3441907134 >> v1:172.21.15.33:6789/0 conn(0x7f8338153c80 legacy=0x7f833815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 -- 172.21.15.31:0/3441907134 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833815d620 con 0x7f83380c42a0 2023-10-30T13:42:35.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.248+0000 7f83377fe640 1 --2- 172.21.15.31:0/3441907134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c42a0 0x7f83380ca1e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8330009e50 tx=0x7f833001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1153f53178bd883c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:35.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f83367fc640 1 -- 172.21.15.31:0/3441907134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83300af400 con 0x7f83380c42a0 2023-10-30T13:42:35.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f83367fc640 1 -- 172.21.15.31:0/3441907134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83300ab050 con 0x7f83380c42a0 2023-10-30T13:42:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f83367fc640 1 -- 172.21.15.31:0/3441907134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83300af780 con 0x7f83380c42a0 2023-10-30T13:42:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- 172.21.15.31:0/3441907134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c42a0 msgr2=0x7f83380ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3441907134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c42a0 0x7f83380ca1e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8330009e50 tx=0x7f833001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- 172.21.15.31:0/3441907134 shutdown_connections 2023-10-30T13:42:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3441907134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c42a0 0x7f83380ca1e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3441907134 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c4a10 0x7f833815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- 172.21.15.31:0/3441907134 >> 172.21.15.31:0/3441907134 conn(0x7f83380715c0 msgr2=0x7f83380719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- 172.21.15.31:0/3441907134 shutdown_connections 2023-10-30T13:42:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- 172.21.15.31:0/3441907134 wait complete. 2023-10-30T13:42:35.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 Processor -- start 2023-10-30T13:42:35.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- start start 2023-10-30T13:42:35.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c42a0 0x7f83381f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:35.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c4a10 0x7f83380b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:35.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8338153c80 0x7f833814f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:35.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380b2610 con 0x7f83380c4a10 2023-10-30T13:42:35.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380b3a60 con 0x7f8338153c80 2023-10-30T13:42:35.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f833d8f7640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83381f0630 con 0x7f83380c42a0 2023-10-30T13:42:35.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f832ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c4a10 0x7f83380b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:35.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f832ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c4a10 0x7f83380b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44952/0 (socket says 172.21.15.31:44952) 2023-10-30T13:42:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f832ffff640 1 -- 172.21.15.31:0/3537852809 learned_addr learned my addr 172.21.15.31:0/3537852809 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f83377fe640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8338153c80 0x7f833814f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.252+0000 7f8336ffd640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c42a0 0x7f83381f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f832ffff640 1 -- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c42a0 msgr2=0x7f83381f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f832ffff640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c42a0 0x7f83381f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f832ffff640 1 -- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8338153c80 msgr2=0x7f833814f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:35.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f832ffff640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8338153c80 0x7f833814f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f832ffff640 1 -- 172.21.15.31:0/3537852809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330009b30 con 0x7f83380c4a10 2023-10-30T13:42:35.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f832ffff640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c4a10 0x7f83380b3080 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f832000e960 tx=0x7f832000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f8334ff9640 1 -- 172.21.15.31:0/3537852809 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320009690 con 0x7f83380c4a10 2023-10-30T13:42:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f8334ff9640 1 -- 172.21.15.31:0/3537852809 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832000ca00 con 0x7f83380c4a10 2023-10-30T13:42:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83380b3cd0 con 0x7f83380c4a10 2023-10-30T13:42:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f8334ff9640 1 -- 172.21.15.31:0/3537852809 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832000cdb0 con 0x7f83380c4a10 2023-10-30T13:42:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83380b4570 con 0x7f83380c4a10 2023-10-30T13:42:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833804e170 con 0x7f83380c4a10 2023-10-30T13:42:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f8334ff9640 1 -- 172.21.15.31:0/3537852809 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8320013470 con 0x7f83380c4a10 2023-10-30T13:42:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f8334ff9640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83180664a0 0x7f8318068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.256+0000 7f8334ff9640 1 -- 172.21.15.31:0/3537852809 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f8320084280 con 0x7f83380c4a10 2023-10-30T13:42:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.260+0000 7f8336ffd640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83180664a0 0x7f8318068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.260+0000 7f8336ffd640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83180664a0 0x7f8318068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8328000f70 tx=0x7f8328009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.260+0000 7f8334ff9640 1 -- 172.21.15.31:0/3537852809 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8320051580 con 0x7f83380c4a10 2023-10-30T13:42:35.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f833814e460 con 0x7f83380c4a10 2023-10-30T13:42:35.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f8334ff9640 1 -- 172.21.15.31:0/3537852809 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f8320054ba0 con 0x7f83380c4a10 2023-10-30T13:42:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:35.466 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:24.344953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5082,"mds_2":4739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4739":{"gid":4739,"name":"d","rank":2,"incarnation":22,"state":"up:stopping","state_seq":41,"addr":"172.21.15.33:6836/3073449389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3073449389},{"type":"v1","addr":"172.21.15.33:6836","nonce":3073449389}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:35.467 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 42 2023-10-30T13:42:35.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83180664a0 msgr2=0x7f8318068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:35.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83180664a0 0x7f8318068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8328000f70 tx=0x7f8328009040 comp rx=0 tx=0).stop 2023-10-30T13:42:35.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c4a10 msgr2=0x7f83380b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:35.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c4a10 0x7f83380b3080 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f832000e960 tx=0x7f832000ee20 comp rx=0 tx=0).stop 2023-10-30T13:42:35.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 shutdown_connections 2023-10-30T13:42:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83180664a0 0x7f8318068960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8338153c80 0x7f833814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83380c4a10 0x7f83380b3080 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 --2- 172.21.15.31:0/3537852809 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83380c42a0 0x7f83381f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:35.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 >> 172.21.15.31:0/3537852809 conn(0x7f83380715c0 msgr2=0x7f83380719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:35.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.464+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 shutdown_connections 2023-10-30T13:42:35.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:35.468+0000 7f833d8f7640 1 -- 172.21.15.31:0/3537852809 wait complete. 2023-10-30T13:42:35.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:37.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/34' 2023-10-30T13:42:37.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/38' 2023-10-30T13:42:37.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/28' 2023-10-30T13:42:37.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/24' 2023-10-30T13:42:37.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/29' 2023-10-30T13:42:37.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/32' 2023-10-30T13:42:37.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/22' 2023-10-30T13:42:37.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/25' 2023-10-30T13:42:37.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/30' 2023-10-30T13:42:37.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/33' 2023-10-30T13:42:37.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/23' 2023-10-30T13:42:37.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/35' 2023-10-30T13:42:37.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/26' 2023-10-30T13:42:37.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './206/37' 2023-10-30T13:42:37.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './206' 2023-10-30T13:42:37.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/21' 2023-10-30T13:42:37.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/27' 2023-10-30T13:42:37.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/31' 2023-10-30T13:42:37.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/36' 2023-10-30T13:42:37.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/20' 2023-10-30T13:42:37.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/39' 2023-10-30T13:42:37.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/34' 2023-10-30T13:42:37.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/38' 2023-10-30T13:42:37.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/28' 2023-10-30T13:42:37.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/24' 2023-10-30T13:42:37.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/29' 2023-10-30T13:42:37.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/32' 2023-10-30T13:42:37.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/22' 2023-10-30T13:42:37.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/25' 2023-10-30T13:42:37.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/30' 2023-10-30T13:42:37.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/33' 2023-10-30T13:42:37.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/23' 2023-10-30T13:42:37.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/35' 2023-10-30T13:42:37.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/26' 2023-10-30T13:42:37.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './207/37' 2023-10-30T13:42:37.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './207' 2023-10-30T13:42:37.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/21' 2023-10-30T13:42:37.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/27' 2023-10-30T13:42:37.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/31' 2023-10-30T13:42:37.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/36' 2023-10-30T13:42:37.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/20' 2023-10-30T13:42:37.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/39' 2023-10-30T13:42:37.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/34' 2023-10-30T13:42:37.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/38' 2023-10-30T13:42:37.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/28' 2023-10-30T13:42:37.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/24' 2023-10-30T13:42:37.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/29' 2023-10-30T13:42:37.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/32' 2023-10-30T13:42:37.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/22' 2023-10-30T13:42:37.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/25' 2023-10-30T13:42:37.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/30' 2023-10-30T13:42:37.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/33' 2023-10-30T13:42:37.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/23' 2023-10-30T13:42:37.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/35' 2023-10-30T13:42:37.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/26' 2023-10-30T13:42:37.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './208/37' 2023-10-30T13:42:37.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './208' 2023-10-30T13:42:37.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/21' 2023-10-30T13:42:37.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/27' 2023-10-30T13:42:37.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/31' 2023-10-30T13:42:37.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/36' 2023-10-30T13:42:37.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/20' 2023-10-30T13:42:37.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/39' 2023-10-30T13:42:37.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/34' 2023-10-30T13:42:37.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/38' 2023-10-30T13:42:37.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/28' 2023-10-30T13:42:37.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/24' 2023-10-30T13:42:37.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/29' 2023-10-30T13:42:37.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/32' 2023-10-30T13:42:37.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/22' 2023-10-30T13:42:37.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/25' 2023-10-30T13:42:37.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/30' 2023-10-30T13:42:37.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/33' 2023-10-30T13:42:37.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/23' 2023-10-30T13:42:37.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/35' 2023-10-30T13:42:37.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/26' 2023-10-30T13:42:37.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './209/37' 2023-10-30T13:42:37.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './209' 2023-10-30T13:42:37.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/21' 2023-10-30T13:42:37.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/27' 2023-10-30T13:42:37.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/31' 2023-10-30T13:42:37.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/36' 2023-10-30T13:42:37.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/20' 2023-10-30T13:42:37.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/39' 2023-10-30T13:42:37.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/34' 2023-10-30T13:42:37.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/38' 2023-10-30T13:42:37.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/28' 2023-10-30T13:42:37.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/24' 2023-10-30T13:42:37.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/29' 2023-10-30T13:42:37.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/32' 2023-10-30T13:42:37.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/22' 2023-10-30T13:42:37.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/25' 2023-10-30T13:42:37.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/30' 2023-10-30T13:42:37.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/33' 2023-10-30T13:42:37.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/23' 2023-10-30T13:42:37.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/35' 2023-10-30T13:42:37.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/26' 2023-10-30T13:42:37.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './210/37' 2023-10-30T13:42:37.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './210' 2023-10-30T13:42:37.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/21' 2023-10-30T13:42:37.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/27' 2023-10-30T13:42:37.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/31' 2023-10-30T13:42:37.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/36' 2023-10-30T13:42:37.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/20' 2023-10-30T13:42:37.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/39' 2023-10-30T13:42:37.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/34' 2023-10-30T13:42:37.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/38' 2023-10-30T13:42:37.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/28' 2023-10-30T13:42:37.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/24' 2023-10-30T13:42:37.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/29' 2023-10-30T13:42:37.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/32' 2023-10-30T13:42:37.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/22' 2023-10-30T13:42:37.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/25' 2023-10-30T13:42:37.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/30' 2023-10-30T13:42:37.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/33' 2023-10-30T13:42:37.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/23' 2023-10-30T13:42:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/35' 2023-10-30T13:42:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/26' 2023-10-30T13:42:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './211/37' 2023-10-30T13:42:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './211' 2023-10-30T13:42:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/21' 2023-10-30T13:42:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/27' 2023-10-30T13:42:37.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/31' 2023-10-30T13:42:37.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/36' 2023-10-30T13:42:37.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/20' 2023-10-30T13:42:37.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/39' 2023-10-30T13:42:37.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/34' 2023-10-30T13:42:37.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/38' 2023-10-30T13:42:37.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/28' 2023-10-30T13:42:37.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/24' 2023-10-30T13:42:37.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/29' 2023-10-30T13:42:37.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/32' 2023-10-30T13:42:37.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/22' 2023-10-30T13:42:37.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/25' 2023-10-30T13:42:37.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/30' 2023-10-30T13:42:37.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/33' 2023-10-30T13:42:37.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/23' 2023-10-30T13:42:37.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/35' 2023-10-30T13:42:37.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/26' 2023-10-30T13:42:37.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './212/37' 2023-10-30T13:42:37.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './212' 2023-10-30T13:42:37.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/21' 2023-10-30T13:42:37.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/27' 2023-10-30T13:42:37.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/31' 2023-10-30T13:42:37.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/36' 2023-10-30T13:42:37.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/20' 2023-10-30T13:42:37.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/39' 2023-10-30T13:42:37.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/34' 2023-10-30T13:42:37.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/38' 2023-10-30T13:42:37.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/28' 2023-10-30T13:42:37.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/24' 2023-10-30T13:42:37.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/29' 2023-10-30T13:42:37.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/32' 2023-10-30T13:42:37.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/22' 2023-10-30T13:42:37.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/25' 2023-10-30T13:42:37.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/30' 2023-10-30T13:42:37.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/33' 2023-10-30T13:42:37.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/23' 2023-10-30T13:42:37.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/35' 2023-10-30T13:42:37.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/26' 2023-10-30T13:42:37.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './213/37' 2023-10-30T13:42:37.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './213' 2023-10-30T13:42:37.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/21' 2023-10-30T13:42:37.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/27' 2023-10-30T13:42:37.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/31' 2023-10-30T13:42:37.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/36' 2023-10-30T13:42:37.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/20' 2023-10-30T13:42:37.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/39' 2023-10-30T13:42:37.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/34' 2023-10-30T13:42:37.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/38' 2023-10-30T13:42:37.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/28' 2023-10-30T13:42:37.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/24' 2023-10-30T13:42:37.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/29' 2023-10-30T13:42:37.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/32' 2023-10-30T13:42:37.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/22' 2023-10-30T13:42:37.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/25' 2023-10-30T13:42:37.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/30' 2023-10-30T13:42:37.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/33' 2023-10-30T13:42:37.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/23' 2023-10-30T13:42:37.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/35' 2023-10-30T13:42:37.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/26' 2023-10-30T13:42:37.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './214/37' 2023-10-30T13:42:37.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './214' 2023-10-30T13:42:37.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/21' 2023-10-30T13:42:37.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/27' 2023-10-30T13:42:37.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/31' 2023-10-30T13:42:37.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/36' 2023-10-30T13:42:37.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/20' 2023-10-30T13:42:37.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/39' 2023-10-30T13:42:37.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/34' 2023-10-30T13:42:37.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/38' 2023-10-30T13:42:37.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/28' 2023-10-30T13:42:37.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/24' 2023-10-30T13:42:37.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/29' 2023-10-30T13:42:37.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/32' 2023-10-30T13:42:37.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/22' 2023-10-30T13:42:37.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/25' 2023-10-30T13:42:37.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/30' 2023-10-30T13:42:37.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/33' 2023-10-30T13:42:37.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/23' 2023-10-30T13:42:37.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/35' 2023-10-30T13:42:37.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/26' 2023-10-30T13:42:37.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './215/37' 2023-10-30T13:42:37.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './215' 2023-10-30T13:42:37.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/21' 2023-10-30T13:42:37.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/27' 2023-10-30T13:42:37.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/31' 2023-10-30T13:42:37.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/36' 2023-10-30T13:42:37.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/20' 2023-10-30T13:42:37.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/39' 2023-10-30T13:42:37.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/34' 2023-10-30T13:42:37.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/38' 2023-10-30T13:42:37.228 INFO:tasks.ceph.mds.d.smithi033.stdout:starting mds.d at 2023-10-30T13:42:37.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.640+0000 7ff96f246640 1 Processor -- start 2023-10-30T13:42:37.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.640+0000 7ff96f246640 1 -- start start 2023-10-30T13:42:37.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.640+0000 7ff96f246640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff96806a0f0 0x7ff968153830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:37.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96f246640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff96806a4d0 0x7ff968153d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:37.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96f246640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9680c1e60 con 0x7ff9680c0260 2023-10-30T13:42:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96f246640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff96815a3f0 con 0x7ff96806a0f0 2023-10-30T13:42:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96f246640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9680cb2b0 con 0x7ff96806a4d0 2023-10-30T13:42:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96cfe2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff96806a0f0 0x7ff968153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96cfe2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff96806a0f0 0x7ff968153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44966/0 (socket says 172.21.15.31:44966) 2023-10-30T13:42:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96d7e3640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff9680c0260 0x7ff9680cb790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45230/0 (socket says 172.21.15.31:45230) 2023-10-30T13:42:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2932137961 learned_addr learned my addr 172.21.15.31:0/2932137961 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:37.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2932137961 >> v2:172.21.15.33:3300/0 conn(0x7ff96806a4d0 msgr2=0x7ff968153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:37.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff967fff640 1 --2- 172.21.15.31:0/2932137961 >> v2:172.21.15.33:3300/0 conn(0x7ff96806a4d0 0x7ff968153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff9677fe640 1 -- 172.21.15.31:0/2932137961 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546779199 0 0) 0x7ff9680c1e60 con 0x7ff9680c0260 2023-10-30T13:42:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96cfe2640 1 --2- 172.21.15.31:0/2932137961 >> v2:172.21.15.33:3300/0 conn(0x7ff96806a4d0 0x7ff968153d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2932137961 >> v1:172.21.15.31:6789/0 conn(0x7ff9680c0260 legacy=0x7ff9680cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.644+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2932137961 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96815a8c0 con 0x7ff96806a0f0 2023-10-30T13:42:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96cfe2640 1 --2- 172.21.15.31:0/2932137961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff968153830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff96806acc0 tx=0x7ff95c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd57bd324130a992 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff9677fe640 1 -- 172.21.15.31:0/2932137961 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0aa430 con 0x7ff96806a0f0 2023-10-30T13:42:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff9677fe640 1 -- 172.21.15.31:0/2932137961 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c0aa590 con 0x7ff96806a0f0 2023-10-30T13:42:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff9677fe640 1 -- 172.21.15.31:0/2932137961 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0aa920 con 0x7ff96806a0f0 2023-10-30T13:42:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- 172.21.15.31:0/2932137961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 msgr2=0x7ff968153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 --2- 172.21.15.31:0/2932137961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff968153830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff96806acc0 tx=0x7ff95c01c4d0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- 172.21.15.31:0/2932137961 shutdown_connections 2023-10-30T13:42:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 --2- 172.21.15.31:0/2932137961 >> v2:172.21.15.33:3300/0 conn(0x7ff96806a4d0 0x7ff968153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 --2- 172.21.15.31:0/2932137961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff968153830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- 172.21.15.31:0/2932137961 >> 172.21.15.31:0/2932137961 conn(0x7ff96806de90 msgr2=0x7ff96806e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- 172.21.15.31:0/2932137961 shutdown_connections 2023-10-30T13:42:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- 172.21.15.31:0/2932137961 wait complete. 2023-10-30T13:42:37.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 Processor -- start 2023-10-30T13:42:37.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- start start 2023-10-30T13:42:37.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff9680ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:37.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff96806a4d0 0x7ff9680caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:37.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9680c0260 0x7ff96814d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:37.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9680b0660 con 0x7ff96806a0f0 2023-10-30T13:42:37.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9680af7d0 con 0x7ff9680c0260 2023-10-30T13:42:37.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96f246640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9680af9e0 con 0x7ff96806a4d0 2023-10-30T13:42:37.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.648+0000 7ff96cfe2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff9680ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:37.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff9680ae780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44978/0 (socket says 172.21.15.31:44978) 2023-10-30T13:42:37.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2140434213 learned_addr learned my addr 172.21.15.31:0/2140434213 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:37.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96d7e3640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9680c0260 0x7ff96814d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:37.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff967fff640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff96806a4d0 0x7ff9680caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:37.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff96806a4d0 msgr2=0x7ff9680caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff96806a4d0 0x7ff9680caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9680c0260 msgr2=0x7ff96814d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9680c0260 0x7ff96814d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 -- 172.21.15.31:0/2140434213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c0094d0 con 0x7ff96806a0f0 2023-10-30T13:42:37.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96cfe2640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff9680ae780 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0ac040 tx=0x7ff95c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:37.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff9657fa640 1 -- 172.21.15.31:0/2140434213 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0b38c0 con 0x7ff96806a0f0 2023-10-30T13:42:37.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff9657fa640 1 -- 172.21.15.31:0/2140434213 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c0b3a40 con 0x7ff96806a0f0 2023-10-30T13:42:37.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff9657fa640 1 -- 172.21.15.31:0/2140434213 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0b3df0 con 0x7ff96806a0f0 2023-10-30T13:42:37.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9680afcc0 con 0x7ff96806a0f0 2023-10-30T13:42:37.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9680b0280 con 0x7ff96806a0f0 2023-10-30T13:42:37.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96804e170 con 0x7ff96806a0f0 2023-10-30T13:42:37.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff9657fa640 1 -- 172.21.15.31:0/2140434213 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff95c0b2380 con 0x7ff96806a0f0 2023-10-30T13:42:37.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.652+0000 7ff9657fa640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff94c0664d0 0x7ff94c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:37.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.656+0000 7ff9657fa640 1 -- 172.21.15.31:0/2140434213 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ff95c11e410 con 0x7ff96806a0f0 2023-10-30T13:42:37.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.656+0000 7ff967fff640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff94c0664d0 0x7ff94c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:37.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.656+0000 7ff967fff640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff94c0664d0 0x7ff94c068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff9681f2dd0 tx=0x7ff954008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:37.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.660+0000 7ff9657fa640 1 -- 172.21.15.31:0/2140434213 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff95c0eb7c0 con 0x7ff96806a0f0 2023-10-30T13:42:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.844+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff968164100 con 0x7ff96806a0f0 2023-10-30T13:42:37.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff9657fa640 1 -- 172.21.15.31:0/2140434213 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5721 (secure 0 0 0) 0x7ff95c0eede0 con 0x7ff96806a0f0 2023-10-30T13:42:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:37.849 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:37.849 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 44 2023-10-30T13:42:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff94c0664d0 msgr2=0x7ff94c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff94c0664d0 0x7ff94c068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff9681f2dd0 tx=0x7ff954008040 comp rx=0 tx=0).stop 2023-10-30T13:42:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 msgr2=0x7ff9680ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff9680ae780 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0ac040 tx=0x7ff95c01ad10 comp rx=0 tx=0).stop 2023-10-30T13:42:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 shutdown_connections 2023-10-30T13:42:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff94c0664d0 0x7ff94c068990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9680c0260 0x7ff96814d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff96806a4d0 0x7ff9680caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 --2- 172.21.15.31:0/2140434213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff96806a0f0 0x7ff9680ae780 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 >> 172.21.15.31:0/2140434213 conn(0x7ff96806de90 msgr2=0x7ff968071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 shutdown_connections 2023-10-30T13:42:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:37.848+0000 7ff96f246640 1 -- 172.21.15.31:0/2140434213 wait complete. 2023-10-30T13:42:37.870 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:39.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:40.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f760a965640 1 Processor -- start 2023-10-30T13:42:40.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f760a965640 1 -- start start 2023-10-30T13:42:40.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f760a965640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f760415cf90 0x7f76040c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f760a965640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7604159f20 con 0x7f76040c4a10 2023-10-30T13:42:40.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f760a965640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040c47e0 con 0x7f7604153c80 2023-10-30T13:42:40.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f760a965640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760415dec0 con 0x7f760415cf90 2023-10-30T13:42:40.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f76037fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f76040c4a10 0x7f76040ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45244/0 (socket says 172.21.15.31:45244) 2023-10-30T13:42:40.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f76037fe640 1 -- 172.21.15.31:0/2038422430 learned_addr learned my addr 172.21.15.31:0/2038422430 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:40.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7608f02640 1 --2- 172.21.15.31:0/2038422430 >> v2:172.21.15.33:3300/0 conn(0x7f760415cf90 0x7f76040c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583225674 0 0) 0x7f7604159f20 con 0x7f76040c4a10 2023-10-30T13:42:40.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ec003530 con 0x7f76040c4a10 2023-10-30T13:42:40.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375230383 0 0) 0x7f75ec003530 con 0x7f76040c4a10 2023-10-30T13:42:40.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 >> v2:172.21.15.33:3300/0 conn(0x7f760415cf90 msgr2=0x7f76040c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:40.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 --2- 172.21.15.31:0/2038422430 >> v2:172.21.15.33:3300/0 conn(0x7f760415cf90 0x7f76040c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 >> v1:172.21.15.33:6789/0 conn(0x7f7604153c80 legacy=0x7f760415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:40.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760415d620 con 0x7f76040c4a10 2023-10-30T13:42:40.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7608f02640 1 --2- 172.21.15.31:0/2038422430 >> v2:172.21.15.33:3300/0 conn(0x7f760415cf90 0x7f76040c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:40.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f75f8001680 con 0x7f76040c4a10 2023-10-30T13:42:40.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 >> v1:172.21.15.31:6789/0 conn(0x7f76040c4a10 legacy=0x7f76040ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:40.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75ec003980 0x7f75ec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75ec0047d0 0x7f75ec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec005190 0x7f75ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760415dec0 con 0x7f75ec003980 2023-10-30T13:42:40.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec00c040 con 0x7f75ec005190 2023-10-30T13:42:40.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec00c200 con 0x7f75ec0047d0 2023-10-30T13:42:40.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7603fff640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75ec003980 0x7f75ec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f76037fe640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75ec0047d0 0x7f75ec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7608f02640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec005190 0x7f75ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7603fff640 1 -- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75ec0047d0 msgr2=0x7f75ec004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7603fff640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75ec0047d0 0x7f75ec004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7603fff640 1 -- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec005190 msgr2=0x7f75ec00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7603fff640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec005190 0x7f75ec00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7603fff640 1 -- 172.21.15.31:0/2038422430 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec003530 con 0x7f75ec003980 2023-10-30T13:42:40.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.028+0000 7f7608f02640 1 -- 172.21.15.31:0/2038422430 reap_dead start 2023-10-30T13:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f7603fff640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75ec003980 0x7f75ec003da0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75f4001780 tx=0x7f75f401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385dbba842238ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40a4ea0 con 0x7f75ec003980 2023-10-30T13:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f40a7050 con 0x7f75ec003980 2023-10-30T13:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f7602ffd640 1 -- 172.21.15.31:0/2038422430 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4003600 con 0x7f75ec003980 2023-10-30T13:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f760a965640 1 -- 172.21.15.31:0/2038422430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75ec003980 msgr2=0x7f75ec003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f760a965640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75ec003980 0x7f75ec003da0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75f4001780 tx=0x7f75f401c3e0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f760a965640 1 -- 172.21.15.31:0/2038422430 shutdown_connections 2023-10-30T13:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f760a965640 1 --2- 172.21.15.31:0/2038422430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75ec003980 0x7f75ec003da0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f760a965640 1 -- 172.21.15.31:0/2038422430 >> 172.21.15.31:0/2038422430 conn(0x7f76040715c0 msgr2=0x7f76040719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f760a965640 1 -- 172.21.15.31:0/2038422430 shutdown_connections 2023-10-30T13:42:40.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.032+0000 7f760a965640 1 -- 172.21.15.31:0/2038422430 wait complete. 2023-10-30T13:42:40.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 Processor -- start 2023-10-30T13:42:40.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 -- start start 2023-10-30T13:42:40.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec003980 0x7f7604158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76040c9270 0x7f76041553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76041f03b0 0x7f7604156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604158c80 con 0x7f76041f03b0 2023-10-30T13:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604155df0 con 0x7f75ec003980 2023-10-30T13:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76041599a0 con 0x7f76040c9270 2023-10-30T13:42:40.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76041f03b0 0x7f7604156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76041f03b0 0x7f7604156330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44996/0 (socket says 172.21.15.31:44996) 2023-10-30T13:42:40.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 -- 172.21.15.31:0/1504302225 learned_addr learned my addr 172.21.15.31:0/1504302225 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:40.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f76037fe640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76040c9270 0x7f76041553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7603fff640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec003980 0x7f7604158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 -- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76040c9270 msgr2=0x7f76041553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:40.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76040c9270 0x7f76041553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 -- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec003980 msgr2=0x7f7604158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:40.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec003980 0x7f7604158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 -- 172.21.15.31:0/1504302225 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76041596c0 con 0x7f76041f03b0 2023-10-30T13:42:40.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7608f02640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76041f03b0 0x7f7604156330 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f76040c97d0 tx=0x7f75fc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:40.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7600ff9640 1 -- 172.21.15.31:0/1504302225 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc00e070 con 0x7f76041f03b0 2023-10-30T13:42:40.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7600ff9640 1 -- 172.21.15.31:0/1504302225 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc0039a0 con 0x7f76041f03b0 2023-10-30T13:42:40.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f7600ff9640 1 -- 172.21.15.31:0/1504302225 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc003b30 con 0x7f76041f03b0 2023-10-30T13:42:40.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.036+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76041543b0 con 0x7f76041f03b0 2023-10-30T13:42:40.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.040+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7604154970 con 0x7f76041f03b0 2023-10-30T13:42:40.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.040+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760404e170 con 0x7f76041f03b0 2023-10-30T13:42:40.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.040+0000 7f7600ff9640 1 -- 172.21.15.31:0/1504302225 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75fc003c90 con 0x7f76041f03b0 2023-10-30T13:42:40.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.040+0000 7f7600ff9640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75f00664d0 0x7f75f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:40.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.040+0000 7f7600ff9640 1 -- 172.21.15.31:0/1504302225 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f75fc084750 con 0x7f76041f03b0 2023-10-30T13:42:40.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.040+0000 7f7603fff640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75f00664d0 0x7f75f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:40.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.044+0000 7f7603fff640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75f00664d0 0x7f75f0068990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75f4001750 tx=0x7f75f40011a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:40.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.044+0000 7f7600ff9640 1 -- 172.21.15.31:0/1504302225 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75fc051a70 con 0x7f76041f03b0 2023-10-30T13:42:40.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76040cad40 con 0x7f76041f03b0 2023-10-30T13:42:40.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f7600ff9640 1 -- 172.21.15.31:0/1504302225 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f75fc055090 con 0x7f76041f03b0 2023-10-30T13:42:40.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:40.238 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:40.238 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:40.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75f00664d0 msgr2=0x7f75f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75f00664d0 0x7f75f0068990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75f4001750 tx=0x7f75f40011a0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76041f03b0 msgr2=0x7f7604156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76041f03b0 0x7f7604156330 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f76040c97d0 tx=0x7f75fc0115a0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 shutdown_connections 2023-10-30T13:42:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75f00664d0 0x7f75f0068990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76041f03b0 0x7f7604156330 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f76040c9270 0x7f76041553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 --2- 172.21.15.31:0/1504302225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec003980 0x7f7604158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 >> 172.21.15.31:0/1504302225 conn(0x7f76040715c0 msgr2=0x7f76040cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 shutdown_connections 2023-10-30T13:42:40.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:40.236+0000 7f760a965640 1 -- 172.21.15.31:0/1504302225 wait complete. 2023-10-30T13:42:40.259 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:42.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:42.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.464+0000 7facdaa54640 1 Processor -- start 2023-10-30T13:42:42.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.464+0000 7facdaa54640 1 -- start start 2023-10-30T13:42:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facdaa54640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd4150830 0x7facd406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facdaa54640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7facd4070520 0x7facd40709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facdaa54640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40c2310 con 0x7facd40cb7d0 2023-10-30T13:42:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facdaa54640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd4156710 con 0x7facd4070520 2023-10-30T13:42:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facdaa54640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd40c4360 con 0x7facd4150830 2023-10-30T13:42:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd3fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd4150830 0x7facd406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd8ff1640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7facd4070520 0x7facd40709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd3fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd4150830 0x7facd406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:45874/0 (socket says 172.21.15.31:45874) 2023-10-30T13:42:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd3fff640 1 -- 172.21.15.31:0/1191101353 learned_addr learned my addr 172.21.15.31:0/1191101353 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd37fe640 1 --1- 172.21.15.31:0/1191101353 >> v1:172.21.15.33:6789/0 conn(0x7facd40cb7d0 0x7facd40c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:56584/0 (socket says 172.21.15.31:56584) 2023-10-30T13:42:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd3fff640 1 -- 172.21.15.31:0/1191101353 >> v2:172.21.15.33:3300/0 conn(0x7facd4070520 msgr2=0x7facd40709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd3fff640 1 --2- 172.21.15.31:0/1191101353 >> v2:172.21.15.33:3300/0 conn(0x7facd4070520 0x7facd40709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd3fff640 1 -- 172.21.15.31:0/1191101353 >> v1:172.21.15.33:6789/0 conn(0x7facd40cb7d0 legacy=0x7facd40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.468+0000 7facd3fff640 1 -- 172.21.15.31:0/1191101353 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd41569b0 con 0x7facd4150830 2023-10-30T13:42:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facd3fff640 1 --2- 172.21.15.31:0/1191101353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd4150830 0x7facd406b500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7facc4009870 tx=0x7facc401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d49f3790f2ca02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facd2ffd640 1 -- 172.21.15.31:0/1191101353 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40a8a30 con 0x7facd4150830 2023-10-30T13:42:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facd2ffd640 1 -- 172.21.15.31:0/1191101353 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc40a8b90 con 0x7facd4150830 2023-10-30T13:42:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facd2ffd640 1 -- 172.21.15.31:0/1191101353 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40b0440 con 0x7facd4150830 2023-10-30T13:42:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 -- 172.21.15.31:0/1191101353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd4150830 msgr2=0x7facd406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:42.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 --2- 172.21.15.31:0/1191101353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd4150830 0x7facd406b500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7facc4009870 tx=0x7facc401ed30 comp rx=0 tx=0).stop 2023-10-30T13:42:42.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 -- 172.21.15.31:0/1191101353 shutdown_connections 2023-10-30T13:42:42.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 --2- 172.21.15.31:0/1191101353 >> v2:172.21.15.33:3300/0 conn(0x7facd4070520 0x7facd40709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 --2- 172.21.15.31:0/1191101353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd4150830 0x7facd406b500 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 -- 172.21.15.31:0/1191101353 >> 172.21.15.31:0/1191101353 conn(0x7facd40cac70 msgr2=0x7facd40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:42.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 -- 172.21.15.31:0/1191101353 shutdown_connections 2023-10-30T13:42:42.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.472+0000 7facdaa54640 1 -- 172.21.15.31:0/1191101353 wait complete. 2023-10-30T13:42:42.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 Processor -- start 2023-10-30T13:42:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 -- start start 2023-10-30T13:42:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facd4070520 0x7facd40c0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd40cb7d0 0x7facd41ef500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facd4150830 0x7facd4159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd406c8d0 con 0x7facd4070520 2023-10-30T13:42:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd41efa40 con 0x7facd4150830 2023-10-30T13:42:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd40bef80 con 0x7facd40cb7d0 2023-10-30T13:42:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facd4070520 0x7facd40c0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facd4070520 0x7facd40c0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45002/0 (socket says 172.21.15.31:45002) 2023-10-30T13:42:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 -- 172.21.15.31:0/143608842 learned_addr learned my addr 172.21.15.31:0/143608842 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd37fe640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd40cb7d0 0x7facd41ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd8ff1640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facd4150830 0x7facd4159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 -- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd40cb7d0 msgr2=0x7facd41ef500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd40cb7d0 0x7facd41ef500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 -- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facd4150830 msgr2=0x7facd4159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facd4150830 0x7facd4159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 -- 172.21.15.31:0/143608842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40094d0 con 0x7facd4070520 2023-10-30T13:42:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd3fff640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facd4070520 0x7facd40c0cf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7facc40b3040 tx=0x7facc401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd0ff9640 1 -- 172.21.15.31:0/143608842 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40b0dd0 con 0x7facd4070520 2023-10-30T13:42:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd0ff9640 1 -- 172.21.15.31:0/143608842 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc40bb460 con 0x7facd4070520 2023-10-30T13:42:42.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd406c610 con 0x7facd4070520 2023-10-30T13:42:42.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facd0ff9640 1 -- 172.21.15.31:0/143608842 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40bb810 con 0x7facd4070520 2023-10-30T13:42:42.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd40be4a0 con 0x7facd4070520 2023-10-30T13:42:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.476+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd4066510 con 0x7facd4070520 2023-10-30T13:42:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.480+0000 7facd0ff9640 1 -- 172.21.15.31:0/143608842 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facc40b83f0 con 0x7facd4070520 2023-10-30T13:42:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.480+0000 7facd0ff9640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facc00664d0 0x7facc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:42.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.480+0000 7facd0ff9640 1 -- 172.21.15.31:0/143608842 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7facc4121920 con 0x7facd4070520 2023-10-30T13:42:42.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.480+0000 7facd37fe640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facc00664d0 0x7facc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:42.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.480+0000 7facd37fe640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facc00664d0 0x7facc0068990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7facd415aaa0 tx=0x7facc8007e50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:42.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.484+0000 7facd0ff9640 1 -- 172.21.15.31:0/143608842 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facc40eebc0 con 0x7facd4070520 2023-10-30T13:42:42.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facd404e170 con 0x7facd4070520 2023-10-30T13:42:42.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facd0ff9640 1 -- 172.21.15.31:0/143608842 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7facc40c0090 con 0x7facd4070520 2023-10-30T13:42:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:42.749 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:42.750 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:42.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facc00664d0 msgr2=0x7facc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:42.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facc00664d0 0x7facc0068990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7facd415aaa0 tx=0x7facc8007e50 comp rx=0 tx=0).stop 2023-10-30T13:42:42.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facd4070520 msgr2=0x7facd40c0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:42.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facd4070520 0x7facd40c0cf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7facc40b3040 tx=0x7facc401bad0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 shutdown_connections 2023-10-30T13:42:42.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7facc00664d0 0x7facc0068990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facd4150830 0x7facd4159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7facd40cb7d0 0x7facd41ef500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 --2- 172.21.15.31:0/143608842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facd4070520 0x7facd40c0cf0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:42.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 >> 172.21.15.31:0/143608842 conn(0x7facd40cac70 msgr2=0x7facd40c7680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:42.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 shutdown_connections 2023-10-30T13:42:42.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:42.748+0000 7facdaa54640 1 -- 172.21.15.31:0/143608842 wait complete. 2023-10-30T13:42:42.771 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:44.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:44.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc5929c5640 1 Processor -- start 2023-10-30T13:42:44.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc5929c5640 1 -- start start 2023-10-30T13:42:44.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc5929c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 0x7fc58c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc5929c5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c159f20 con 0x7fc58c0c4a10 2023-10-30T13:42:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc5929c5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c0c47e0 con 0x7fc58c153c80 2023-10-30T13:42:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc5929c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c15dec0 con 0x7fc58c15cf90 2023-10-30T13:42:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc58c0c4a10 0x7fc58c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37834/0 (socket says 172.21.15.31:37834) 2023-10-30T13:42:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 -- 172.21.15.31:0/2907522310 learned_addr learned my addr 172.21.15.31:0/2907522310 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc590f62640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 0x7fc58c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367436176 0 0) 0x7fc58c159f20 con 0x7fc58c0c4a10 2023-10-30T13:42:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc574003530 con 0x7fc58c0c4a10 2023-10-30T13:42:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 792311131 0 0) 0x7fc574003530 con 0x7fc58c0c4a10 2023-10-30T13:42:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 msgr2=0x7fc58c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 0x7fc58c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 >> v1:172.21.15.33:6789/0 conn(0x7fc58c153c80 legacy=0x7fc58c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c15d620 con 0x7fc58c0c4a10 2023-10-30T13:42:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fc580001680 con 0x7fc58c0c4a10 2023-10-30T13:42:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 >> v1:172.21.15.31:6789/0 conn(0x7fc58c0c4a10 legacy=0x7fc58c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5740039b0 0x7fc574003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc574004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 0x7fc574005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c15dec0 con 0x7fc5740047d0 2023-10-30T13:42:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57400bb00 con 0x7fc5740039b0 2023-10-30T13:42:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57400bcc0 con 0x7fc58c15cf90 2023-10-30T13:42:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc574004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58bfff640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5740039b0 0x7fc574003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc590f62640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 0x7fc574005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 -- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 msgr2=0x7fc574005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 0x7fc574005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 -- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5740039b0 msgr2=0x7fc574003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5740039b0 0x7fc574003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58b7fe640 1 -- 172.21.15.31:0/2907522310 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574003530 con 0x7fc5740047d0 2023-10-30T13:42:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc58bfff640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5740039b0 0x7fc574003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.936+0000 7fc590f62640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c15cf90 0x7fc574005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc58b7fe640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc574004c50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0c4e30 tx=0x7fc58001cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e46bf54348491f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580005ae0 con 0x7fc5740047d0 2023-10-30T13:42:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc580005c40 con 0x7fc5740047d0 2023-10-30T13:42:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc58affd640 1 -- 172.21.15.31:0/2907522310 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5800a58c0 con 0x7fc5740047d0 2023-10-30T13:42:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc5929c5640 1 -- 172.21.15.31:0/2907522310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 msgr2=0x7fc574004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc5929c5640 1 --2- 172.21.15.31:0/2907522310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc574004c50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0c4e30 tx=0x7fc58001cbc0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc590f62640 1 -- 172.21.15.31:0/2907522310 reap_dead start 2023-10-30T13:42:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc5929c5640 1 -- 172.21.15.31:0/2907522310 shutdown_connections 2023-10-30T13:42:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc5929c5640 1 -- 172.21.15.31:0/2907522310 >> 172.21.15.31:0/2907522310 conn(0x7fc58c0715c0 msgr2=0x7fc58c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc5929c5640 1 -- 172.21.15.31:0/2907522310 shutdown_connections 2023-10-30T13:42:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.940+0000 7fc5929c5640 1 -- 172.21.15.31:0/2907522310 wait complete. 2023-10-30T13:42:44.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 Processor -- start 2023-10-30T13:42:44.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 -- start start 2023-10-30T13:42:44.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc58c156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c14e190 0x7fc58c15c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc58c14e5d0 0x7fc58c149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1542f0 con 0x7fc5740047d0 2023-10-30T13:42:44.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c159280 con 0x7fc58c14e5d0 2023-10-30T13:42:44.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1f24b0 con 0x7fc58c14e190 2023-10-30T13:42:44.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc58c156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc58c156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51840/0 (socket says 172.21.15.31:51840) 2023-10-30T13:42:44.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 -- 172.21.15.31:0/179482428 learned_addr learned my addr 172.21.15.31:0/179482428 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:44.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58b7fe640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c14e190 0x7fc58c15c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc590f62640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc58c14e5d0 0x7fc58c149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 -- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c14e190 msgr2=0x7fc58c15c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c14e190 0x7fc58c15c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 -- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc58c14e5d0 msgr2=0x7fc58c149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc58c14e5d0 0x7fc58c149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 -- 172.21.15.31:0/179482428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c154ac0 con 0x7fc5740047d0 2023-10-30T13:42:44.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc58bfff640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc58c156e10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00e4d0 tx=0x7fc57c00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:44.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc588ff9640 1 -- 172.21.15.31:0/179482428 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0052e0 con 0x7fc5740047d0 2023-10-30T13:42:44.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc588ff9640 1 -- 172.21.15.31:0/179482428 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c005440 con 0x7fc5740047d0 2023-10-30T13:42:44.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc588ff9640 1 -- 172.21.15.31:0/179482428 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c003820 con 0x7fc5740047d0 2023-10-30T13:42:44.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c0cf650 con 0x7fc5740047d0 2023-10-30T13:42:44.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.944+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c159800 con 0x7fc5740047d0 2023-10-30T13:42:44.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.948+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c159b70 con 0x7fc5740047d0 2023-10-30T13:42:44.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.948+0000 7fc588ff9640 1 -- 172.21.15.31:0/179482428 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc57c003980 con 0x7fc5740047d0 2023-10-30T13:42:44.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.948+0000 7fc588ff9640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5780664d0 0x7fc578068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.948+0000 7fc588ff9640 1 -- 172.21.15.31:0/179482428 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fc57c083d60 con 0x7fc5740047d0 2023-10-30T13:42:44.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.948+0000 7fc58b7fe640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5780664d0 0x7fc578068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.948+0000 7fc58b7fe640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5780664d0 0x7fc578068990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ab000 tx=0x7fc5800a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:44.952+0000 7fc588ff9640 1 -- 172.21.15.31:0/179482428 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc57c051060 con 0x7fc5740047d0 2023-10-30T13:42:45.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.140+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc58c04e170 con 0x7fc5740047d0 2023-10-30T13:42:45.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc588ff9640 1 -- 172.21.15.31:0/179482428 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7fc57c054680 con 0x7fc5740047d0 2023-10-30T13:42:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:45.145 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:45.145 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:45.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5780664d0 msgr2=0x7fc578068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5780664d0 0x7fc578068990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ab000 tx=0x7fc5800a7070 comp rx=0 tx=0).stop 2023-10-30T13:42:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 msgr2=0x7fc58c156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc58c156e10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00e4d0 tx=0x7fc57c00e990 comp rx=0 tx=0).stop 2023-10-30T13:42:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 shutdown_connections 2023-10-30T13:42:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5780664d0 0x7fc578068990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:45.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc58c14e5d0 0x7fc58c149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:45.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc58c14e190 0x7fc58c15c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:45.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 --2- 172.21.15.31:0/179482428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5740047d0 0x7fc58c156e10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:45.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 >> 172.21.15.31:0/179482428 conn(0x7fc58c0715c0 msgr2=0x7fc58c0cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:45.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 shutdown_connections 2023-10-30T13:42:45.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:45.144+0000 7fc5929c5640 1 -- 172.21.15.31:0/179482428 wait complete. 2023-10-30T13:42:45.167 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:45.168 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:37.134913+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6836/210349556", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 210349556, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 210349556, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 45, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5470, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:45.350 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/28' 2023-10-30T13:42:45.350 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/24' 2023-10-30T13:42:45.350 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/29' 2023-10-30T13:42:45.351 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/32' 2023-10-30T13:42:45.351 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/22' 2023-10-30T13:42:45.351 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/25' 2023-10-30T13:42:45.351 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/30' 2023-10-30T13:42:45.351 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/33' 2023-10-30T13:42:45.351 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/23' 2023-10-30T13:42:45.352 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/35' 2023-10-30T13:42:45.352 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/26' 2023-10-30T13:42:45.352 INFO:tasks.workunit.client.0.smithi031.stdout:removed './216/37' 2023-10-30T13:42:45.352 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './216' 2023-10-30T13:42:45.352 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/21' 2023-10-30T13:42:45.353 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/27' 2023-10-30T13:42:45.353 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/31' 2023-10-30T13:42:45.353 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/36' 2023-10-30T13:42:45.353 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/20' 2023-10-30T13:42:45.353 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/39' 2023-10-30T13:42:45.353 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/34' 2023-10-30T13:42:45.354 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/38' 2023-10-30T13:42:45.354 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/28' 2023-10-30T13:42:45.354 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/24' 2023-10-30T13:42:45.354 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/29' 2023-10-30T13:42:45.354 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/32' 2023-10-30T13:42:45.355 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/22' 2023-10-30T13:42:45.355 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/25' 2023-10-30T13:42:45.355 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/30' 2023-10-30T13:42:45.355 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/33' 2023-10-30T13:42:45.355 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/23' 2023-10-30T13:42:45.355 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/35' 2023-10-30T13:42:45.356 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/26' 2023-10-30T13:42:45.356 INFO:tasks.workunit.client.0.smithi031.stdout:removed './217/37' 2023-10-30T13:42:45.356 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './217' 2023-10-30T13:42:45.356 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/21' 2023-10-30T13:42:45.356 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/27' 2023-10-30T13:42:45.357 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/31' 2023-10-30T13:42:45.357 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/36' 2023-10-30T13:42:45.357 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/20' 2023-10-30T13:42:45.357 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/39' 2023-10-30T13:42:45.357 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/34' 2023-10-30T13:42:45.357 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/38' 2023-10-30T13:42:45.358 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/28' 2023-10-30T13:42:45.358 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/24' 2023-10-30T13:42:45.358 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/29' 2023-10-30T13:42:45.358 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/32' 2023-10-30T13:42:45.358 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/22' 2023-10-30T13:42:45.358 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/25' 2023-10-30T13:42:45.359 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/30' 2023-10-30T13:42:45.359 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/33' 2023-10-30T13:42:45.359 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/23' 2023-10-30T13:42:45.359 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/35' 2023-10-30T13:42:45.359 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/26' 2023-10-30T13:42:45.359 INFO:tasks.workunit.client.0.smithi031.stdout:removed './218/37' 2023-10-30T13:42:45.360 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './218' 2023-10-30T13:42:45.360 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/21' 2023-10-30T13:42:45.360 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/27' 2023-10-30T13:42:45.360 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/31' 2023-10-30T13:42:45.360 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/36' 2023-10-30T13:42:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/20' 2023-10-30T13:42:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/39' 2023-10-30T13:42:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/34' 2023-10-30T13:42:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/38' 2023-10-30T13:42:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/28' 2023-10-30T13:42:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/24' 2023-10-30T13:42:45.362 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/29' 2023-10-30T13:42:45.362 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/32' 2023-10-30T13:42:45.362 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/22' 2023-10-30T13:42:45.362 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/25' 2023-10-30T13:42:45.362 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/30' 2023-10-30T13:42:45.363 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/33' 2023-10-30T13:42:45.363 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/23' 2023-10-30T13:42:45.363 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/35' 2023-10-30T13:42:45.363 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/26' 2023-10-30T13:42:45.363 INFO:tasks.workunit.client.0.smithi031.stdout:removed './219/37' 2023-10-30T13:42:45.363 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './219' 2023-10-30T13:42:45.364 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/21' 2023-10-30T13:42:45.364 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/27' 2023-10-30T13:42:45.364 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/31' 2023-10-30T13:42:45.364 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/36' 2023-10-30T13:42:45.364 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/20' 2023-10-30T13:42:45.364 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/39' 2023-10-30T13:42:45.365 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/34' 2023-10-30T13:42:45.365 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/38' 2023-10-30T13:42:45.365 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/28' 2023-10-30T13:42:45.365 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/24' 2023-10-30T13:42:45.365 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/29' 2023-10-30T13:42:45.365 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/32' 2023-10-30T13:42:45.366 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/22' 2023-10-30T13:42:45.366 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/25' 2023-10-30T13:42:45.366 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/30' 2023-10-30T13:42:45.366 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/33' 2023-10-30T13:42:45.366 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/23' 2023-10-30T13:42:45.367 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/35' 2023-10-30T13:42:45.367 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/26' 2023-10-30T13:42:45.367 INFO:tasks.workunit.client.0.smithi031.stdout:removed './220/37' 2023-10-30T13:42:45.367 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './220' 2023-10-30T13:42:45.367 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/21' 2023-10-30T13:42:45.367 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/27' 2023-10-30T13:42:45.368 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/31' 2023-10-30T13:42:45.368 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/36' 2023-10-30T13:42:45.368 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/20' 2023-10-30T13:42:45.368 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/39' 2023-10-30T13:42:45.368 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/34' 2023-10-30T13:42:45.369 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/38' 2023-10-30T13:42:45.369 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/28' 2023-10-30T13:42:45.369 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/24' 2023-10-30T13:42:45.369 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/29' 2023-10-30T13:42:45.369 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/32' 2023-10-30T13:42:45.369 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/22' 2023-10-30T13:42:45.370 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/25' 2023-10-30T13:42:45.370 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/30' 2023-10-30T13:42:45.370 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/33' 2023-10-30T13:42:45.370 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/23' 2023-10-30T13:42:45.370 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/35' 2023-10-30T13:42:45.371 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/26' 2023-10-30T13:42:45.371 INFO:tasks.workunit.client.0.smithi031.stdout:removed './221/37' 2023-10-30T13:42:45.371 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './221' 2023-10-30T13:42:45.371 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/21' 2023-10-30T13:42:45.371 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/27' 2023-10-30T13:42:45.371 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/31' 2023-10-30T13:42:45.372 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/36' 2023-10-30T13:42:45.372 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/20' 2023-10-30T13:42:45.372 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/39' 2023-10-30T13:42:45.372 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/34' 2023-10-30T13:42:45.372 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/38' 2023-10-30T13:42:45.372 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/28' 2023-10-30T13:42:45.373 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/24' 2023-10-30T13:42:45.373 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/29' 2023-10-30T13:42:45.373 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/32' 2023-10-30T13:42:45.373 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/22' 2023-10-30T13:42:45.373 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/25' 2023-10-30T13:42:45.373 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/30' 2023-10-30T13:42:45.374 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/33' 2023-10-30T13:42:45.374 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/23' 2023-10-30T13:42:45.374 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/35' 2023-10-30T13:42:45.374 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/26' 2023-10-30T13:42:45.374 INFO:tasks.workunit.client.0.smithi031.stdout:removed './222/37' 2023-10-30T13:42:45.375 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './222' 2023-10-30T13:42:45.375 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/21' 2023-10-30T13:42:45.375 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/27' 2023-10-30T13:42:45.375 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/31' 2023-10-30T13:42:45.375 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/36' 2023-10-30T13:42:45.376 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/20' 2023-10-30T13:42:45.376 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/39' 2023-10-30T13:42:45.376 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/34' 2023-10-30T13:42:45.376 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/38' 2023-10-30T13:42:45.376 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/28' 2023-10-30T13:42:45.376 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/24' 2023-10-30T13:42:45.377 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/29' 2023-10-30T13:42:45.377 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/32' 2023-10-30T13:42:45.377 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/22' 2023-10-30T13:42:45.377 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/25' 2023-10-30T13:42:45.377 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/30' 2023-10-30T13:42:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/33' 2023-10-30T13:42:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/23' 2023-10-30T13:42:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/35' 2023-10-30T13:42:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/26' 2023-10-30T13:42:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:removed './223/37' 2023-10-30T13:42:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './223' 2023-10-30T13:42:45.379 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/21' 2023-10-30T13:42:45.379 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/27' 2023-10-30T13:42:45.379 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/31' 2023-10-30T13:42:45.379 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/36' 2023-10-30T13:42:45.379 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/20' 2023-10-30T13:42:45.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/39' 2023-10-30T13:42:45.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/34' 2023-10-30T13:42:45.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/38' 2023-10-30T13:42:45.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/28' 2023-10-30T13:42:45.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/24' 2023-10-30T13:42:45.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/29' 2023-10-30T13:42:45.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/32' 2023-10-30T13:42:45.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/22' 2023-10-30T13:42:45.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/25' 2023-10-30T13:42:45.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/30' 2023-10-30T13:42:45.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/33' 2023-10-30T13:42:45.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/23' 2023-10-30T13:42:45.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/35' 2023-10-30T13:42:45.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/26' 2023-10-30T13:42:45.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './224/37' 2023-10-30T13:42:45.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './224' 2023-10-30T13:42:45.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/21' 2023-10-30T13:42:45.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/27' 2023-10-30T13:42:45.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/31' 2023-10-30T13:42:45.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/36' 2023-10-30T13:42:45.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/20' 2023-10-30T13:42:45.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/39' 2023-10-30T13:42:45.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/34' 2023-10-30T13:42:45.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/38' 2023-10-30T13:42:45.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/28' 2023-10-30T13:42:45.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/24' 2023-10-30T13:42:45.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/29' 2023-10-30T13:42:45.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/32' 2023-10-30T13:42:45.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/22' 2023-10-30T13:42:45.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/25' 2023-10-30T13:42:45.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/30' 2023-10-30T13:42:45.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/33' 2023-10-30T13:42:45.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/23' 2023-10-30T13:42:45.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/35' 2023-10-30T13:42:45.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/26' 2023-10-30T13:42:45.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './225/37' 2023-10-30T13:42:45.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './225' 2023-10-30T13:42:45.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/21' 2023-10-30T13:42:45.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/27' 2023-10-30T13:42:45.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/31' 2023-10-30T13:42:45.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/36' 2023-10-30T13:42:45.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/20' 2023-10-30T13:42:45.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/39' 2023-10-30T13:42:45.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/34' 2023-10-30T13:42:45.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/38' 2023-10-30T13:42:45.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/28' 2023-10-30T13:42:45.388 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/24' 2023-10-30T13:42:47.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:47.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ec51d640 1 Processor -- start 2023-10-30T13:42:47.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ec51d640 1 -- start start 2023-10-30T13:42:47.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ec51d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f15e4150830 0x7f15e406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:47.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ec51d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f15e4070520 0x7f15e4070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:47.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ec51d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e40c2310 con 0x7f15e40cb7d0 2023-10-30T13:42:47.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ec51d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e4156710 con 0x7f15e4150830 2023-10-30T13:42:47.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ec51d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e40c4360 con 0x7f15e4070520 2023-10-30T13:42:47.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ea2b9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f15e4150830 0x7f15e406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:47.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ea2b9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f15e4150830 0x7f15e406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51842/0 (socket says 172.21.15.31:51842) 2023-10-30T13:42:47.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ea2b9640 1 -- 172.21.15.31:0/3043894406 learned_addr learned my addr 172.21.15.31:0/3043894406 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:47.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15eaaba640 1 --2- 172.21.15.31:0/3043894406 >> v2:172.21.15.33:3300/0 conn(0x7f15e4070520 0x7f15e4070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:47.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ea2b9640 1 -- 172.21.15.31:0/3043894406 >> v2:172.21.15.33:3300/0 conn(0x7f15e4070520 msgr2=0x7f15e4070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:47.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ea2b9640 1 --2- 172.21.15.31:0/3043894406 >> v2:172.21.15.33:3300/0 conn(0x7f15e4070520 0x7f15e4070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ea2b9640 1 -- 172.21.15.31:0/3043894406 >> v1:172.21.15.31:6789/0 conn(0x7f15e40cb7d0 legacy=0x7f15e40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:47.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.320+0000 7f15ea2b9640 1 -- 172.21.15.31:0/3043894406 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e41569b0 con 0x7f15e4150830 2023-10-30T13:42:47.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ea2b9640 1 --2- 172.21.15.31:0/3043894406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e406b500 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f15e4066e10 tx=0x7f15e00213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec28683cf11c58b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:47.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15e92b7640 1 -- 172.21.15.31:0/3043894406 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e00b1600 con 0x7f15e4150830 2023-10-30T13:42:47.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15e92b7640 1 -- 172.21.15.31:0/3043894406 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e00ad050 con 0x7f15e4150830 2023-10-30T13:42:47.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15e92b7640 1 -- 172.21.15.31:0/3043894406 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e00b1980 con 0x7f15e4150830 2023-10-30T13:42:47.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 -- 172.21.15.31:0/3043894406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 msgr2=0x7f15e406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:47.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 --2- 172.21.15.31:0/3043894406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e406b500 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f15e4066e10 tx=0x7f15e00213c0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 -- 172.21.15.31:0/3043894406 shutdown_connections 2023-10-30T13:42:47.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 --2- 172.21.15.31:0/3043894406 >> v2:172.21.15.33:3300/0 conn(0x7f15e4070520 0x7f15e4070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 --2- 172.21.15.31:0/3043894406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e406b500 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 -- 172.21.15.31:0/3043894406 >> 172.21.15.31:0/3043894406 conn(0x7f15e40cac70 msgr2=0x7f15e40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:47.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 -- 172.21.15.31:0/3043894406 shutdown_connections 2023-10-30T13:42:47.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.324+0000 7f15ec51d640 1 -- 172.21.15.31:0/3043894406 wait complete. 2023-10-30T13:42:47.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 Processor -- start 2023-10-30T13:42:47.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 -- start start 2023-10-30T13:42:47.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e4070520 0x7f15e4154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:47.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f15e40cb7d0 0x7f15e4158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:47.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e41f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:47.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e41596b0 con 0x7f15e4150830 2023-10-30T13:42:47.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e41587b0 con 0x7f15e4070520 2023-10-30T13:42:47.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4156540 con 0x7f15e40cb7d0 2023-10-30T13:42:47.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e41f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:47.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e41f2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51846/0 (socket says 172.21.15.31:51846) 2023-10-30T13:42:47.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 -- 172.21.15.31:0/2231685307 learned_addr learned my addr 172.21.15.31:0/2231685307 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:47.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15e9ab8640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f15e40cb7d0 0x7f15e4158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:47.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ea2b9640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e4070520 0x7f15e4154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:47.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 -- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f15e40cb7d0 msgr2=0x7f15e4158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:47.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f15e40cb7d0 0x7f15e4158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 -- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e4070520 msgr2=0x7f15e4154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:47.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e4070520 0x7f15e4154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 -- 172.21.15.31:0/2231685307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e000b770 con 0x7f15e4150830 2023-10-30T13:42:47.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15eaaba640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e41f2310 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f15dc007e70 tx=0x7f15dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:47.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15daffd640 1 -- 172.21.15.31:0/2231685307 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc014420 con 0x7f15e4150830 2023-10-30T13:42:47.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15daffd640 1 -- 172.21.15.31:0/2231685307 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc0145a0 con 0x7f15e4150830 2023-10-30T13:42:47.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e4156260 con 0x7f15e4150830 2023-10-30T13:42:47.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15daffd640 1 -- 172.21.15.31:0/2231685307 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc014950 con 0x7f15e4150830 2023-10-30T13:42:47.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e4158160 con 0x7f15e4150830 2023-10-30T13:42:47.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.328+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e4156f50 con 0x7f15e4150830 2023-10-30T13:42:47.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.332+0000 7f15daffd640 1 -- 172.21.15.31:0/2231685307 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15dc014ab0 con 0x7f15e4150830 2023-10-30T13:42:47.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.332+0000 7f15daffd640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15c80664f0 0x7f15c80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:47.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.332+0000 7f15daffd640 1 -- 172.21.15.31:0/2231685307 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f15dc084960 con 0x7f15e4150830 2023-10-30T13:42:47.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.332+0000 7f15ea2b9640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15c80664f0 0x7f15c80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:47.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.332+0000 7f15ea2b9640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15c80664f0 0x7f15c80689b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f15e00b3000 tx=0x7f15e0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:47.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.336+0000 7f15daffd640 1 -- 172.21.15.31:0/2231685307 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15e4156f50 con 0x7f15e4150830 2023-10-30T13:42:47.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.524+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15e4158960 con 0x7f15e4150830 2023-10-30T13:42:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15daffd640 1 -- 172.21.15.31:0/2231685307 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f15e4158960 con 0x7f15e4150830 2023-10-30T13:42:47.528 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:47.529 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:47.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15c80664f0 msgr2=0x7f15c80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:47.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15c80664f0 0x7f15c80689b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f15e00b3000 tx=0x7f15e0008040 comp rx=0 tx=0).stop 2023-10-30T13:42:47.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 msgr2=0x7f15e41f2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:47.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e41f2310 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f15dc007e70 tx=0x7f15dc00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 shutdown_connections 2023-10-30T13:42:47.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15c80664f0 0x7f15c80689b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e4150830 0x7f15e41f2310 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f15e40cb7d0 0x7f15e4158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 --2- 172.21.15.31:0/2231685307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e4070520 0x7f15e4154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:47.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 >> 172.21.15.31:0/2231685307 conn(0x7f15e40cac70 msgr2=0x7f15e40c7680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:47.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 shutdown_connections 2023-10-30T13:42:47.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:47.528+0000 7f15ec51d640 1 -- 172.21.15.31:0/2231685307 wait complete. 2023-10-30T13:42:47.549 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:49.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:49.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5df0491640 1 Processor -- start 2023-10-30T13:42:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5df0491640 1 -- start start 2023-10-30T13:42:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5df0491640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f5de8153c80 0x7f5de815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5df0491640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5de80c42a0 0x7f5de80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5df0491640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de8159f20 con 0x7f5de80c4a10 2023-10-30T13:42:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5df0491640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de815dec0 con 0x7f5de80c42a0 2023-10-30T13:42:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5df0491640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de81599a0 con 0x7f5de8153c80 2023-10-30T13:42:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5deea2e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5de80c42a0 0x7f5de80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5deea2e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5de80c42a0 0x7f5de80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51850/0 (socket says 172.21.15.31:51850) 2023-10-30T13:42:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5deea2e640 1 -- 172.21.15.31:0/392636973 learned_addr learned my addr 172.21.15.31:0/392636973 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5dee22d640 1 --2- 172.21.15.31:0/392636973 >> v2:172.21.15.33:3300/0 conn(0x7f5de8153c80 0x7f5de815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5deea2e640 1 -- 172.21.15.31:0/392636973 >> v2:172.21.15.33:3300/0 conn(0x7f5de8153c80 msgr2=0x7f5de815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5deea2e640 1 --2- 172.21.15.31:0/392636973 >> v2:172.21.15.33:3300/0 conn(0x7f5de8153c80 0x7f5de815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5deea2e640 1 -- 172.21.15.31:0/392636973 >> v1:172.21.15.33:6789/0 conn(0x7f5de80c4a10 legacy=0x7f5de815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.704+0000 7f5deea2e640 1 -- 172.21.15.31:0/392636973 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de815d620 con 0x7f5de80c42a0 2023-10-30T13:42:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5deea2e640 1 --2- 172.21.15.31:0/392636973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de80c42a0 0x7f5de80ca1e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5de80d00f0 tx=0x7f5de001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32721b45f8f16b47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5ded22b640 1 -- 172.21.15.31:0/392636973 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00af400 con 0x7f5de80c42a0 2023-10-30T13:42:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5ded22b640 1 -- 172.21.15.31:0/392636973 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de00ab050 con 0x7f5de80c42a0 2023-10-30T13:42:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5ded22b640 1 -- 172.21.15.31:0/392636973 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00af780 con 0x7f5de80c42a0 2023-10-30T13:42:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 -- 172.21.15.31:0/392636973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de80c42a0 msgr2=0x7f5de80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 --2- 172.21.15.31:0/392636973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de80c42a0 0x7f5de80ca1e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5de80d00f0 tx=0x7f5de001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 -- 172.21.15.31:0/392636973 shutdown_connections 2023-10-30T13:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 --2- 172.21.15.31:0/392636973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de80c42a0 0x7f5de80ca1e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 --2- 172.21.15.31:0/392636973 >> v2:172.21.15.33:3300/0 conn(0x7f5de8153c80 0x7f5de815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 -- 172.21.15.31:0/392636973 >> 172.21.15.31:0/392636973 conn(0x7f5de80715c0 msgr2=0x7f5de80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 -- 172.21.15.31:0/392636973 shutdown_connections 2023-10-30T13:42:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.708+0000 7f5df0491640 1 -- 172.21.15.31:0/392636973 wait complete. 2023-10-30T13:42:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 Processor -- start 2023-10-30T13:42:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 -- start start 2023-10-30T13:42:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5de80c42a0 0x7f5de8152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5de80c4a10 0x7f5de8154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8153c80 0x7f5de81f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de814a4d0 con 0x7f5de8153c80 2023-10-30T13:42:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de80cef00 con 0x7f5de80c42a0 2023-10-30T13:42:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de8150270 con 0x7f5de80c4a10 2023-10-30T13:42:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8153c80 0x7f5de81f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8153c80 0x7f5de81f21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51858/0 (socket says 172.21.15.31:51858) 2023-10-30T13:42:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 -- 172.21.15.31:0/3100535359 learned_addr learned my addr 172.21.15.31:0/3100535359 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5dee22d640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5de80c42a0 0x7f5de8152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deda2c640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5de80c4a10 0x7f5de8154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 -- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5de80c4a10 msgr2=0x7f5de8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5de80c4a10 0x7f5de8154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 -- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5de80c42a0 msgr2=0x7f5de8152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5de80c42a0 0x7f5de8152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 -- 172.21.15.31:0/3100535359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0009b30 con 0x7f5de8153c80 2023-10-30T13:42:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5deea2e640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8153c80 0x7f5de81f21f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5de00b2040 tx=0x7f5de001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5ddeffd640 1 -- 172.21.15.31:0/3100535359 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00afdc0 con 0x7f5de8153c80 2023-10-30T13:42:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5ddeffd640 1 -- 172.21.15.31:0/3100535359 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de0009510 con 0x7f5de8153c80 2023-10-30T13:42:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de80cebd0 con 0x7f5de8153c80 2023-10-30T13:42:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5ddeffd640 1 -- 172.21.15.31:0/3100535359 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00b8640 con 0x7f5de8153c80 2023-10-30T13:42:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de80cf790 con 0x7f5de8153c80 2023-10-30T13:42:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.712+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de80cb660 con 0x7f5de8153c80 2023-10-30T13:42:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.716+0000 7f5ddeffd640 1 -- 172.21.15.31:0/3100535359 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5de00b6040 con 0x7f5de8153c80 2023-10-30T13:42:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.716+0000 7f5ddeffd640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5dcc0664a0 0x7f5dcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.716+0000 7f5ddeffd640 1 -- 172.21.15.31:0/3100535359 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f5de01220b0 con 0x7f5de8153c80 2023-10-30T13:42:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.716+0000 7f5dee22d640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5dcc0664a0 0x7f5dcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.716+0000 7f5dee22d640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5dcc0664a0 0x7f5dcc068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5de400b930 tx=0x7f5de4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.720+0000 7f5ddeffd640 1 -- 172.21.15.31:0/3100535359 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5de00ef290 con 0x7f5de8153c80 2023-10-30T13:42:49.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.912+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5de80cb8d0 con 0x7f5de8153c80 2023-10-30T13:42:49.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.912+0000 7f5ddeffd640 1 -- 172.21.15.31:0/3100535359 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f5de00f28b0 con 0x7f5de8153c80 2023-10-30T13:42:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:49.916 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:49.916 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:49.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5dcc0664a0 msgr2=0x7f5dcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5dcc0664a0 0x7f5dcc068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5de400b930 tx=0x7f5de4008040 comp rx=0 tx=0).stop 2023-10-30T13:42:49.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8153c80 msgr2=0x7f5de81f21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8153c80 0x7f5de81f21f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5de00b2040 tx=0x7f5de001bf60 comp rx=0 tx=0).stop 2023-10-30T13:42:49.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 shutdown_connections 2023-10-30T13:42:49.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5dcc0664a0 0x7f5dcc068960 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8153c80 0x7f5de81f21f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5de80c4a10 0x7f5de8154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 --2- 172.21.15.31:0/3100535359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5de80c42a0 0x7f5de8152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 >> 172.21.15.31:0/3100535359 conn(0x7f5de80715c0 msgr2=0x7f5de8158790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:49.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 shutdown_connections 2023-10-30T13:42:49.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:49.916+0000 7f5df0491640 1 -- 172.21.15.31:0/3100535359 wait complete. 2023-10-30T13:42:49.936 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:51.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:52.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e7d3e640 1 Processor -- start 2023-10-30T13:42:52.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e7d3e640 1 -- start start 2023-10-30T13:42:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e7d3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f88e0153c80 0x7f88e015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e7d3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e00c4a10 0x7f88e00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e7d3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e0159f20 con 0x7f88e015a3b0 2023-10-30T13:42:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e7d3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e015dec0 con 0x7f88e0153c80 2023-10-30T13:42:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e7d3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e01599a0 con 0x7f88e00c4a10 2023-10-30T13:42:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e5ada640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f88e0153c80 0x7f88e015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e5ada640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f88e0153c80 0x7f88e015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51860/0 (socket says 172.21.15.31:51860) 2023-10-30T13:42:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e5ada640 1 -- 172.21.15.31:0/562914573 learned_addr learned my addr 172.21.15.31:0/562914573 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e52d9640 1 --2- 172.21.15.31:0/562914573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e00c4a10 0x7f88e00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:52.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e5ada640 1 -- 172.21.15.31:0/562914573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e00c4a10 msgr2=0x7f88e00c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:52.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e5ada640 1 --2- 172.21.15.31:0/562914573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e00c4a10 0x7f88e00c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e5ada640 1 -- 172.21.15.31:0/562914573 >> v1:172.21.15.33:6789/0 conn(0x7f88e015a3b0 legacy=0x7f88e00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:52.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.084+0000 7f88e5ada640 1 -- 172.21.15.31:0/562914573 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e00c47e0 con 0x7f88e0153c80 2023-10-30T13:42:52.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e5ada640 1 --2- 172.21.15.31:0/562914573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e0153c80 0x7f88e015cf90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009870 tx=0x7f88dc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec05c3592cb0d47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:52.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e4ad8640 1 -- 172.21.15.31:0/562914573 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0a8a30 con 0x7f88e0153c80 2023-10-30T13:42:52.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e4ad8640 1 -- 172.21.15.31:0/562914573 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc0a8b90 con 0x7f88e0153c80 2023-10-30T13:42:52.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e4ad8640 1 -- 172.21.15.31:0/562914573 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0b0440 con 0x7f88e0153c80 2023-10-30T13:42:52.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 -- 172.21.15.31:0/562914573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e0153c80 msgr2=0x7f88e015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:52.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/562914573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e0153c80 0x7f88e015cf90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009870 tx=0x7f88dc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:42:52.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 -- 172.21.15.31:0/562914573 shutdown_connections 2023-10-30T13:42:52.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/562914573 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e00c4a10 0x7f88e00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/562914573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e0153c80 0x7f88e015cf90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 -- 172.21.15.31:0/562914573 >> 172.21.15.31:0/562914573 conn(0x7f88e00715c0 msgr2=0x7f88e00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:52.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 -- 172.21.15.31:0/562914573 shutdown_connections 2023-10-30T13:42:52.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.088+0000 7f88e7d3e640 1 -- 172.21.15.31:0/562914573 wait complete. 2023-10-30T13:42:52.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 Processor -- start 2023-10-30T13:42:52.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 -- start start 2023-10-30T13:42:52.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e00c4a10 0x7f88e014ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:52.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88e0153c80 0x7f88e00b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:52.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e015a3b0 0x7f88e00ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:52.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00ce990 con 0x7f88e00c4a10 2023-10-30T13:42:52.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00b26b0 con 0x7f88e0153c80 2023-10-30T13:42:52.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00cdd20 con 0x7f88e015a3b0 2023-10-30T13:42:52.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e00c4a10 0x7f88e014ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:52.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e00c4a10 0x7f88e014ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51876/0 (socket says 172.21.15.31:51876) 2023-10-30T13:42:52.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 -- 172.21.15.31:0/2634726643 learned_addr learned my addr 172.21.15.31:0/2634726643 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:52.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e62db640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e015a3b0 0x7f88e00ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:52.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e52d9640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88e0153c80 0x7f88e00b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:52.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 -- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e015a3b0 msgr2=0x7f88e00ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:52.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e015a3b0 0x7f88e00ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 -- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88e0153c80 msgr2=0x7f88e00b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:52.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88e0153c80 0x7f88e00b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 -- 172.21.15.31:0/2634726643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc0094d0 con 0x7f88e00c4a10 2023-10-30T13:42:52.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e62db640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e015a3b0 0x7f88e00ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:52.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e52d9640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88e0153c80 0x7f88e00b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:52.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e5ada640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e00c4a10 0x7f88e014ef80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0b3040 tx=0x7f88dc01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:52.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88d67fc640 1 -- 172.21.15.31:0/2634726643 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0ba600 con 0x7f88e00c4a10 2023-10-30T13:42:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88d67fc640 1 -- 172.21.15.31:0/2634726643 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc0ba780 con 0x7f88e00c4a10 2023-10-30T13:42:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88d67fc640 1 -- 172.21.15.31:0/2634726643 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0bab30 con 0x7f88e00c4a10 2023-10-30T13:42:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e00cf400 con 0x7f88e00c4a10 2023-10-30T13:42:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.092+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e00cf680 con 0x7f88e00c4a10 2023-10-30T13:42:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.096+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e004e170 con 0x7f88e00c4a10 2023-10-30T13:42:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.096+0000 7f88d67fc640 1 -- 172.21.15.31:0/2634726643 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88dc0bac90 con 0x7f88e00c4a10 2023-10-30T13:42:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.096+0000 7f88d67fc640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88c40664a0 0x7f88c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.096+0000 7f88d67fc640 1 -- 172.21.15.31:0/2634726643 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f88dc1226f0 con 0x7f88e00c4a10 2023-10-30T13:42:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.096+0000 7f88e52d9640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88c40664a0 0x7f88c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.096+0000 7f88e52d9640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88c40664a0 0x7f88c4068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f88e00ceff0 tx=0x7f88d0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.100+0000 7f88d67fc640 1 -- 172.21.15.31:0/2634726643 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88dc0ef8d0 con 0x7f88e00c4a10 2023-10-30T13:42:52.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.296+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88e00cad40 con 0x7f88e00c4a10 2023-10-30T13:42:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.296+0000 7f88d67fc640 1 -- 172.21.15.31:0/2634726643 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f88dc0b5090 con 0x7f88e00c4a10 2023-10-30T13:42:52.299 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:52.299 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88c40664a0 msgr2=0x7f88c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88c40664a0 0x7f88c4068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f88e00ceff0 tx=0x7f88d0008040 comp rx=0 tx=0).stop 2023-10-30T13:42:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e00c4a10 msgr2=0x7f88e014ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e00c4a10 0x7f88e014ef80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0b3040 tx=0x7f88dc01bad0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 shutdown_connections 2023-10-30T13:42:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f88c40664a0 0x7f88c4068960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88e015a3b0 0x7f88e00ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88e0153c80 0x7f88e00b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 --2- 172.21.15.31:0/2634726643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e00c4a10 0x7f88e014ef80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 >> 172.21.15.31:0/2634726643 conn(0x7f88e00715c0 msgr2=0x7f88e0152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 shutdown_connections 2023-10-30T13:42:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:52.300+0000 7f88e7d3e640 1 -- 172.21.15.31:0/2634726643 wait complete. 2023-10-30T13:42:52.319 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:54.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:54.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.472+0000 7ff645c46640 1 Processor -- start 2023-10-30T13:42:54.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.472+0000 7ff645c46640 1 -- start start 2023-10-30T13:42:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff645c46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff6400c6320 0x7ff64006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff645c46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff64006be90 0x7ff64006c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff645c46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400c55f0 con 0x7ff64014fc50 2023-10-30T13:42:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff645c46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6400c5f00 con 0x7ff6400c6320 2023-10-30T13:42:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff645c46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6401567b0 con 0x7ff64006be90 2023-10-30T13:42:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff6400c6320 0x7ff64006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff6400c6320 0x7ff64006fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51884/0 (socket says 172.21.15.31:51884) 2023-10-30T13:42:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 -- 172.21.15.31:0/344203457 learned_addr learned my addr 172.21.15.31:0/344203457 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff63ffff640 1 --2- 172.21.15.31:0/344203457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff64006be90 0x7ff64006c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 -- 172.21.15.31:0/344203457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff64006be90 msgr2=0x7ff64006c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 --2- 172.21.15.31:0/344203457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff64006be90 0x7ff64006c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 -- 172.21.15.31:0/344203457 >> v1:172.21.15.33:6789/0 conn(0x7ff64014fc50 legacy=0x7ff640070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:54.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 -- 172.21.15.31:0/344203457 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640156a70 con 0x7ff6400c6320 2023-10-30T13:42:54.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.476+0000 7ff637fff640 1 --2- 172.21.15.31:0/344203457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6400c6320 0x7ff64006fdf0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff628009870 tx=0x7ff62801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7ee66d16991479 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff63effd640 1 -- 172.21.15.31:0/344203457 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280a7070 con 0x7ff6400c6320 2023-10-30T13:42:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff63effd640 1 -- 172.21.15.31:0/344203457 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6280ab430 con 0x7ff6400c6320 2023-10-30T13:42:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff63effd640 1 -- 172.21.15.31:0/344203457 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280ab7c0 con 0x7ff6400c6320 2023-10-30T13:42:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- 172.21.15.31:0/344203457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6400c6320 msgr2=0x7ff64006fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 --2- 172.21.15.31:0/344203457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6400c6320 0x7ff64006fdf0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff628009870 tx=0x7ff62801c4b0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- 172.21.15.31:0/344203457 shutdown_connections 2023-10-30T13:42:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 --2- 172.21.15.31:0/344203457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff64006be90 0x7ff64006c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 --2- 172.21.15.31:0/344203457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6400c6320 0x7ff64006fdf0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- 172.21.15.31:0/344203457 >> 172.21.15.31:0/344203457 conn(0x7ff6400cac70 msgr2=0x7ff6400cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:54.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- 172.21.15.31:0/344203457 shutdown_connections 2023-10-30T13:42:54.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- 172.21.15.31:0/344203457 wait complete. 2023-10-30T13:42:54.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 Processor -- start 2023-10-30T13:42:54.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- start start 2023-10-30T13:42:54.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64006be90 0x7ff640158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:54.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff6400c6320 0x7ff6401f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff64014fc50 0x7ff6401597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff640159080 con 0x7ff64006be90 2023-10-30T13:42:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6401510d0 con 0x7ff64014fc50 2023-10-30T13:42:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff645c46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6401f26b0 con 0x7ff6400c6320 2023-10-30T13:42:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff63f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64006be90 0x7ff640158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff63f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64006be90 0x7ff640158b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51890/0 (socket says 172.21.15.31:51890) 2023-10-30T13:42:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.480+0000 7ff63f7fe640 1 -- 172.21.15.31:0/1107926807 learned_addr learned my addr 172.21.15.31:0/1107926807 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff637fff640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff6400c6320 0x7ff6401f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63ffff640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff64014fc50 0x7ff6401597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63f7fe640 1 -- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff6400c6320 msgr2=0x7ff6401f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63f7fe640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff6400c6320 0x7ff6401f1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63f7fe640 1 -- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff64014fc50 msgr2=0x7ff6401597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63f7fe640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff64014fc50 0x7ff6401597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63f7fe640 1 -- 172.21.15.31:0/1107926807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6280094d0 con 0x7ff64006be90 2023-10-30T13:42:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff637fff640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff6400c6320 0x7ff6401f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63f7fe640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64006be90 0x7ff640158b40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff63000c3c0 tx=0x7ff63000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63cff9640 1 -- 172.21.15.31:0/1107926807 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300052d0 con 0x7ff64006be90 2023-10-30T13:42:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63cff9640 1 -- 172.21.15.31:0/1107926807 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff630005430 con 0x7ff64006be90 2023-10-30T13:42:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63cff9640 1 -- 172.21.15.31:0/1107926807 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630003840 con 0x7ff64006be90 2023-10-30T13:42:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6401f2990 con 0x7ff64006be90 2023-10-30T13:42:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff640158060 con 0x7ff64006be90 2023-10-30T13:42:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64004e170 con 0x7ff64006be90 2023-10-30T13:42:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63cff9640 1 -- 172.21.15.31:0/1107926807 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6300039a0 con 0x7ff64006be90 2023-10-30T13:42:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63cff9640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6200664a0 0x7ff620068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.484+0000 7ff63cff9640 1 -- 172.21.15.31:0/1107926807 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ff630083640 con 0x7ff64006be90 2023-10-30T13:42:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.488+0000 7ff637fff640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6200664a0 0x7ff620068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.488+0000 7ff637fff640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6200664a0 0x7ff620068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff6280af000 tx=0x7ff628008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:54.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.488+0000 7ff63cff9640 1 -- 172.21.15.31:0/1107926807 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff630050820 con 0x7ff64006be90 2023-10-30T13:42:54.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff640072030 con 0x7ff64006be90 2023-10-30T13:42:54.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff63cff9640 1 -- 172.21.15.31:0/1107926807 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7ff630053e40 con 0x7ff64006be90 2023-10-30T13:42:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:54.681 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:54.682 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:54.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6200664a0 msgr2=0x7ff620068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:54.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6200664a0 0x7ff620068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff6280af000 tx=0x7ff628008040 comp rx=0 tx=0).stop 2023-10-30T13:42:54.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64006be90 msgr2=0x7ff640158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:54.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64006be90 0x7ff640158b40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff63000c3c0 tx=0x7ff63000c880 comp rx=0 tx=0).stop 2023-10-30T13:42:54.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 shutdown_connections 2023-10-30T13:42:54.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff6200664a0 0x7ff620068960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff64014fc50 0x7ff6401597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff6400c6320 0x7ff6401f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 --2- 172.21.15.31:0/1107926807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64006be90 0x7ff640158b40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:54.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 >> 172.21.15.31:0/1107926807 conn(0x7ff6400cac70 msgr2=0x7ff6400c49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:54.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 shutdown_connections 2023-10-30T13:42:54.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:54.680+0000 7ff645c46640 1 -- 172.21.15.31:0/1107926807 wait complete. 2023-10-30T13:42:54.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:55.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:55.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:56.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:56.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa239f5640 1 Processor -- start 2023-10-30T13:42:56.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa239f5640 1 -- start start 2023-10-30T13:42:56.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa239f5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa239f5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ffa1c0c82d0 0x7ffa1c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa239f5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c151aa0 con 0x7ffa1c0cd4a0 2023-10-30T13:42:56.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa239f5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1c0c74a0 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa239f5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c158510 con 0x7ffa1c1516c0 2023-10-30T13:42:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa20f90640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ffa1c0c82d0 0x7ffa1c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa21791640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa21f92640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7ffa1c0cd4a0 0x7ffa1c0c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:56918/0 (socket says 172.21.15.31:56918) 2023-10-30T13:42:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.856+0000 7ffa21f92640 1 -- 172.21.15.31:0/2561415721 learned_addr learned my addr 172.21.15.31:0/2561415721 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:56.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa21791640 1 -- 172.21.15.31:0/2561415721 >> v2:172.21.15.33:3300/0 conn(0x7ffa1c0c82d0 msgr2=0x7ffa1c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:56.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa21791640 1 --2- 172.21.15.31:0/2561415721 >> v2:172.21.15.33:3300/0 conn(0x7ffa1c0c82d0 0x7ffa1c0ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa21791640 1 -- 172.21.15.31:0/2561415721 >> v1:172.21.15.33:6789/0 conn(0x7ffa1c0cd4a0 legacy=0x7ffa1c0c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa21791640 1 -- 172.21.15.31:0/2561415721 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c158af0 con 0x7ffa1c1516c0 2023-10-30T13:42:56.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa20f90640 1 --2- 172.21.15.31:0/2561415721 >> v2:172.21.15.33:3300/0 conn(0x7ffa1c0c82d0 0x7ffa1c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:56.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa21791640 1 --2- 172.21.15.31:0/2561415721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0c7d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa180097c0 tx=0x7ffa1801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab5c7443d871822 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:56.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa13fff640 1 -- 172.21.15.31:0/2561415721 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180ab600 con 0x7ffa1c1516c0 2023-10-30T13:42:56.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa13fff640 1 -- 172.21.15.31:0/2561415721 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa180a7050 con 0x7ffa1c1516c0 2023-10-30T13:42:56.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.860+0000 7ffa13fff640 1 -- 172.21.15.31:0/2561415721 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180ab980 con 0x7ffa1c1516c0 2023-10-30T13:42:56.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- 172.21.15.31:0/2561415721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 msgr2=0x7ffa1c0c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 --2- 172.21.15.31:0/2561415721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0c7d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa180097c0 tx=0x7ffa1801c480 comp rx=0 tx=0).stop 2023-10-30T13:42:56.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- 172.21.15.31:0/2561415721 shutdown_connections 2023-10-30T13:42:56.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 --2- 172.21.15.31:0/2561415721 >> v2:172.21.15.33:3300/0 conn(0x7ffa1c0c82d0 0x7ffa1c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 --2- 172.21.15.31:0/2561415721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0c7d90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- 172.21.15.31:0/2561415721 >> 172.21.15.31:0/2561415721 conn(0x7ffa1c071fb0 msgr2=0x7ffa1c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:56.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- 172.21.15.31:0/2561415721 shutdown_connections 2023-10-30T13:42:56.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- 172.21.15.31:0/2561415721 wait complete. 2023-10-30T13:42:56.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 Processor -- start 2023-10-30T13:42:56.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- start start 2023-10-30T13:42:56.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa1c0c82d0 0x7ffa1c143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c0cd4a0 0x7ffa1c0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0cc370 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c1437d0 con 0x7ffa1c0cd4a0 2023-10-30T13:42:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa239f5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c1f2790 con 0x7ffa1c1516c0 2023-10-30T13:42:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa21791640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa1c0c82d0 0x7ffa1c143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa21791640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa1c0c82d0 0x7ffa1c143a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45956/0 (socket says 172.21.15.31:45956) 2023-10-30T13:42:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa21791640 1 -- 172.21.15.31:0/1878645260 learned_addr learned my addr 172.21.15.31:0/1878645260 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa20f90640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c0cd4a0 0x7ffa1c0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.864+0000 7ffa21f92640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa21791640 1 -- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 msgr2=0x7ffa1c0cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa21791640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa21791640 1 -- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c0cd4a0 msgr2=0x7ffa1c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa21791640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c0cd4a0 0x7ffa1c0cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa21791640 1 -- 172.21.15.31:0/1878645260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c009420 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa21791640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa1c0c82d0 0x7ffa1c143a60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ffa1801a0b0 tx=0x7ffa18019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa11ffb640 1 -- 172.21.15.31:0/1878645260 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180ba070 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa11ffb640 1 -- 172.21.15.31:0/1878645260 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa180b4400 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa11ffb640 1 -- 172.21.15.31:0/1878645260 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180b47d0 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa18009420 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1c161b90 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1c04e170 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.868+0000 7ffa11ffb640 1 -- 172.21.15.31:0/1878645260 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa180b2040 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.872+0000 7ffa11ffb640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffa000664a0 0x7ffa00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.872+0000 7ffa11ffb640 1 -- 172.21.15.31:0/1878645260 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ffa1811d8b0 con 0x7ffa1c0c82d0 2023-10-30T13:42:56.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.872+0000 7ffa20f90640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffa000664a0 0x7ffa00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.872+0000 7ffa20f90640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffa000664a0 0x7ffa00068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c06d000 tx=0x7ffa0c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:56.876+0000 7ffa11ffb640 1 -- 172.21.15.31:0/1878645260 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa180eaa90 con 0x7ffa1c0c82d0 2023-10-30T13:42:57.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.064+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa1c14cc90 con 0x7ffa1c0c82d0 2023-10-30T13:42:57.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa11ffb640 1 -- 172.21.15.31:0/1878645260 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7ffa180b0070 con 0x7ffa1c0c82d0 2023-10-30T13:42:57.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:57.068 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:57.069 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:57.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffa000664a0 msgr2=0x7ffa00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:57.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffa000664a0 0x7ffa00068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c06d000 tx=0x7ffa0c009190 comp rx=0 tx=0).stop 2023-10-30T13:42:57.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa1c0c82d0 msgr2=0x7ffa1c143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:57.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa1c0c82d0 0x7ffa1c143a60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ffa1801a0b0 tx=0x7ffa18019220 comp rx=0 tx=0).stop 2023-10-30T13:42:57.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 shutdown_connections 2023-10-30T13:42:57.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffa000664a0 0x7ffa00068960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:57.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffa1c1516c0 0x7ffa1c0cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:57.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c0cd4a0 0x7ffa1c0cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:57.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 --2- 172.21.15.31:0/1878645260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa1c0c82d0 0x7ffa1c143a60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:57.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 >> 172.21.15.31:0/1878645260 conn(0x7ffa1c071fb0 msgr2=0x7ffa1c160180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:57.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 shutdown_connections 2023-10-30T13:42:57.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:57.068+0000 7ffa239f5640 1 -- 172.21.15.31:0/1878645260 wait complete. 2023-10-30T13:42:57.089 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:57.090 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:37.134913+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6836/210349556", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 210349556, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 210349556, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 45, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5470, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:59.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:59.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f6563640 1 Processor -- start 2023-10-30T13:42:59.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f6563640 1 -- start start 2023-10-30T13:42:59.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f6563640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f24f0153b50 0x7f24f01597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f6563640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f015a770 con 0x7f24f00c62b0 2023-10-30T13:42:59.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f6563640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f00c9d40 con 0x7f24f00cb180 2023-10-30T13:42:59.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f6563640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f0159cf0 con 0x7f24f0153b50 2023-10-30T13:42:59.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f4d60640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f24f00c62b0 0x7f24f00ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54476/0 (socket says 172.21.15.31:54476) 2023-10-30T13:42:59.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f4d60640 1 -- 172.21.15.31:0/759046960 learned_addr learned my addr 172.21.15.31:0/759046960 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:42:59.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5561640 1 --2- 172.21.15.31:0/759046960 >> v2:172.21.15.33:3300/0 conn(0x7f24f0153b50 0x7f24f01597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760062904 0 0) 0x7f24e0001120 con 0x7f24f00c62b0 2023-10-30T13:42:59.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24dc003530 con 0x7f24f00c62b0 2023-10-30T13:42:59.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2234551872 0 0) 0x7f24dc003530 con 0x7f24f00c62b0 2023-10-30T13:42:59.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 >> v2:172.21.15.33:3300/0 conn(0x7f24f0153b50 msgr2=0x7f24f01597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:59.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 --2- 172.21.15.31:0/759046960 >> v2:172.21.15.33:3300/0 conn(0x7f24f0153b50 0x7f24f01597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 >> v1:172.21.15.33:6789/0 conn(0x7f24f00cb180 legacy=0x7f24f006ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:59.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f0153f30 con 0x7f24f00c62b0 2023-10-30T13:42:59.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f24e0001810 con 0x7f24f00c62b0 2023-10-30T13:42:59.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 >> v1:172.21.15.31:6789/0 conn(0x7f24f00c62b0 legacy=0x7f24f00ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:59.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24dc003980 0x7f24dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24dc0047d0 0x7f24dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc005190 0x7f24dc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0159cf0 con 0x7f24dc003980 2023-10-30T13:42:59.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc00c040 con 0x7f24dc005190 2023-10-30T13:42:59.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc00c200 con 0x7f24dc0047d0 2023-10-30T13:42:59.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5561640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24dc003980 0x7f24dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f4d60640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24dc0047d0 0x7f24dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5d62640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc005190 0x7f24dc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5561640 1 -- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24dc0047d0 msgr2=0x7f24dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:59.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5561640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24dc0047d0 0x7f24dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5561640 1 -- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc005190 msgr2=0x7f24dc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:59.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5561640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc005190 0x7f24dc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5561640 1 -- 172.21.15.31:0/759046960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc003530 con 0x7f24dc003980 2023-10-30T13:42:59.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.248+0000 7f24f5d62640 1 -- 172.21.15.31:0/759046960 reap_dead start 2023-10-30T13:42:59.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f5561640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24dc003980 0x7f24dc003da0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009ae0 tx=0x7f24ec01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db34117d0217fa6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:59.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0a5640 con 0x7f24dc003980 2023-10-30T13:42:59.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec0a57a0 con 0x7f24dc003980 2023-10-30T13:42:59.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24e7fff640 1 -- 172.21.15.31:0/759046960 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0a5b50 con 0x7f24dc003980 2023-10-30T13:42:59.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f6563640 1 -- 172.21.15.31:0/759046960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24dc003980 msgr2=0x7f24dc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:59.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f6563640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24dc003980 0x7f24dc003da0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009ae0 tx=0x7f24ec01ca50 comp rx=0 tx=0).stop 2023-10-30T13:42:59.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f6563640 1 -- 172.21.15.31:0/759046960 shutdown_connections 2023-10-30T13:42:59.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f6563640 1 --2- 172.21.15.31:0/759046960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24dc003980 0x7f24dc003da0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f6563640 1 -- 172.21.15.31:0/759046960 >> 172.21.15.31:0/759046960 conn(0x7f24f0065a80 msgr2=0x7f24f0065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:59.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f6563640 1 -- 172.21.15.31:0/759046960 shutdown_connections 2023-10-30T13:42:59.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.252+0000 7f24f6563640 1 -- 172.21.15.31:0/759046960 wait complete. 2023-10-30T13:42:59.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 Processor -- start 2023-10-30T13:42:59.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 -- start start 2023-10-30T13:42:59.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc003980 0x7f24f01ef770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24f00c3730 0x7f24f0149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24f0149fb0 0x7f24f0157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f00c32a0 con 0x7f24f00c3730 2023-10-30T13:42:59.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f00c5660 con 0x7f24dc003980 2023-10-30T13:42:59.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f00c5940 con 0x7f24f0149fb0 2023-10-30T13:42:59.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24f00c3730 0x7f24f0149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24f00c3730 0x7f24f0149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45964/0 (socket says 172.21.15.31:45964) 2023-10-30T13:42:59.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 -- 172.21.15.31:0/1474782949 learned_addr learned my addr 172.21.15.31:0/1474782949 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:42:59.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f5561640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc003980 0x7f24f01ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f5d62640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24f0149fb0 0x7f24f0157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 -- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24f0149fb0 msgr2=0x7f24f0157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:59.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24f0149fb0 0x7f24f0157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 -- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc003980 msgr2=0x7f24f01ef770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:59.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc003980 0x7f24f01ef770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 -- 172.21.15.31:0/1474782949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f00c2f30 con 0x7f24f00c3730 2023-10-30T13:42:59.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f5561640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc003980 0x7f24f01ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:59.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f4d60640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24f00c3730 0x7f24f0149a70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f24e0011820 tx=0x7f24e0011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:59.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24e5ffb640 1 -- 172.21.15.31:0/1474782949 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e000fb60 con 0x7f24f00c3730 2023-10-30T13:42:59.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24e5ffb640 1 -- 172.21.15.31:0/1474782949 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e000fd00 con 0x7f24f00c3730 2023-10-30T13:42:59.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f0066620 con 0x7f24f00c3730 2023-10-30T13:42:59.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24e5ffb640 1 -- 172.21.15.31:0/1474782949 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0015650 con 0x7f24f00c3730 2023-10-30T13:42:59.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.256+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f00c4bf0 con 0x7f24f00c3730 2023-10-30T13:42:59.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.260+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f00c2d90 con 0x7f24f00c3730 2023-10-30T13:42:59.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.260+0000 7f24e5ffb640 1 -- 172.21.15.31:0/1474782949 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f24e00157b0 con 0x7f24f00c3730 2023-10-30T13:42:59.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.260+0000 7f24e5ffb640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24d40664a0 0x7f24d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:59.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.260+0000 7f24e5ffb640 1 -- 172.21.15.31:0/1474782949 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f24e00872e0 con 0x7f24f00c3730 2023-10-30T13:42:59.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.260+0000 7f24f5561640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24d40664a0 0x7f24d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:59.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.260+0000 7f24f5561640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24d40664a0 0x7f24d4068960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0af000 tx=0x7f24ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:59.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.264+0000 7f24e5ffb640 1 -- 172.21.15.31:0/1474782949 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f24f00c2d90 con 0x7f24f00c3730 2023-10-30T13:42:59.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24f014a390 con 0x7f24f00c3730 2023-10-30T13:42:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24e5ffb640 1 -- 172.21.15.31:0/1474782949 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f24f014a390 con 0x7f24f00c3730 2023-10-30T13:42:59.457 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:42:59.457 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:42:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24d40664a0 msgr2=0x7f24d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24d40664a0 0x7f24d4068960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0af000 tx=0x7f24ec008040 comp rx=0 tx=0).stop 2023-10-30T13:42:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24f00c3730 msgr2=0x7f24f0149a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24f00c3730 0x7f24f0149a70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f24e0011820 tx=0x7f24e0011ce0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 shutdown_connections 2023-10-30T13:42:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24d40664a0 0x7f24d4068960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f24f0149fb0 0x7f24f0157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24f00c3730 0x7f24f0149a70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 --2- 172.21.15.31:0/1474782949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24dc003980 0x7f24f01ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 >> 172.21.15.31:0/1474782949 conn(0x7f24f0065a80 msgr2=0x7f24f00c6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 shutdown_connections 2023-10-30T13:42:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:42:59.456+0000 7f24f6563640 1 -- 172.21.15.31:0/1474782949 wait complete. 2023-10-30T13:42:59.478 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:01.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac9901640 1 Processor -- start 2023-10-30T13:43:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac9901640 1 -- start start 2023-10-30T13:43:01.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac9901640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5ac4151840 0x7f5ac40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac9901640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac40c5030 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac9901640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac4157830 con 0x7f5ac4151c20 2023-10-30T13:43:01.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac9901640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac4157b10 con 0x7f5ac4151840 2023-10-30T13:43:01.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac2ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5ac4151840 0x7f5ac40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac37fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f5ac40cb8b0 0x7f5ac406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54478/0 (socket says 172.21.15.31:54478) 2023-10-30T13:43:01.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac37fe640 1 -- 172.21.15.31:0/3566831209 learned_addr learned my addr 172.21.15.31:0/3566831209 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:01.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac27fc640 1 -- 172.21.15.31:0/3566831209 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134773050 0 0) 0x7f5ac40c5030 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac27fc640 1 -- 172.21.15.31:0/3566831209 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab0003530 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/3566831209 >> v1:172.21.15.33:6789/0 conn(0x7f5ac4151c20 legacy=0x7f5ac4072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/3566831209 >> v1:172.21.15.31:6789/0 conn(0x7f5ac40cb8b0 legacy=0x7f5ac406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.636+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/3566831209 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4157df0 con 0x7f5ac4151840 2023-10-30T13:43:01.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac2ffd640 1 --2- 172.21.15.31:0/3566831209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac4151840 0x7f5ac40c49a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40097c0 tx=0x7f5ab401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba10d6e46731db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:01.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac27fc640 1 -- 172.21.15.31:0/3566831209 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40a8970 con 0x7f5ac4151840 2023-10-30T13:43:01.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac27fc640 1 -- 172.21.15.31:0/3566831209 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab40a8ad0 con 0x7f5ac4151840 2023-10-30T13:43:01.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac27fc640 1 -- 172.21.15.31:0/3566831209 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40a8e60 con 0x7f5ac4151840 2023-10-30T13:43:01.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 -- 172.21.15.31:0/3566831209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac4151840 msgr2=0x7f5ac40c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 --2- 172.21.15.31:0/3566831209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac4151840 0x7f5ac40c49a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40097c0 tx=0x7f5ab401ec70 comp rx=0 tx=0).stop 2023-10-30T13:43:01.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 -- 172.21.15.31:0/3566831209 shutdown_connections 2023-10-30T13:43:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 --2- 172.21.15.31:0/3566831209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac4151840 0x7f5ac40c49a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 -- 172.21.15.31:0/3566831209 >> 172.21.15.31:0/3566831209 conn(0x7f5ac40cac70 msgr2=0x7f5ac40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 -- 172.21.15.31:0/3566831209 shutdown_connections 2023-10-30T13:43:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 -- 172.21.15.31:0/3566831209 wait complete. 2023-10-30T13:43:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 Processor -- start 2023-10-30T13:43:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.640+0000 7f5ac9901640 1 -- start start 2023-10-30T13:43:01.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40cb8b0 0x7f5ac406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ac4151840 0x7f5ac41ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ac4151c20 0x7f5ac4146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4155630 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4155830 con 0x7f5ac4151c20 2023-10-30T13:43:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4070dd0 con 0x7f5ac4151840 2023-10-30T13:43:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40cb8b0 0x7f5ac406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40cb8b0 0x7f5ac406b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45970/0 (socket says 172.21.15.31:45970) 2023-10-30T13:43:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/1197576617 learned_addr learned my addr 172.21.15.31:0/1197576617 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac37fe640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ac4151c20 0x7f5ac4146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5abbfff640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ac4151840 0x7f5ac41ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ac4151840 msgr2=0x7f5ac41ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ac4151840 0x7f5ac41ee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ac4151c20 msgr2=0x7f5ac4146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ac4151c20 0x7f5ac4146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/1197576617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4009420 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac2ffd640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40cb8b0 0x7f5ac406b760 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40b3040 tx=0x7f5ab401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac0ff9640 1 -- 172.21.15.31:0/1197576617 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40b9630 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac0ff9640 1 -- 172.21.15.31:0/1197576617 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab40b97b0 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac0ff9640 1 -- 172.21.15.31:0/1197576617 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40b9b60 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac4155ad0 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac41f1f60 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.644+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac40c55f0 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.648+0000 7f5ac0ff9640 1 -- 172.21.15.31:0/1197576617 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ab40b7040 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.648+0000 7f5ac0ff9640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5aa40664d0 0x7f5aa4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.648+0000 7f5ac0ff9640 1 -- 172.21.15.31:0/1197576617 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f5ab4121ff0 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.648+0000 7f5abbfff640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5aa40664d0 0x7f5aa4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.648+0000 7f5abbfff640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5aa40664d0 0x7f5aa4068990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5ac41e02b0 tx=0x7f5aac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:01.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.652+0000 7f5ac0ff9640 1 -- 172.21.15.31:0/1197576617 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ab40ef310 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.840+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ac404e170 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.840+0000 7f5ac0ff9640 1 -- 172.21.15.31:0/1197576617 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f5ab40b5070 con 0x7f5ac40cb8b0 2023-10-30T13:43:01.844 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:01.844 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:01.844 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:43:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5aa40664d0 msgr2=0x7f5aa4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5aa40664d0 0x7f5aa4068990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5ac41e02b0 tx=0x7f5aac009040 comp rx=0 tx=0).stop 2023-10-30T13:43:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40cb8b0 msgr2=0x7f5ac406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40cb8b0 0x7f5ac406b760 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40b3040 tx=0x7f5ab401ba10 comp rx=0 tx=0).stop 2023-10-30T13:43:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 shutdown_connections 2023-10-30T13:43:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5aa40664d0 0x7f5aa4068990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ac4151c20 0x7f5ac4146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ac4151840 0x7f5ac41ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 --2- 172.21.15.31:0/1197576617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40cb8b0 0x7f5ac406b760 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 >> 172.21.15.31:0/1197576617 conn(0x7f5ac40cac70 msgr2=0x7f5ac414d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 shutdown_connections 2023-10-30T13:43:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:01.844+0000 7f5ac9901640 1 -- 172.21.15.31:0/1197576617 wait complete. 2023-10-30T13:43:01.865 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:03.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:04.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa373e640 1 Processor -- start 2023-10-30T13:43:04.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa373e640 1 -- start start 2023-10-30T13:43:04.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa373e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3f9c06b760 0x7f3f9c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:04.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa373e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06c420 0x7f3f9c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:04.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa373e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c071e00 con 0x7f3f9c06bb40 2023-10-30T13:43:04.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa373e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9c070020 con 0x7f3f9c06b760 2023-10-30T13:43:04.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa373e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c15eb90 con 0x7f3f9c06c420 2023-10-30T13:43:04.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.020+0000 7f3fa1cdb640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06c420 0x7f3f9c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:04.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa14da640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3f9c06b760 0x7f3f9c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:04.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa1cdb640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06c420 0x7f3f9c0c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:51052/0 (socket says 172.21.15.31:51052) 2023-10-30T13:43:04.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/1934353031 learned_addr learned my addr 172.21.15.31:0/1934353031 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa0cd9640 1 --1- 172.21.15.31:0/1934353031 >> v1:172.21.15.33:6789/0 conn(0x7f3f9c06bb40 0x7f3f9c157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:56940/0 (socket says 172.21.15.31:56940) 2023-10-30T13:43:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/1934353031 >> v2:172.21.15.33:3300/0 conn(0x7f3f9c06b760 msgr2=0x7f3f9c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa1cdb640 1 --2- 172.21.15.31:0/1934353031 >> v2:172.21.15.33:3300/0 conn(0x7f3f9c06b760 0x7f3f9c15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/1934353031 >> v1:172.21.15.33:6789/0 conn(0x7f3f9c06bb40 legacy=0x7f3f9c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/1934353031 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c070f10 con 0x7f3f9c06c420 2023-10-30T13:43:04.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.024+0000 7f3fa1cdb640 1 --2- 172.21.15.31:0/1934353031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06c420 0x7f3f9c0c57a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009e50 tx=0x7f3f9401c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8d9ed05134b70f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3f93fff640 1 -- 172.21.15.31:0/1934353031 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f940a58d0 con 0x7f3f9c06c420 2023-10-30T13:43:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3f93fff640 1 -- 172.21.15.31:0/1934353031 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f940a5a30 con 0x7f3f9c06c420 2023-10-30T13:43:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3f93fff640 1 -- 172.21.15.31:0/1934353031 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f940a5dc0 con 0x7f3f9c06c420 2023-10-30T13:43:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- 172.21.15.31:0/1934353031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06c420 msgr2=0x7f3f9c0c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 --2- 172.21.15.31:0/1934353031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06c420 0x7f3f9c0c57a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009e50 tx=0x7f3f9401c9e0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- 172.21.15.31:0/1934353031 shutdown_connections 2023-10-30T13:43:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 --2- 172.21.15.31:0/1934353031 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06c420 0x7f3f9c0c57a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 --2- 172.21.15.31:0/1934353031 >> v2:172.21.15.33:3300/0 conn(0x7f3f9c06b760 0x7f3f9c15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- 172.21.15.31:0/1934353031 >> 172.21.15.31:0/1934353031 conn(0x7f3f9c0cac70 msgr2=0x7f3f9c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- 172.21.15.31:0/1934353031 shutdown_connections 2023-10-30T13:43:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- 172.21.15.31:0/1934353031 wait complete. 2023-10-30T13:43:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 Processor -- start 2023-10-30T13:43:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- start start 2023-10-30T13:43:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f9c06b760 0x7f3f9c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06bb40 0x7f3f9c1f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f9c06c420 0x7f3f9c1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c14d870 con 0x7f3f9c06c420 2023-10-30T13:43:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0c2f60 con 0x7f3f9c06b760 2023-10-30T13:43:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa373e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c14cfd0 con 0x7f3f9c06bb40 2023-10-30T13:43:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa1cdb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f9c06c420 0x7f3f9c1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa14da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f9c06b760 0x7f3f9c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa1cdb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f9c06c420 0x7f3f9c1561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45972/0 (socket says 172.21.15.31:45972) 2023-10-30T13:43:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/2063470081 learned_addr learned my addr 172.21.15.31:0/2063470081 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.028+0000 7f3fa0cd9640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06bb40 0x7f3f9c1f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06bb40 msgr2=0x7f3f9c1f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa1cdb640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06bb40 0x7f3f9c1f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f9c06b760 msgr2=0x7f3f9c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa1cdb640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f9c06b760 0x7f3f9c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa1cdb640 1 -- 172.21.15.31:0/2063470081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94009b30 con 0x7f3f9c06c420 2023-10-30T13:43:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa1cdb640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f9c06c420 0x7f3f9c1561b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3f940af040 tx=0x7f3f94019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3f91ffb640 1 -- 172.21.15.31:0/2063470081 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b8670 con 0x7f3f9c06c420 2023-10-30T13:43:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3f91ffb640 1 -- 172.21.15.31:0/2063470081 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f940b87d0 con 0x7f3f9c06c420 2023-10-30T13:43:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3f91ffb640 1 -- 172.21.15.31:0/2063470081 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b8b60 con 0x7f3f9c06c420 2023-10-30T13:43:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c14d2b0 con 0x7f3f9c06c420 2023-10-30T13:43:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c0c2bf0 con 0x7f3f9c06c420 2023-10-30T13:43:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9c066510 con 0x7f3f9c06c420 2023-10-30T13:43:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.032+0000 7f3f91ffb640 1 -- 172.21.15.31:0/2063470081 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f940b8cc0 con 0x7f3f9c06c420 2023-10-30T13:43:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.036+0000 7f3f91ffb640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f800664a0 0x7f3f80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.036+0000 7f3f91ffb640 1 -- 172.21.15.31:0/2063470081 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f3f9411f430 con 0x7f3f9c06c420 2023-10-30T13:43:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.036+0000 7f3fa14da640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f800664a0 0x7f3f80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:04.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.036+0000 7f3fa14da640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f800664a0 0x7f3f80068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009ac0 tx=0x7f3f98009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.040+0000 7f3f91ffb640 1 -- 172.21.15.31:0/2063470081 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f940ec7e0 con 0x7f3f9c06c420 2023-10-30T13:43:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.228+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f9c04e170 con 0x7f3f9c06c420 2023-10-30T13:43:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.228+0000 7f3f91ffb640 1 -- 172.21.15.31:0/2063470081 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f3f940efe00 con 0x7f3f9c06c420 2023-10-30T13:43:04.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:04.231 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:43:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f800664a0 msgr2=0x7f3f80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f800664a0 0x7f3f80068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009ac0 tx=0x7f3f98009190 comp rx=0 tx=0).stop 2023-10-30T13:43:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f9c06c420 msgr2=0x7f3f9c1561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f9c06c420 0x7f3f9c1561b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3f940af040 tx=0x7f3f94019780 comp rx=0 tx=0).stop 2023-10-30T13:43:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 shutdown_connections 2023-10-30T13:43:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f800664a0 0x7f3f80068960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f9c06c420 0x7f3f9c1561b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f9c06bb40 0x7f3f9c1f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 --2- 172.21.15.31:0/2063470081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f9c06b760 0x7f3f9c0c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:04.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 >> 172.21.15.31:0/2063470081 conn(0x7f3f9c0cac70 msgr2=0x7f3f9c152260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:04.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 shutdown_connections 2023-10-30T13:43:04.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:04.232+0000 7f3fa373e640 1 -- 172.21.15.31:0/2063470081 wait complete. 2023-10-30T13:43:04.252 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:06.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:06.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.404+0000 7f97574d6640 1 Processor -- start 2023-10-30T13:43:06.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.404+0000 7f97574d6640 1 -- start start 2023-10-30T13:43:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f97574d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9750153c80 0x7f975015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f97574d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c42a0 0x7f97500ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f97574d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9750159f20 con 0x7f97500c4a10 2023-10-30T13:43:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f97574d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975015dec0 con 0x7f9750153c80 2023-10-30T13:43:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f97574d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97501599a0 con 0x7f97500c42a0 2023-10-30T13:43:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755272640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9750153c80 0x7f975015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755272640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9750153c80 0x7f975015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53194/0 (socket says 172.21.15.31:53194) 2023-10-30T13:43:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755272640 1 -- 172.21.15.31:0/3405675719 learned_addr learned my addr 172.21.15.31:0/3405675719 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755a73640 1 --2- 172.21.15.31:0/3405675719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c42a0 0x7f97500ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755272640 1 -- 172.21.15.31:0/3405675719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c42a0 msgr2=0x7f97500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755272640 1 --2- 172.21.15.31:0/3405675719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c42a0 0x7f97500ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755272640 1 -- 172.21.15.31:0/3405675719 >> v1:172.21.15.33:6789/0 conn(0x7f97500c4a10 legacy=0x7f975015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:06.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.408+0000 7f9755272640 1 -- 172.21.15.31:0/3405675719 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975015d620 con 0x7f9750153c80 2023-10-30T13:43:06.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f9755272640 1 --2- 172.21.15.31:0/3405675719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f975015cf90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f974c009870 tx=0x7f974c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3051b529c7539101 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:06.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f9747fff640 1 -- 172.21.15.31:0/3405675719 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c0a89b0 con 0x7f9750153c80 2023-10-30T13:43:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f9747fff640 1 -- 172.21.15.31:0/3405675719 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c0a8b10 con 0x7f9750153c80 2023-10-30T13:43:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f9747fff640 1 -- 172.21.15.31:0/3405675719 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c0a8ea0 con 0x7f9750153c80 2023-10-30T13:43:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 -- 172.21.15.31:0/3405675719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 msgr2=0x7f975015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 --2- 172.21.15.31:0/3405675719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f975015cf90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f974c009870 tx=0x7f974c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 -- 172.21.15.31:0/3405675719 shutdown_connections 2023-10-30T13:43:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 --2- 172.21.15.31:0/3405675719 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c42a0 0x7f97500ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 --2- 172.21.15.31:0/3405675719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f975015cf90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 -- 172.21.15.31:0/3405675719 >> 172.21.15.31:0/3405675719 conn(0x7f97500715c0 msgr2=0x7f97500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:06.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 -- 172.21.15.31:0/3405675719 shutdown_connections 2023-10-30T13:43:06.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 -- 172.21.15.31:0/3405675719 wait complete. 2023-10-30T13:43:06.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 Processor -- start 2023-10-30T13:43:06.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.412+0000 7f97574d6640 1 -- start start 2023-10-30T13:43:06.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97500c42a0 0x7f975015c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:06.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c4a10 0x7f975015b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:06.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f9750155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:06.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750155780 con 0x7f9750153c80 2023-10-30T13:43:06.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97500ce090 con 0x7f97500c42a0 2023-10-30T13:43:06.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97500ce2f0 con 0x7f97500c4a10 2023-10-30T13:43:06.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f9750155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:06.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f9750155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53210/0 (socket says 172.21.15.31:53210) 2023-10-30T13:43:06.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 -- 172.21.15.31:0/790965344 learned_addr learned my addr 172.21.15.31:0/790965344 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:06.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755272640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97500c42a0 0x7f975015c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:06.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9754a71640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c4a10 0x7f975015b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:06.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 -- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c4a10 msgr2=0x7f975015b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c4a10 0x7f975015b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 -- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97500c42a0 msgr2=0x7f975015c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97500c42a0 0x7f975015c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 -- 172.21.15.31:0/790965344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0094d0 con 0x7f9750153c80 2023-10-30T13:43:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9755a73640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f9750155c30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9748007e40 tx=0x7f974800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9745ffb640 1 -- 172.21.15.31:0/790965344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748014620 con 0x7f9750153c80 2023-10-30T13:43:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9745ffb640 1 -- 172.21.15.31:0/790965344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974800ee90 con 0x7f9750153c80 2023-10-30T13:43:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9745ffb640 1 -- 172.21.15.31:0/790965344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97480149e0 con 0x7f9750153c80 2023-10-30T13:43:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97500ce590 con 0x7f9750153c80 2023-10-30T13:43:06.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97500ceb50 con 0x7f9750153c80 2023-10-30T13:43:06.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9750155950 con 0x7f9750153c80 2023-10-30T13:43:06.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.416+0000 7f9745ffb640 1 -- 172.21.15.31:0/790965344 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9748014b40 con 0x7f9750153c80 2023-10-30T13:43:06.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.420+0000 7f9745ffb640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97340664d0 0x7f9734068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:06.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.420+0000 7f9745ffb640 1 -- 172.21.15.31:0/790965344 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f97480848d0 con 0x7f9750153c80 2023-10-30T13:43:06.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.420+0000 7f9755272640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97340664d0 0x7f9734068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:06.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.420+0000 7f9755272640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97340664d0 0x7f9734068990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f974c0b2000 tx=0x7f974c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:06.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.424+0000 7f9745ffb640 1 -- 172.21.15.31:0/790965344 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9750155950 con 0x7f9750153c80 2023-10-30T13:43:06.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.612+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9750155950 con 0x7f9750153c80 2023-10-30T13:43:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.612+0000 7f9745ffb640 1 -- 172.21.15.31:0/790965344 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f9750155950 con 0x7f9750153c80 2023-10-30T13:43:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:06.615 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:43:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97340664d0 msgr2=0x7f9734068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97340664d0 0x7f9734068990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f974c0b2000 tx=0x7f974c009040 comp rx=0 tx=0).stop 2023-10-30T13:43:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 msgr2=0x7f9750155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f9750155c30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9748007e40 tx=0x7f974800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 shutdown_connections 2023-10-30T13:43:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f97340664d0 0x7f9734068990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9750153c80 0x7f9750155c30 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f97500c4a10 0x7f975015b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 --2- 172.21.15.31:0/790965344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97500c42a0 0x7f975015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 >> 172.21.15.31:0/790965344 conn(0x7f97500715c0 msgr2=0x7f9750157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 shutdown_connections 2023-10-30T13:43:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:06.616+0000 7f97574d6640 1 -- 172.21.15.31:0/790965344 wait complete. 2023-10-30T13:43:06.635 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:08.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:08.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.788+0000 7f4b5586b640 1 Processor -- start 2023-10-30T13:43:08.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.788+0000 7f4b5586b640 1 -- start start 2023-10-30T13:43:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b5586b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4b50153c10 0x7f4b50065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b5586b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5006e390 con 0x7f4b50159970 2023-10-30T13:43:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b5586b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b500660e0 con 0x7f4b50070c30 2023-10-30T13:43:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b5586b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b50072390 con 0x7f4b50153c10 2023-10-30T13:43:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b4effd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4b50153c10 0x7f4b50065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:08.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b4f7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f4b50159970 0x7f4b50159dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52236/0 (socket says 172.21.15.31:52236) 2023-10-30T13:43:08.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b4f7fe640 1 -- 172.21.15.31:0/2280688338 learned_addr learned my addr 172.21.15.31:0/2280688338 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:08.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b4effd640 1 -- 172.21.15.31:0/2280688338 >> v1:172.21.15.33:6789/0 conn(0x7f4b50070c30 legacy=0x7f4b50071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:08.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b4effd640 1 -- 172.21.15.31:0/2280688338 >> v1:172.21.15.31:6789/0 conn(0x7f4b50159970 legacy=0x7f4b50159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:08.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b4effd640 1 -- 172.21.15.31:0/2280688338 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b500bec10 con 0x7f4b50153c10 2023-10-30T13:43:08.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.792+0000 7f4b4e7fc640 1 -- 172.21.15.31:0/2280688338 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944065293 0 0) 0x7f4b5006e390 con 0x7f4b50159970 2023-10-30T13:43:08.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b4effd640 1 --2- 172.21.15.31:0/2280688338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50153c10 0x7f4b50065ba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4b400097c0 tx=0x7f4b4001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7331264372cb586a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:08.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b4e7fc640 1 -- 172.21.15.31:0/2280688338 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b400ab600 con 0x7f4b50153c10 2023-10-30T13:43:08.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b4e7fc640 1 -- 172.21.15.31:0/2280688338 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b400a7050 con 0x7f4b50153c10 2023-10-30T13:43:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b4e7fc640 1 -- 172.21.15.31:0/2280688338 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b400ab980 con 0x7f4b50153c10 2023-10-30T13:43:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 -- 172.21.15.31:0/2280688338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50153c10 msgr2=0x7f4b50065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 --2- 172.21.15.31:0/2280688338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50153c10 0x7f4b50065ba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4b400097c0 tx=0x7f4b4001c3f0 comp rx=0 tx=0).stop 2023-10-30T13:43:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 -- 172.21.15.31:0/2280688338 shutdown_connections 2023-10-30T13:43:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 --2- 172.21.15.31:0/2280688338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50153c10 0x7f4b50065ba0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 -- 172.21.15.31:0/2280688338 >> 172.21.15.31:0/2280688338 conn(0x7f4b500c7f60 msgr2=0x7f4b500c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 -- 172.21.15.31:0/2280688338 shutdown_connections 2023-10-30T13:43:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 -- 172.21.15.31:0/2280688338 wait complete. 2023-10-30T13:43:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 Processor -- start 2023-10-30T13:43:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.796+0000 7f4b5586b640 1 -- start start 2023-10-30T13:43:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50070c30 0x7f4b501f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b50153c10 0x7f4b500c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b50159970 0x7f4b5014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b501f5eb0 con 0x7f4b50070c30 2023-10-30T13:43:08.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b501f2390 con 0x7f4b50159970 2023-10-30T13:43:08.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5014cd10 con 0x7f4b50153c10 2023-10-30T13:43:08.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50070c30 0x7f4b501f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:08.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50070c30 0x7f4b501f5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53224/0 (socket says 172.21.15.31:53224) 2023-10-30T13:43:08.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 -- 172.21.15.31:0/272440543 learned_addr learned my addr 172.21.15.31:0/272440543 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b47fff640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b50153c10 0x7f4b500c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4f7fe640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b50159970 0x7f4b5014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 -- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b50153c10 msgr2=0x7f4b500c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b50153c10 0x7f4b500c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 -- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b50159970 msgr2=0x7f4b5014d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b50159970 0x7f4b5014d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 -- 172.21.15.31:0/272440543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40009420 con 0x7f4b50070c30 2023-10-30T13:43:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4f7fe640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b50159970 0x7f4b5014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:43:08.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4effd640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50070c30 0x7f4b501f5970 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b400ae040 tx=0x7f4b4001ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:08.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4cff9640 1 -- 172.21.15.31:0/272440543 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b400b7650 con 0x7f4b50070c30 2023-10-30T13:43:08.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4cff9640 1 -- 172.21.15.31:0/272440543 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b400b77b0 con 0x7f4b50070c30 2023-10-30T13:43:08.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4cff9640 1 -- 172.21.15.31:0/272440543 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b400b7b60 con 0x7f4b50070c30 2023-10-30T13:43:08.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5014cff0 con 0x7f4b50070c30 2023-10-30T13:43:08.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b501f2020 con 0x7f4b50070c30 2023-10-30T13:43:08.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5004e170 con 0x7f4b50070c30 2023-10-30T13:43:08.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.800+0000 7f4b4cff9640 1 -- 172.21.15.31:0/272440543 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b400a4d50 con 0x7f4b50070c30 2023-10-30T13:43:08.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.804+0000 7f4b4cff9640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b300664a0 0x7f4b30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:08.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.804+0000 7f4b4cff9640 1 -- 172.21.15.31:0/272440543 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f4b4011e710 con 0x7f4b50070c30 2023-10-30T13:43:08.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.804+0000 7f4b47fff640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b300664a0 0x7f4b30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:08.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.804+0000 7f4b47fff640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b300664a0 0x7f4b30068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4b500729c0 tx=0x7f4b38009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:08.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.808+0000 7f4b4cff9640 1 -- 172.21.15.31:0/272440543 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b400eba10 con 0x7f4b50070c30 2023-10-30T13:43:08.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.996+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b5014bfa0 con 0x7f4b50070c30 2023-10-30T13:43:08.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:08.996+0000 7f4b4cff9640 1 -- 172.21.15.31:0/272440543 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f4b400ef030 con 0x7f4b50070c30 2023-10-30T13:43:08.999 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:08.999 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:09.000 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:43:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b300664a0 msgr2=0x7f4b30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b300664a0 0x7f4b30068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4b500729c0 tx=0x7f4b38009040 comp rx=0 tx=0).stop 2023-10-30T13:43:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50070c30 msgr2=0x7f4b501f5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50070c30 0x7f4b501f5970 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b400ae040 tx=0x7f4b4001ac30 comp rx=0 tx=0).stop 2023-10-30T13:43:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 shutdown_connections 2023-10-30T13:43:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b300664a0 0x7f4b30068960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b50159970 0x7f4b5014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b50153c10 0x7f4b500c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 --2- 172.21.15.31:0/272440543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b50070c30 0x7f4b501f5970 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 >> 172.21.15.31:0/272440543 conn(0x7f4b500c7f60 msgr2=0x7f4b50157c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 shutdown_connections 2023-10-30T13:43:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:09.000+0000 7f4b5586b640 1 -- 172.21.15.31:0/272440543 wait complete. 2023-10-30T13:43:09.020 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:43:09.020 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4282": { "addr": "172.21.15.33:6839/2419808617", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 2419808617, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 2419808617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5082": { "addr": "172.21.15.31:6837/1215843283", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1215843283, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1215843283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5082, "incarnation": 31, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:37.134913+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5082 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6836/210349556", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 210349556, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 210349556, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 45, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5470, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:43:11.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.172+0000 7f416c19d640 1 Processor -- start 2023-10-30T13:43:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416c19d640 1 -- start start 2023-10-30T13:43:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416c19d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f41641537f0 0x7f41640701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416c19d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41640c2aa0 con 0x7f416406b2f0 2023-10-30T13:43:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416c19d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41640708e0 con 0x7f416415a2e0 2023-10-30T13:43:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416c19d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41640c3ab0 con 0x7f41641537f0 2023-10-30T13:43:11.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f416406b2f0 0x7f416406b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52250/0 (socket says 172.21.15.31:52250) 2023-10-30T13:43:11.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 -- 172.21.15.31:0/3450967934 learned_addr learned my addr 172.21.15.31:0/3450967934 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:11.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4169f39640 1 --2- 172.21.15.31:0/3450967934 >> v2:172.21.15.33:3300/0 conn(0x7f41641537f0 0x7f41640701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62429428 0 0) 0x7f41640c2aa0 con 0x7f416406b2f0 2023-10-30T13:43:11.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4150003530 con 0x7f416406b2f0 2023-10-30T13:43:11.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168306972 0 0) 0x7f4150003530 con 0x7f416406b2f0 2023-10-30T13:43:11.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 >> v2:172.21.15.33:3300/0 conn(0x7f41641537f0 msgr2=0x7f41640701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:11.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 --2- 172.21.15.31:0/3450967934 >> v2:172.21.15.33:3300/0 conn(0x7f41641537f0 0x7f41640701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 >> v1:172.21.15.33:6789/0 conn(0x7f416415a2e0 legacy=0x7f41640ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:11.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41640c1eb0 con 0x7f416406b2f0 2023-10-30T13:43:11.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f415c001a00 con 0x7f416406b2f0 2023-10-30T13:43:11.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 >> v1:172.21.15.31:6789/0 conn(0x7f416406b2f0 legacy=0x7f416406b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:11.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4150003980 0x7f4150003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41500047d0 0x7f4150004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4150005190 0x7f415000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415000c010 con 0x7f4150005190 2023-10-30T13:43:11.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415000c180 con 0x7f41500047d0 2023-10-30T13:43:11.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415000c340 con 0x7f4150003980 2023-10-30T13:43:11.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4150005190 0x7f415000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4169738640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41500047d0 0x7f4150004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4169f39640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4150003980 0x7f4150003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 -- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4150003980 msgr2=0x7f4150003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:11.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4150003980 0x7f4150003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 -- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41500047d0 msgr2=0x7f4150004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:11.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41500047d0 0x7f4150004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f416a73a640 1 -- 172.21.15.31:0/3450967934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4150003530 con 0x7f4150005190 2023-10-30T13:43:11.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.176+0000 7f4169f39640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4150003980 0x7f4150003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:11.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416a73a640 1 -- 172.21.15.31:0/3450967934 reap_dead start 2023-10-30T13:43:11.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416a73a640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4150005190 0x7f415000b990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f415c00a820 tx=0x7f415c01d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b9491a8ffb6ad5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:11.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c0072d0 con 0x7f4150005190 2023-10-30T13:43:11.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415c0a8070 con 0x7f4150005190 2023-10-30T13:43:11.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f4168f37640 1 -- 172.21.15.31:0/3450967934 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c0076b0 con 0x7f4150005190 2023-10-30T13:43:11.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416c19d640 1 -- 172.21.15.31:0/3450967934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4150005190 msgr2=0x7f415000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:11.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416c19d640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4150005190 0x7f415000b990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f415c00a820 tx=0x7f415c01d320 comp rx=0 tx=0).stop 2023-10-30T13:43:11.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416c19d640 1 -- 172.21.15.31:0/3450967934 shutdown_connections 2023-10-30T13:43:11.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416c19d640 1 --2- 172.21.15.31:0/3450967934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4150005190 0x7f415000b990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416c19d640 1 -- 172.21.15.31:0/3450967934 >> 172.21.15.31:0/3450967934 conn(0x7f4164071fb0 msgr2=0x7f41640723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:11.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416c19d640 1 -- 172.21.15.31:0/3450967934 shutdown_connections 2023-10-30T13:43:11.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.180+0000 7f416c19d640 1 -- 172.21.15.31:0/3450967934 wait complete. 2023-10-30T13:43:11.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 Processor -- start 2023-10-30T13:43:11.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 -- start start 2023-10-30T13:43:11.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4150005190 0x7f4164158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416414f7f0 0x7f416414ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f41641562d0 0x7f4164156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641ef100 con 0x7f416414f7f0 2023-10-30T13:43:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641ee830 con 0x7f4150005190 2023-10-30T13:43:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416414c430 con 0x7f41641562d0 2023-10-30T13:43:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416414f7f0 0x7f416414ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416414f7f0 0x7f416414ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53238/0 (socket says 172.21.15.31:53238) 2023-10-30T13:43:11.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 -- 172.21.15.31:0/688737367 learned_addr learned my addr 172.21.15.31:0/688737367 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:11.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169f39640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4150005190 0x7f4164158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416a73a640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f41641562d0 0x7f4164156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 -- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f41641562d0 msgr2=0x7f4164156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f41641562d0 0x7f4164156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 -- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4150005190 msgr2=0x7f4164158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4150005190 0x7f4164158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 -- 172.21.15.31:0/688737367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416414c6d0 con 0x7f416414f7f0 2023-10-30T13:43:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169f39640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4150005190 0x7f4164158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f4169738640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416414f7f0 0x7f416414ed50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f415400f620 tx=0x7f415400fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f415a7fc640 1 -- 172.21.15.31:0/688737367 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41540052e0 con 0x7f416414f7f0 2023-10-30T13:43:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f415a7fc640 1 -- 172.21.15.31:0/688737367 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4154005460 con 0x7f416414f7f0 2023-10-30T13:43:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f415a7fc640 1 -- 172.21.15.31:0/688737367 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154003840 con 0x7f416414f7f0 2023-10-30T13:43:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41641eeab0 con 0x7f416414f7f0 2023-10-30T13:43:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.184+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4164156c90 con 0x7f416414f7f0 2023-10-30T13:43:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.188+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416404e170 con 0x7f416414f7f0 2023-10-30T13:43:11.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.188+0000 7f415a7fc640 1 -- 172.21.15.31:0/688737367 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41540039a0 con 0x7f416414f7f0 2023-10-30T13:43:11.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.188+0000 7f415a7fc640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41480664a0 0x7f4148068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:11.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.188+0000 7f415a7fc640 1 -- 172.21.15.31:0/688737367 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f4154083f80 con 0x7f416414f7f0 2023-10-30T13:43:11.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.188+0000 7f4169f39640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41480664a0 0x7f4148068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:11.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.188+0000 7f4169f39640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41480664a0 0x7f4148068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4160009a70 tx=0x7f4160008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:11.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.192+0000 7f415a7fc640 1 -- 172.21.15.31:0/688737367 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41540511f0 con 0x7f416414f7f0 2023-10-30T13:43:11.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.380+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4164151340 con 0x7f416414f7f0 2023-10-30T13:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f415a7fc640 1 -- 172.21.15.31:0/688737367 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6509 (secure 0 0 0) 0x7f4154054810 con 0x7f416414f7f0 2023-10-30T13:43:11.384 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:11.384 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:42:37.134913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4282,"mds_1":5082},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5082":{"gid":5082,"name":"a","rank":1,"incarnation":31,"state":"up:stopping","state_seq":31,"addr":"172.21.15.31:6837/1215843283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1215843283},{"type":"v1","addr":"172.21.15.31:6837","nonce":1215843283}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2023-10-30T13:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41480664a0 msgr2=0x7f4148068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41480664a0 0x7f4148068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4160009a70 tx=0x7f4160008040 comp rx=0 tx=0).stop 2023-10-30T13:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416414f7f0 msgr2=0x7f416414ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416414f7f0 0x7f416414ed50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f415400f620 tx=0x7f415400fae0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 shutdown_connections 2023-10-30T13:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41480664a0 0x7f4148068960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f41641562d0 0x7f4164156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416414f7f0 0x7f416414ed50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 --2- 172.21.15.31:0/688737367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4150005190 0x7f4164158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 >> 172.21.15.31:0/688737367 conn(0x7f4164071fb0 msgr2=0x7f4164150390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 shutdown_connections 2023-10-30T13:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:11.384+0000 7f416c19d640 1 -- 172.21.15.31:0/688737367 wait complete. 2023-10-30T13:43:11.405 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:12.964 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2023-10-30T13:43:13.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9371794640 1 Processor -- start 2023-10-30T13:43:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9371794640 1 -- start start 2023-10-30T13:43:13.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9371794640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0cb7d0 0x7f936c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9371794640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f936c0c3e40 0x7f936c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9371794640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936c0c2310 con 0x7f936c150830 2023-10-30T13:43:13.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9371794640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936c06bc10 con 0x7f936c0c3e40 2023-10-30T13:43:13.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9371794640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c156710 con 0x7f936c0cb7d0 2023-10-30T13:43:13.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f936c150830 0x7f936c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52260/0 (socket says 172.21.15.31:52260) 2023-10-30T13:43:13.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936affd640 1 -- 172.21.15.31:0/609548194 learned_addr learned my addr 172.21.15.31:0/609548194 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936b7fe640 1 --2- 172.21.15.31:0/609548194 >> v2:172.21.15.31:3300/0 conn(0x7f936c0c3e40 0x7f936c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9363fff640 1 --2- 172.21.15.31:0/609548194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0cb7d0 0x7f936c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936a7fc640 1 -- 172.21.15.31:0/609548194 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559761081 0 0) 0x7f936c0c2310 con 0x7f936c150830 2023-10-30T13:43:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936a7fc640 1 -- 172.21.15.31:0/609548194 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9358003530 con 0x7f936c150830 2023-10-30T13:43:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936b7fe640 1 -- 172.21.15.31:0/609548194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0cb7d0 msgr2=0x7f936c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936b7fe640 1 --2- 172.21.15.31:0/609548194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0cb7d0 0x7f936c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936b7fe640 1 -- 172.21.15.31:0/609548194 >> v1:172.21.15.31:6789/0 conn(0x7f936c150830 legacy=0x7f936c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f936b7fe640 1 -- 172.21.15.31:0/609548194 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c0c4360 con 0x7f936c0c3e40 2023-10-30T13:43:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.560+0000 7f9363fff640 1 --2- 172.21.15.31:0/609548194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0cb7d0 0x7f936c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:13.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f936b7fe640 1 --2- 172.21.15.31:0/609548194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c0c3e40 0x7f936c070520 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9364009d90 tx=0x7f936401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a821a9a9d41416 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:13.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f936a7fc640 1 -- 172.21.15.31:0/609548194 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640af600 con 0x7f936c0c3e40 2023-10-30T13:43:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f936a7fc640 1 -- 172.21.15.31:0/609548194 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93640ab050 con 0x7f936c0c3e40 2023-10-30T13:43:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f936a7fc640 1 -- 172.21.15.31:0/609548194 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640af980 con 0x7f936c0c3e40 2023-10-30T13:43:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 -- 172.21.15.31:0/609548194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c0c3e40 msgr2=0x7f936c070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 --2- 172.21.15.31:0/609548194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c0c3e40 0x7f936c070520 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9364009d90 tx=0x7f936401f130 comp rx=0 tx=0).stop 2023-10-30T13:43:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 -- 172.21.15.31:0/609548194 shutdown_connections 2023-10-30T13:43:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 --2- 172.21.15.31:0/609548194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c0c3e40 0x7f936c070520 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 --2- 172.21.15.31:0/609548194 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0cb7d0 0x7f936c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 -- 172.21.15.31:0/609548194 >> 172.21.15.31:0/609548194 conn(0x7f936c0cac70 msgr2=0x7f936c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 -- 172.21.15.31:0/609548194 shutdown_connections 2023-10-30T13:43:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.564+0000 7f9371794640 1 -- 172.21.15.31:0/609548194 wait complete. 2023-10-30T13:43:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 Processor -- start 2023-10-30T13:43:13.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 -- start start 2023-10-30T13:43:13.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0c3e40 0x7f936c070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f936c0cb7d0 0x7f936c1ec9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c150830 0x7f936c1ecf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c1ec4f0 con 0x7f936c150830 2023-10-30T13:43:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c1f0490 con 0x7f936c0cb7d0 2023-10-30T13:43:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c1e0e60 con 0x7f936c0c3e40 2023-10-30T13:43:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c150830 0x7f936c1ecf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c150830 0x7f936c1ecf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53288/0 (socket says 172.21.15.31:53288) 2023-10-30T13:43:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 -- 172.21.15.31:0/3646215568 learned_addr learned my addr 172.21.15.31:0/3646215568 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:13.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9363fff640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f936c0cb7d0 0x7f936c1ec9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936affd640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0c3e40 0x7f936c070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 -- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0c3e40 msgr2=0x7f936c070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0c3e40 0x7f936c070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 -- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f936c0cb7d0 msgr2=0x7f936c1ec9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f936c0cb7d0 0x7f936c1ec9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 -- 172.21.15.31:0/3646215568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9364009a70 con 0x7f936c150830 2023-10-30T13:43:13.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f936b7fe640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c150830 0x7f936c1ecf10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f936401cd60 tx=0x7f936401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:13.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f93637fe640 1 -- 172.21.15.31:0/3646215568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640afca0 con 0x7f936c150830 2023-10-30T13:43:13.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f93637fe640 1 -- 172.21.15.31:0/3646215568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93640afe00 con 0x7f936c150830 2023-10-30T13:43:13.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f93637fe640 1 -- 172.21.15.31:0/3646215568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640ba660 con 0x7f936c150830 2023-10-30T13:43:13.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c1e1140 con 0x7f936c150830 2023-10-30T13:43:13.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.568+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c15a980 con 0x7f936c150830 2023-10-30T13:43:13.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.572+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936c04e170 con 0x7f936c150830 2023-10-30T13:43:13.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.572+0000 7f93637fe640 1 -- 172.21.15.31:0/3646215568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93640ba7c0 con 0x7f936c150830 2023-10-30T13:43:13.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.572+0000 7f93637fe640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f934c0664a0 0x7f934c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.572+0000 7f93637fe640 1 -- 172.21.15.31:0/3646215568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f9364122910 con 0x7f936c150830 2023-10-30T13:43:13.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.572+0000 7f936affd640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f934c0664a0 0x7f934c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.572+0000 7f936affd640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f934c0664a0 0x7f934c068960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f935c007fd0 tx=0x7f935c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:13.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.576+0000 7f93637fe640 1 -- 172.21.15.31:0/3646215568 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93640efc30 con 0x7f936c150830 2023-10-30T13:43:13.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f936c070300 con 0x7f936c150830 2023-10-30T13:43:13.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f93637fe640 1 -- 172.21.15.31:0/3646215568 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5700 (secure 0 0 0) 0x7f93640b4090 con 0x7f936c150830 2023-10-30T13:43:13.769 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:13.770 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:13.770 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2023-10-30T13:43:13.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f934c0664a0 msgr2=0x7f934c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f934c0664a0 0x7f934c068960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f935c007fd0 tx=0x7f935c009040 comp rx=0 tx=0).stop 2023-10-30T13:43:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c150830 msgr2=0x7f936c1ecf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c150830 0x7f936c1ecf10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f936401cd60 tx=0x7f936401bed0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 shutdown_connections 2023-10-30T13:43:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f934c0664a0 0x7f934c068960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f936c150830 0x7f936c1ecf10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f936c0cb7d0 0x7f936c1ec9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 --2- 172.21.15.31:0/3646215568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f936c0c3e40 0x7f936c070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 >> 172.21.15.31:0/3646215568 conn(0x7f936c0cac70 msgr2=0x7f936c14eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 shutdown_connections 2023-10-30T13:43:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.768+0000 7f9371794640 1 -- 172.21.15.31:0/3646215568 wait complete. 2023-10-30T13:43:13.792 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 1 alive and active, now stable! 2023-10-30T13:43:13.792 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2023-10-30T13:43:13.792 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T13:43:13.793 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:43:13.817 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T13:43:13.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:13.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.960+0000 7fe71a017640 1 Processor -- start 2023-10-30T13:43:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.960+0000 7fe71a017640 1 -- start start 2023-10-30T13:43:13.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe71a017640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe7140701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe71a017640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe71406b2f0 0x7fe71406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe71a017640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7140c2aa0 con 0x7fe71415a2e0 2023-10-30T13:43:13.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe71a017640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140c3ab0 con 0x7fe71406b2f0 2023-10-30T13:43:13.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe71a017640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140706e0 con 0x7fe7141537f0 2023-10-30T13:43:13.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe71406b2f0 0x7fe71406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe71406b2f0 0x7fe71406b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53296/0 (socket says 172.21.15.31:53296) 2023-10-30T13:43:13.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 -- 172.21.15.31:0/1145126755 learned_addr learned my addr 172.21.15.31:0/1145126755 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:13.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe7137fe640 1 --2- 172.21.15.31:0/1145126755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe7140701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 -- 172.21.15.31:0/1145126755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 msgr2=0x7fe7140701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:13.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 --2- 172.21.15.31:0/1145126755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe7140701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 -- 172.21.15.31:0/1145126755 >> v1:172.21.15.33:6789/0 conn(0x7fe71415a2e0 legacy=0x7fe7140ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 -- 172.21.15.31:0/1145126755 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71406bca0 con 0x7fe71406b2f0 2023-10-30T13:43:13.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.964+0000 7fe713fff640 1 --2- 172.21.15.31:0/1145126755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71406b2f0 0x7fe71406b760 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009e50 tx=0x7fe70c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67e33e9d51806 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:13.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe7127fc640 1 -- 172.21.15.31:0/1145126755 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0af400 con 0x7fe71406b2f0 2023-10-30T13:43:13.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe7127fc640 1 -- 172.21.15.31:0/1145126755 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c0ab050 con 0x7fe71406b2f0 2023-10-30T13:43:13.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe7127fc640 1 -- 172.21.15.31:0/1145126755 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0af780 con 0x7fe71406b2f0 2023-10-30T13:43:13.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- 172.21.15.31:0/1145126755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71406b2f0 msgr2=0x7fe71406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 --2- 172.21.15.31:0/1145126755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71406b2f0 0x7fe71406b760 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009e50 tx=0x7fe70c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- 172.21.15.31:0/1145126755 shutdown_connections 2023-10-30T13:43:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 --2- 172.21.15.31:0/1145126755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71406b2f0 0x7fe71406b760 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 --2- 172.21.15.31:0/1145126755 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe7140701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- 172.21.15.31:0/1145126755 >> 172.21.15.31:0/1145126755 conn(0x7fe714071fb0 msgr2=0x7fe7140723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- 172.21.15.31:0/1145126755 shutdown_connections 2023-10-30T13:43:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- 172.21.15.31:0/1145126755 wait complete. 2023-10-30T13:43:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 Processor -- start 2023-10-30T13:43:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- start start 2023-10-30T13:43:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe71406b2f0 0x7fe7141596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe71415d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71415a2e0 0x7fe71415d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141f10e0 con 0x7fe71415a2e0 2023-10-30T13:43:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141f0210 con 0x7fe71406b2f0 2023-10-30T13:43:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe71a017640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141f0460 con 0x7fe7141537f0 2023-10-30T13:43:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe713fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71415a2e0 0x7fe71415d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe713fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71415a2e0 0x7fe71415d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53308/0 (socket says 172.21.15.31:53308) 2023-10-30T13:43:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe713fff640 1 -- 172.21.15.31:0/4102386457 learned_addr learned my addr 172.21.15.31:0/4102386457 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe7137fe640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe71406b2f0 0x7fe7141596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.968+0000 7fe712ffd640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe71415d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe713fff640 1 -- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 msgr2=0x7fe71415d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe713fff640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe71415d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe713fff640 1 -- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe71406b2f0 msgr2=0x7fe7141596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe713fff640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe71406b2f0 0x7fe7141596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe713fff640 1 -- 172.21.15.31:0/4102386457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c009b30 con 0x7fe71415a2e0 2023-10-30T13:43:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe713fff640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71415a2e0 0x7fe71415d910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe70c01ce20 tx=0x7fe70c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe6fbfff640 1 -- 172.21.15.31:0/4102386457 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0c0070 con 0x7fe71415a2e0 2023-10-30T13:43:13.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe6fbfff640 1 -- 172.21.15.31:0/4102386457 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c009510 con 0x7fe71415a2e0 2023-10-30T13:43:13.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe6fbfff640 1 -- 172.21.15.31:0/4102386457 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0afe40 con 0x7fe71415a2e0 2023-10-30T13:43:13.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7141f0740 con 0x7fe71415a2e0 2023-10-30T13:43:13.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7141f0d00 con 0x7fe71415a2e0 2023-10-30T13:43:13.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7141f1220 con 0x7fe71415a2e0 2023-10-30T13:43:13.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe6fbfff640 1 -- 172.21.15.31:0/4102386457 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe70c0b6040 con 0x7fe71415a2e0 2023-10-30T13:43:13.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe6fbfff640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7000664a0 0x7fe700068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:13.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.972+0000 7fe6fbfff640 1 -- 172.21.15.31:0/4102386457 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fe70c121b80 con 0x7fe71415a2e0 2023-10-30T13:43:13.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.976+0000 7fe7137fe640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7000664a0 0x7fe700068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:13.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.976+0000 7fe7137fe640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7000664a0 0x7fe700068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe70400b9d0 tx=0x7fe704008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:13.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:13.976+0000 7fe6fbfff640 1 -- 172.21.15.31:0/4102386457 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7141f1220 con 0x7fe71415a2e0 2023-10-30T13:43:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.176+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7141f1220 con 0x7fe71415a2e0 2023-10-30T13:43:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe6fbfff640 1 -- 172.21.15.31:0/4102386457 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7fe7141f1220 con 0x7fe71415a2e0 2023-10-30T13:43:14.180 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:14.180 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:14.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7000664a0 msgr2=0x7fe700068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:14.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7000664a0 0x7fe700068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe70400b9d0 tx=0x7fe704008040 comp rx=0 tx=0).stop 2023-10-30T13:43:14.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71415a2e0 msgr2=0x7fe71415d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71415a2e0 0x7fe71415d910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe70c01ce20 tx=0x7fe70c01bf90 comp rx=0 tx=0).stop 2023-10-30T13:43:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 shutdown_connections 2023-10-30T13:43:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7000664a0 0x7fe700068960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe71415a2e0 0x7fe71415d910 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7141537f0 0x7fe71415d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 --2- 172.21.15.31:0/4102386457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe71406b2f0 0x7fe7141596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 >> 172.21.15.31:0/4102386457 conn(0x7fe714071fb0 msgr2=0x7fe71406c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 shutdown_connections 2023-10-30T13:43:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:14.180+0000 7fe71a017640 1 -- 172.21.15.31:0/4102386457 wait complete. 2023-10-30T13:43:14.202 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:16.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f183b13e640 1 Processor -- start 2023-10-30T13:43:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f183b13e640 1 -- start start 2023-10-30T13:43:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f183b13e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 0x7f183406f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f183b13e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18340ca180 con 0x7f18340cb8b0 2023-10-30T13:43:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f183b13e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18340ca460 con 0x7f18340ca9b0 2023-10-30T13:43:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f183b13e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1834150d00 con 0x7f18340c8060 2023-10-30T13:43:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f1833fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f18340cb8b0 0x7f18340c8730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49254/0 (socket says 172.21.15.31:49254) 2023-10-30T13:43:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f1833fff640 1 -- 172.21.15.31:0/4020157893 learned_addr learned my addr 172.21.15.31:0/4020157893 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f1838eda640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 0x7f183406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688730219 0 0) 0x7f18340ca180 con 0x7f18340cb8b0 2023-10-30T13:43:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.356+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1824003530 con 0x7f18340cb8b0 2023-10-30T13:43:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016859953 0 0) 0x7f18340ca460 con 0x7f18340ca9b0 2023-10-30T13:43:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18340ca180 con 0x7f18340ca9b0 2023-10-30T13:43:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762368492 0 0) 0x7f1824003530 con 0x7f18340cb8b0 2023-10-30T13:43:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 msgr2=0x7f183406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 0x7f183406f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 >> v1:172.21.15.33:6789/0 conn(0x7f18340ca9b0 legacy=0x7f1834160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183406e7b0 con 0x7f18340cb8b0 2023-10-30T13:43:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f1820001680 con 0x7f18340cb8b0 2023-10-30T13:43:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 >> v1:172.21.15.31:6789/0 conn(0x7f18340cb8b0 legacy=0x7f18340c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1824003b40 0x7f1824003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 0x7f18240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1824004d10 0x7f1824005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1834150d00 con 0x7f1824003b40 2023-10-30T13:43:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182400ba40 con 0x7f1824004d10 2023-10-30T13:43:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182400de10 con 0x7f18340c8060 2023-10-30T13:43:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f1838eda640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1824003b40 0x7f1824003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f1833fff640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 0x7f18240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f18396db640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1824004d10 0x7f1824005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f1838eda640 1 -- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 msgr2=0x7f18240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f1838eda640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f18340c8060 0x7f18240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f1838eda640 1 -- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1824004d10 msgr2=0x7f1824005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f1838eda640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1824004d10 0x7f1824005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.360+0000 7f1838eda640 1 -- 172.21.15.31:0/4020157893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824003530 con 0x7f1824003b40 2023-10-30T13:43:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f1838eda640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1824003b40 0x7f1824003f40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1828009ae0 tx=0x7f182801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d5488bf3c7fb1c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18280a9430 con 0x7f1824003b40 2023-10-30T13:43:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18280a9590 con 0x7f1824003b40 2023-10-30T13:43:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f18337fe640 1 -- 172.21.15.31:0/4020157893 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18280a9940 con 0x7f1824003b40 2023-10-30T13:43:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- 172.21.15.31:0/4020157893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1824003b40 msgr2=0x7f1824003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 --2- 172.21.15.31:0/4020157893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1824003b40 0x7f1824003f40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1828009ae0 tx=0x7f182801ca50 comp rx=0 tx=0).stop 2023-10-30T13:43:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f18396db640 1 -- 172.21.15.31:0/4020157893 reap_dead start 2023-10-30T13:43:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- 172.21.15.31:0/4020157893 shutdown_connections 2023-10-30T13:43:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- 172.21.15.31:0/4020157893 >> 172.21.15.31:0/4020157893 conn(0x7f183406afa0 msgr2=0x7f183406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- 172.21.15.31:0/4020157893 shutdown_connections 2023-10-30T13:43:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- 172.21.15.31:0/4020157893 wait complete. 2023-10-30T13:43:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 Processor -- start 2023-10-30T13:43:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- start start 2023-10-30T13:43:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1824003b40 0x7f18341ef280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18341f1d60 0x7f18341ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1834157690 0x7f1834158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1834159250 con 0x7f18341f1d60 2023-10-30T13:43:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18341f22b0 con 0x7f1834157690 2023-10-30T13:43:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.364+0000 7f183b13e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18341ede10 con 0x7f1824003b40 2023-10-30T13:43:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18341f1d60 0x7f18341ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18341f1d60 0x7f18341ee8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42052/0 (socket says 172.21.15.31:42052) 2023-10-30T13:43:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 -- 172.21.15.31:0/1510781315 learned_addr learned my addr 172.21.15.31:0/1510781315 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1838eda640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1824003b40 0x7f18341ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f18396db640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1834157690 0x7f1834158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 -- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1824003b40 msgr2=0x7f18341ef280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1824003b40 0x7f18341ef280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 -- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1834157690 msgr2=0x7f1834158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1834157690 0x7f1834158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 -- 172.21.15.31:0/1510781315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1834158750 con 0x7f18341f1d60 2023-10-30T13:43:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f18396db640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1834157690 0x7f1834158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f1833fff640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18341f1d60 0x7f18341ee8d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f182000e790 tx=0x7f182000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f18317fa640 1 -- 172.21.15.31:0/1510781315 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000eea0 con 0x7f18341f1d60 2023-10-30T13:43:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f18317fa640 1 -- 172.21.15.31:0/1510781315 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182000c8d0 con 0x7f18341f1d60 2023-10-30T13:43:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f18317fa640 1 -- 172.21.15.31:0/1510781315 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000cc80 con 0x7f18341f1d60 2023-10-30T13:43:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1834158a30 con 0x7f18341f1d60 2023-10-30T13:43:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18341ee3b0 con 0x7f18341f1d60 2023-10-30T13:43:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183404e170 con 0x7f18341f1d60 2023-10-30T13:43:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.368+0000 7f18317fa640 1 -- 172.21.15.31:0/1510781315 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f182001d460 con 0x7f18341f1d60 2023-10-30T13:43:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.372+0000 7f18317fa640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18180664d0 0x7f1818068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.372+0000 7f18317fa640 1 -- 172.21.15.31:0/1510781315 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f1820083ed0 con 0x7f18341f1d60 2023-10-30T13:43:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.372+0000 7f1838eda640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18180664d0 0x7f1818068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.372+0000 7f1838eda640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18180664d0 0x7f1818068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18280ae000 tx=0x7f18280046d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.376+0000 7f18317fa640 1 -- 172.21.15.31:0/1510781315 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f18200511d0 con 0x7f18341f1d60 2023-10-30T13:43:16.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18341ee570 con 0x7f18341f1d60 2023-10-30T13:43:16.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f18317fa640 1 -- 172.21.15.31:0/1510781315 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7f18200547f0 con 0x7f18341f1d60 2023-10-30T13:43:16.566 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:16.566 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:16.566 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:16.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18180664d0 msgr2=0x7f1818068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18180664d0 0x7f1818068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18280ae000 tx=0x7f18280046d0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18341f1d60 msgr2=0x7f18341ee8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18341f1d60 0x7f18341ee8d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f182000e790 tx=0x7f182000ec50 comp rx=0 tx=0).stop 2023-10-30T13:43:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 shutdown_connections 2023-10-30T13:43:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f18180664d0 0x7f1818068990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1834157690 0x7f1834158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18341f1d60 0x7f18341ee8d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 --2- 172.21.15.31:0/1510781315 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1824003b40 0x7f18341ef280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 >> 172.21.15.31:0/1510781315 conn(0x7f183406afa0 msgr2=0x7f1834070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.564+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 shutdown_connections 2023-10-30T13:43:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:16.568+0000 7f183b13e640 1 -- 172.21.15.31:0/1510781315 wait complete. 2023-10-30T13:43:16.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:18.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.740+0000 7f4962a46640 1 Processor -- start 2023-10-30T13:43:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4962a46640 1 -- start start 2023-10-30T13:43:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4962a46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f495c153c80 0x7f495c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4962a46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f495c0c42a0 0x7f495c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4962a46640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495c159f20 con 0x7f495c0c4a10 2023-10-30T13:43:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4962a46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c15dec0 con 0x7f495c0c42a0 2023-10-30T13:43:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4962a46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c1599a0 con 0x7f495c153c80 2023-10-30T13:43:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4960fe3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f495c0c42a0 0x7f495c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4960fe3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f495c0c42a0 0x7f495c0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42054/0 (socket says 172.21.15.31:42054) 2023-10-30T13:43:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4960fe3640 1 -- 172.21.15.31:0/3467449726 learned_addr learned my addr 172.21.15.31:0/3467449726 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f495b7fe640 1 --1- 172.21.15.31:0/3467449726 >> v1:172.21.15.31:6789/0 conn(0x7f495c0c4a10 0x7f495c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49268/0 (socket says 172.21.15.31:49268) 2023-10-30T13:43:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f495bfff640 1 --2- 172.21.15.31:0/3467449726 >> v2:172.21.15.33:3300/0 conn(0x7f495c153c80 0x7f495c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4960fe3640 1 -- 172.21.15.31:0/3467449726 >> v2:172.21.15.33:3300/0 conn(0x7f495c153c80 msgr2=0x7f495c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4960fe3640 1 --2- 172.21.15.31:0/3467449726 >> v2:172.21.15.33:3300/0 conn(0x7f495c153c80 0x7f495c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4960fe3640 1 -- 172.21.15.31:0/3467449726 >> v1:172.21.15.31:6789/0 conn(0x7f495c0c4a10 legacy=0x7f495c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:18.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f4960fe3640 1 -- 172.21.15.31:0/3467449726 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c15d620 con 0x7f495c0c42a0 2023-10-30T13:43:18.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.744+0000 7f495affd640 1 -- 172.21.15.31:0/3467449726 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958059478 0 0) 0x7f495c159f20 con 0x7f495c0c4a10 2023-10-30T13:43:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4960fe3640 1 --2- 172.21.15.31:0/3467449726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0ca1e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f495c0d00f0 tx=0x7f495401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d66df1d83e9ed05 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f495affd640 1 -- 172.21.15.31:0/3467449726 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49540af400 con 0x7f495c0c42a0 2023-10-30T13:43:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f495affd640 1 -- 172.21.15.31:0/3467449726 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49540ab050 con 0x7f495c0c42a0 2023-10-30T13:43:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f495affd640 1 -- 172.21.15.31:0/3467449726 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49540af780 con 0x7f495c0c42a0 2023-10-30T13:43:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 -- 172.21.15.31:0/3467449726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 msgr2=0x7f495c0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 --2- 172.21.15.31:0/3467449726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0ca1e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f495c0d00f0 tx=0x7f495401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 -- 172.21.15.31:0/3467449726 shutdown_connections 2023-10-30T13:43:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 --2- 172.21.15.31:0/3467449726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0ca1e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 --2- 172.21.15.31:0/3467449726 >> v2:172.21.15.33:3300/0 conn(0x7f495c153c80 0x7f495c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 -- 172.21.15.31:0/3467449726 >> 172.21.15.31:0/3467449726 conn(0x7f495c0715c0 msgr2=0x7f495c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 -- 172.21.15.31:0/3467449726 shutdown_connections 2023-10-30T13:43:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 -- 172.21.15.31:0/3467449726 wait complete. 2023-10-30T13:43:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.748+0000 7f4962a46640 1 Processor -- start 2023-10-30T13:43:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 -- start start 2023-10-30T13:43:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f495c0c4a10 0x7f495c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f495c153c80 0x7f495c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0ce360 con 0x7f495c0c42a0 2023-10-30T13:43:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c15d210 con 0x7f495c0c4a10 2023-10-30T13:43:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c1efd70 con 0x7f495c153c80 2023-10-30T13:43:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42060/0 (socket says 172.21.15.31:42060) 2023-10-30T13:43:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 -- 172.21.15.31:0/2995942739 learned_addr learned my addr 172.21.15.31:0/2995942739 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495b7fe640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f495c0c4a10 0x7f495c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4960fe3640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f495c153c80 0x7f495c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 -- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f495c153c80 msgr2=0x7f495c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f495c153c80 0x7f495c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 -- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f495c0c4a10 msgr2=0x7f495c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f495c0c4a10 0x7f495c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 -- 172.21.15.31:0/2995942739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4954009b30 con 0x7f495c0c42a0 2023-10-30T13:43:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f495bfff640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0cf610 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f494c00d5f0 tx=0x7f494c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4958ff9640 1 -- 172.21.15.31:0/2995942739 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c0125f0 con 0x7f495c0c42a0 2023-10-30T13:43:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4958ff9640 1 -- 172.21.15.31:0/2995942739 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c012750 con 0x7f495c0c42a0 2023-10-30T13:43:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495c15cc50 con 0x7f495c0c42a0 2023-10-30T13:43:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4958ff9640 1 -- 172.21.15.31:0/2995942739 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c012ae0 con 0x7f495c0c42a0 2023-10-30T13:43:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c0ce540 con 0x7f495c0c42a0 2023-10-30T13:43:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.752+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c04e170 con 0x7f495c0c42a0 2023-10-30T13:43:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.756+0000 7f4958ff9640 1 -- 172.21.15.31:0/2995942739 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f494c012c40 con 0x7f495c0c42a0 2023-10-30T13:43:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.756+0000 7f4958ff9640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f49480664d0 0x7f4948068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.756+0000 7f4958ff9640 1 -- 172.21.15.31:0/2995942739 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f494c0845b0 con 0x7f495c0c42a0 2023-10-30T13:43:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.756+0000 7f495b7fe640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f49480664d0 0x7f4948068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.756+0000 7f495b7fe640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f49480664d0 0x7f4948068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f495c0c6e60 tx=0x7f4950007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.760+0000 7f4958ff9640 1 -- 172.21.15.31:0/2995942739 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f494c051790 con 0x7f495c0c42a0 2023-10-30T13:43:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.948+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f495c0cad40 con 0x7f495c0c42a0 2023-10-30T13:43:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.948+0000 7f4958ff9640 1 -- 172.21.15.31:0/2995942739 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7f494c054db0 con 0x7f495c0c42a0 2023-10-30T13:43:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:18.951 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:18.951 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:18.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f49480664d0 msgr2=0x7f4948068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:18.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f49480664d0 0x7f4948068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f495c0c6e60 tx=0x7f4950007b80 comp rx=0 tx=0).stop 2023-10-30T13:43:18.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 msgr2=0x7f495c0cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:18.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0cf610 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f494c00d5f0 tx=0x7f494c00dab0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 shutdown_connections 2023-10-30T13:43:18.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f49480664d0 0x7f4948068990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f495c153c80 0x7f495c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f495c0c4a10 0x7f495c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 --2- 172.21.15.31:0/2995942739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f495c0c42a0 0x7f495c0cf610 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:18.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 >> 172.21.15.31:0/2995942739 conn(0x7f495c0715c0 msgr2=0x7f495c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 shutdown_connections 2023-10-30T13:43:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:18.952+0000 7f4962a46640 1 -- 172.21.15.31:0/2995942739 wait complete. 2023-10-30T13:43:18.972 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:20.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.124+0000 7ffb723c4640 1 Processor -- start 2023-10-30T13:43:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.124+0000 7ffb723c4640 1 -- start start 2023-10-30T13:43:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.124+0000 7ffb723c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 0x7ffb6c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb723c4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0c2aa0 con 0x7ffb6c1537f0 2023-10-30T13:43:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb723c4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0708e0 con 0x7ffb6c06b2f0 2023-10-30T13:43:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb723c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0c3ab0 con 0x7ffb6c0701a0 2023-10-30T13:43:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ffb6c1537f0 0x7ffb6c0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49278/0 (socket says 172.21.15.31:49278) 2023-10-30T13:43:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 -- 172.21.15.31:0/2292637732 learned_addr learned my addr 172.21.15.31:0/2292637732 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 0x7ffb6c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321497319 0 0) 0x7ffb6c0c2aa0 con 0x7ffb6c1537f0 2023-10-30T13:43:21.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54003530 con 0x7ffb6c1537f0 2023-10-30T13:43:21.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413810924 0 0) 0x7ffb6c0708e0 con 0x7ffb6c06b2f0 2023-10-30T13:43:21.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c0c2aa0 con 0x7ffb6c06b2f0 2023-10-30T13:43:21.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799846055 0 0) 0x7ffb54003530 con 0x7ffb6c1537f0 2023-10-30T13:43:21.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 msgr2=0x7ffb6c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:21.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 0x7ffb6c15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 >> v1:172.21.15.33:6789/0 conn(0x7ffb6c06b2f0 legacy=0x7ffb6c1641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c0c1eb0 con 0x7ffb6c1537f0 2023-10-30T13:43:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7ffb5c001680 con 0x7ffb6c1537f0 2023-10-30T13:43:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 >> v1:172.21.15.31:6789/0 conn(0x7ffb6c1537f0 legacy=0x7ffb6c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb54003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 0x7ffb540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb54004d10 0x7ffb54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5400ba40 con 0x7ffb54003b40 2023-10-30T13:43:21.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5400bbb0 con 0x7ffb54004d10 2023-10-30T13:43:21.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5400bd70 con 0x7ffb6c0701a0 2023-10-30T13:43:21.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb54003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb70961640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb54004d10 0x7ffb54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 0x7ffb540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 -- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 msgr2=0x7ffb540047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:21.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c0701a0 0x7ffb540047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 -- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb54004d10 msgr2=0x7ffb54005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:21.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb54004d10 0x7ffb54005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.128+0000 7ffb6bfff640 1 -- 172.21.15.31:0/2292637732 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb54003530 con 0x7ffb54003b40 2023-10-30T13:43:21.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb6bfff640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb54003f40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c007fa0 tx=0x7ffb5c01cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc74d26e41773489 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:21.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c005ae0 con 0x7ffb54003b40 2023-10-30T13:43:21.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c005c40 con 0x7ffb54003b40 2023-10-30T13:43:21.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb6affd640 1 -- 172.21.15.31:0/2292637732 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0a5800 con 0x7ffb54003b40 2023-10-30T13:43:21.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 -- 172.21.15.31:0/2292637732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 msgr2=0x7ffb54003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:21.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 --2- 172.21.15.31:0/2292637732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb54003f40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c007fa0 tx=0x7ffb5c01cb00 comp rx=0 tx=0).stop 2023-10-30T13:43:21.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb70961640 1 -- 172.21.15.31:0/2292637732 reap_dead start 2023-10-30T13:43:21.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 -- 172.21.15.31:0/2292637732 shutdown_connections 2023-10-30T13:43:21.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 -- 172.21.15.31:0/2292637732 >> 172.21.15.31:0/2292637732 conn(0x7ffb6c071fb0 msgr2=0x7ffb6c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:21.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 -- 172.21.15.31:0/2292637732 shutdown_connections 2023-10-30T13:43:21.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 -- 172.21.15.31:0/2292637732 wait complete. 2023-10-30T13:43:21.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 Processor -- start 2023-10-30T13:43:21.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.132+0000 7ffb723c4640 1 -- start start 2023-10-30T13:43:21.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb6c0c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb6c1f2d80 0x7ffb6c1ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c14f880 0x7ffb6c14c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0c4e40 con 0x7ffb54003b40 2023-10-30T13:43:21.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c158f30 con 0x7ffb6c1f2d80 2023-10-30T13:43:21.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c159160 con 0x7ffb6c14f880 2023-10-30T13:43:21.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb6c0c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb6c0c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42090/0 (socket says 172.21.15.31:42090) 2023-10-30T13:43:21.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 -- 172.21.15.31:0/3453791555 learned_addr learned my addr 172.21.15.31:0/3453791555 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:21.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb70961640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c14f880 0x7ffb6c14c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb6c1f2d80 0x7ffb6c1ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 -- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c14f880 msgr2=0x7ffb6c14c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:21.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c14f880 0x7ffb6c14c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 -- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb6c1f2d80 msgr2=0x7ffb6c1ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:21.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb6c1f2d80 0x7ffb6c1ef760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 -- 172.21.15.31:0/3453791555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c159420 con 0x7ffb54003b40 2023-10-30T13:43:21.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb70961640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c14f880 0x7ffb6c14c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:43:21.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb6bfff640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb6c0c4900 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c01c930 tx=0x7ffb54005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:21.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb68ff9640 1 -- 172.21.15.31:0/3453791555 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0b4070 con 0x7ffb54003b40 2023-10-30T13:43:21.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb68ff9640 1 -- 172.21.15.31:0/3453791555 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c0a5800 con 0x7ffb54003b40 2023-10-30T13:43:21.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb68ff9640 1 -- 172.21.15.31:0/3453791555 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0a5b90 con 0x7ffb54003b40 2023-10-30T13:43:21.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c159700 con 0x7ffb54003b40 2023-10-30T13:43:21.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c1ef240 con 0x7ffb54003b40 2023-10-30T13:43:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c04e170 con 0x7ffb54003b40 2023-10-30T13:43:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.136+0000 7ffb68ff9640 1 -- 172.21.15.31:0/3453791555 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb5c0a5cf0 con 0x7ffb54003b40 2023-10-30T13:43:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.140+0000 7ffb68ff9640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb58066520 0x7ffb580689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.140+0000 7ffb68ff9640 1 -- 172.21.15.31:0/3453791555 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ffb5c11df00 con 0x7ffb54003b40 2023-10-30T13:43:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.140+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb58066520 0x7ffb580689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:21.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.140+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb58066520 0x7ffb580689e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0ca720 tx=0x7ffb60008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:21.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.144+0000 7ffb68ff9640 1 -- 172.21.15.31:0/3453791555 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb5c0eb170 con 0x7ffb54003b40 2023-10-30T13:43:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb6c0c7370 con 0x7ffb54003b40 2023-10-30T13:43:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb68ff9640 1 -- 172.21.15.31:0/3453791555 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7ffb5c0ee790 con 0x7ffb54003b40 2023-10-30T13:43:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:21.335 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:21.335 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb58066520 msgr2=0x7ffb580689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb58066520 0x7ffb580689e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0ca720 tx=0x7ffb60008040 comp rx=0 tx=0).stop 2023-10-30T13:43:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 msgr2=0x7ffb6c0c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb6c0c4900 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c01c930 tx=0x7ffb54005800 comp rx=0 tx=0).stop 2023-10-30T13:43:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 shutdown_connections 2023-10-30T13:43:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb58066520 0x7ffb580689e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb6c14f880 0x7ffb6c14c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb6c1f2d80 0x7ffb6c1ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 --2- 172.21.15.31:0/3453791555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb54003b40 0x7ffb6c0c4900 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.332+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 >> 172.21.15.31:0/3453791555 conn(0x7ffb6c071fb0 msgr2=0x7ffb6c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.336+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 shutdown_connections 2023-10-30T13:43:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:21.336+0000 7ffb723c4640 1 -- 172.21.15.31:0/3453791555 wait complete. 2023-10-30T13:43:21.355 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:23.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.508+0000 7ff167a46640 1 Processor -- start 2023-10-30T13:43:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.508+0000 7ff167a46640 1 -- start start 2023-10-30T13:43:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff167a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff160151840 0x7ff1600c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff167a46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff160151c20 0x7ff160072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff167a46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff1600cb8b0 0x7ff160072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:23.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff167a46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1600c5030 con 0x7ff1600cb8b0 2023-10-30T13:43:23.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff167a46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff160157830 con 0x7ff160151c20 2023-10-30T13:43:23.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff167a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff160157b10 con 0x7ff160151840 2023-10-30T13:43:23.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff1600cb8b0 0x7ff160072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:23.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff1600cb8b0 0x7ff160072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42104/0 (socket says 172.21.15.31:42104) 2023-10-30T13:43:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 -- 172.21.15.31:0/331829655 learned_addr learned my addr 172.21.15.31:0/331829655 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff1657e2640 1 --2- 172.21.15.31:0/331829655 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff160151840 0x7ff1600c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff164fe1640 1 --2- 172.21.15.31:0/331829655 >> v2:172.21.15.33:3300/0 conn(0x7ff160151c20 0x7ff160072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 -- 172.21.15.31:0/331829655 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff160151840 msgr2=0x7ff1600c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 --2- 172.21.15.31:0/331829655 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff160151840 0x7ff1600c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 -- 172.21.15.31:0/331829655 >> v2:172.21.15.33:3300/0 conn(0x7ff160151c20 msgr2=0x7ff160072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 --2- 172.21.15.31:0/331829655 >> v2:172.21.15.33:3300/0 conn(0x7ff160151c20 0x7ff160072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 -- 172.21.15.31:0/331829655 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff160157df0 con 0x7ff1600cb8b0 2023-10-30T13:43:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.512+0000 7ff165fe3640 1 --2- 172.21.15.31:0/331829655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1600cb8b0 0x7ff160072540 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff16006fa70 tx=0x7ff15801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2017cae473b3f61f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff157fff640 1 -- 172.21.15.31:0/331829655 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580af400 con 0x7ff1600cb8b0 2023-10-30T13:43:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff157fff640 1 -- 172.21.15.31:0/331829655 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1580ab050 con 0x7ff1600cb8b0 2023-10-30T13:43:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff157fff640 1 -- 172.21.15.31:0/331829655 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580af780 con 0x7ff1600cb8b0 2023-10-30T13:43:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 -- 172.21.15.31:0/331829655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1600cb8b0 msgr2=0x7ff160072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 --2- 172.21.15.31:0/331829655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1600cb8b0 0x7ff160072540 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff16006fa70 tx=0x7ff15801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 -- 172.21.15.31:0/331829655 shutdown_connections 2023-10-30T13:43:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 --2- 172.21.15.31:0/331829655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1600cb8b0 0x7ff160072540 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 --2- 172.21.15.31:0/331829655 >> v2:172.21.15.33:3300/0 conn(0x7ff160151c20 0x7ff160072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 --2- 172.21.15.31:0/331829655 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff160151840 0x7ff1600c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 -- 172.21.15.31:0/331829655 >> 172.21.15.31:0/331829655 conn(0x7ff1600cac70 msgr2=0x7ff1600cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 -- 172.21.15.31:0/331829655 shutdown_connections 2023-10-30T13:43:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 -- 172.21.15.31:0/331829655 wait complete. 2023-10-30T13:43:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 Processor -- start 2023-10-30T13:43:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 -- start start 2023-10-30T13:43:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1600cb8b0 0x7ff16014c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff160151840 0x7ff160155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff160151c20 0x7ff160145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.516+0000 7ff167a46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1601553b0 con 0x7ff160151c20 2023-10-30T13:43:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff167a46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1601521b0 con 0x7ff160151840 2023-10-30T13:43:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff167a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff160156850 con 0x7ff1600cb8b0 2023-10-30T13:43:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff160151c20 0x7ff160145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff160151c20 0x7ff160145520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42120/0 (socket says 172.21.15.31:42120) 2023-10-30T13:43:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 -- 172.21.15.31:0/3845881376 learned_addr learned my addr 172.21.15.31:0/3845881376 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff1657e2640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1600cb8b0 0x7ff16014c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff164fe1640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff160151840 0x7ff160155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 -- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1600cb8b0 msgr2=0x7ff16014c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1600cb8b0 0x7ff16014c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 -- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff160151840 msgr2=0x7ff160155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff160151840 0x7ff160155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 -- 172.21.15.31:0/3845881376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff158009b30 con 0x7ff160151c20 2023-10-30T13:43:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff164fe1640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff160151840 0x7ff160155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff165fe3640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff160151c20 0x7ff160145520 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1580b2040 tx=0x7ff15801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff155ffb640 1 -- 172.21.15.31:0/3845881376 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580afde0 con 0x7ff160151c20 2023-10-30T13:43:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff155ffb640 1 -- 172.21.15.31:0/3845881376 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff158009510 con 0x7ff160151c20 2023-10-30T13:43:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff155ffb640 1 -- 172.21.15.31:0/3845881376 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580ba660 con 0x7ff160151c20 2023-10-30T13:43:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1601e3960 con 0x7ff160151c20 2023-10-30T13:43:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff160154a90 con 0x7ff160151c20 2023-10-30T13:43:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff160066510 con 0x7ff160151c20 2023-10-30T13:43:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.520+0000 7ff155ffb640 1 -- 172.21.15.31:0/3845881376 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1580b6040 con 0x7ff160151c20 2023-10-30T13:43:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.524+0000 7ff155ffb640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1440664a0 0x7ff144068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.524+0000 7ff155ffb640 1 -- 172.21.15.31:0/3845881376 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ff1581221f0 con 0x7ff160151c20 2023-10-30T13:43:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.524+0000 7ff1657e2640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1440664a0 0x7ff144068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.524+0000 7ff1657e2640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1440664a0 0x7ff144068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0045f0 tx=0x7ff15c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.528+0000 7ff155ffb640 1 -- 172.21.15.31:0/3845881376 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1580ef5b0 con 0x7ff160151c20 2023-10-30T13:43:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.716+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff16004e170 con 0x7ff160151c20 2023-10-30T13:43:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.716+0000 7ff155ffb640 1 -- 172.21.15.31:0/3845881376 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7ff1580f2bd0 con 0x7ff160151c20 2023-10-30T13:43:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:23.718 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.716+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1440664a0 msgr2=0x7ff144068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.716+0000 7ff167a46640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1440664a0 0x7ff144068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0045f0 tx=0x7ff15c008040 comp rx=0 tx=0).stop 2023-10-30T13:43:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.716+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff160151c20 msgr2=0x7ff160145520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.716+0000 7ff167a46640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff160151c20 0x7ff160145520 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1580b2040 tx=0x7ff15801bf60 comp rx=0 tx=0).stop 2023-10-30T13:43:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 shutdown_connections 2023-10-30T13:43:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff1440664a0 0x7ff144068960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff160151c20 0x7ff160145520 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff160151840 0x7ff160155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 --2- 172.21.15.31:0/3845881376 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff1600cb8b0 0x7ff16014c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 >> 172.21.15.31:0/3845881376 conn(0x7ff1600cac70 msgr2=0x7ff160147900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 shutdown_connections 2023-10-30T13:43:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:23.720+0000 7ff167a46640 1 -- 172.21.15.31:0/3845881376 wait complete. 2023-10-30T13:43:23.739 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:25.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.892+0000 7f402ad1d640 1 Processor -- start 2023-10-30T13:43:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.892+0000 7f402ad1d640 1 -- start start 2023-10-30T13:43:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f402ad1d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f4024151c20 0x7f40240c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f402ad1d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40240c5030 con 0x7f40240cb8b0 2023-10-30T13:43:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f402ad1d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4024157830 con 0x7f4024151840 2023-10-30T13:43:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f402ad1d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4024157b10 con 0x7f4024151c20 2023-10-30T13:43:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40292ba640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f40240cb8b0 0x7f402406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55532/0 (socket says 172.21.15.31:55532) 2023-10-30T13:43:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40292ba640 1 -- 172.21.15.31:0/3130929530 learned_addr learned my addr 172.21.15.31:0/3130929530 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4023fff640 1 --2- 172.21.15.31:0/3130929530 >> v2:172.21.15.33:3300/0 conn(0x7f4024151c20 0x7f40240c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824359648 0 0) 0x7f40240c5030 con 0x7f40240cb8b0 2023-10-30T13:43:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4014003530 con 0x7f40240cb8b0 2023-10-30T13:43:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600417513 0 0) 0x7f4014003530 con 0x7f40240cb8b0 2023-10-30T13:43:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 >> v2:172.21.15.33:3300/0 conn(0x7f4024151c20 msgr2=0x7f40240c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 --2- 172.21.15.31:0/3130929530 >> v2:172.21.15.33:3300/0 conn(0x7f4024151c20 0x7f40240c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 >> v1:172.21.15.33:6789/0 conn(0x7f4024151840 legacy=0x7f4024072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4024157df0 con 0x7f40240cb8b0 2023-10-30T13:43:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f401c001a00 con 0x7f40240cb8b0 2023-10-30T13:43:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 >> v1:172.21.15.31:6789/0 conn(0x7f40240cb8b0 legacy=0x7f402406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4014003980 0x7f4014003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40140047d0 0x7f4014004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4014005190 0x7f401400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4024157b10 con 0x7f40140047d0 2023-10-30T13:43:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401400c040 con 0x7f4014005190 2023-10-30T13:43:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401400c200 con 0x7f4014003980 2023-10-30T13:43:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4023fff640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40140047d0 0x7f4014004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40292ba640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4014005190 0x7f401400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4028ab9640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4014003980 0x7f4014003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4023fff640 1 -- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4014003980 msgr2=0x7f4014003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4023fff640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4014003980 0x7f4014003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4023fff640 1 -- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4014005190 msgr2=0x7f401400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4023fff640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4014005190 0x7f401400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f4023fff640 1 -- 172.21.15.31:0/3130929530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4014003530 con 0x7f40140047d0 2023-10-30T13:43:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.896+0000 7f40292ba640 1 -- 172.21.15.31:0/3130929530 reap_dead start 2023-10-30T13:43:25.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f4023fff640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40140047d0 0x7f4014004c50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4010009ae0 tx=0x7f401001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fda504c6c33c488 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:25.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40100a5640 con 0x7f40140047d0 2023-10-30T13:43:25.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40100a57a0 con 0x7f40140047d0 2023-10-30T13:43:25.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f40237fe640 1 -- 172.21.15.31:0/3130929530 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40100a5b50 con 0x7f40140047d0 2023-10-30T13:43:25.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 -- 172.21.15.31:0/3130929530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40140047d0 msgr2=0x7f4014004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:25.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40140047d0 0x7f4014004c50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4010009ae0 tx=0x7f401001ca50 comp rx=0 tx=0).stop 2023-10-30T13:43:25.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 -- 172.21.15.31:0/3130929530 shutdown_connections 2023-10-30T13:43:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 --2- 172.21.15.31:0/3130929530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40140047d0 0x7f4014004c50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 -- 172.21.15.31:0/3130929530 >> 172.21.15.31:0/3130929530 conn(0x7f40240cac70 msgr2=0x7f40240cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 -- 172.21.15.31:0/3130929530 shutdown_connections 2023-10-30T13:43:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 -- 172.21.15.31:0/3130929530 wait complete. 2023-10-30T13:43:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 Processor -- start 2023-10-30T13:43:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.900+0000 7f402ad1d640 1 -- start start 2023-10-30T13:43:25.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40140047d0 0x7f40240717e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40241f2920 0x7f4024153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4024148570 0x7f4024153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40241f2e80 con 0x7f40241f2920 2023-10-30T13:43:25.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40241532b0 con 0x7f4024148570 2023-10-30T13:43:25.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40241508c0 con 0x7f40140047d0 2023-10-30T13:43:25.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40241f2920 0x7f4024153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40241f2920 0x7f4024153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43688/0 (socket says 172.21.15.31:43688) 2023-10-30T13:43:25.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 -- 172.21.15.31:0/84529679 learned_addr learned my addr 172.21.15.31:0/84529679 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:25.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4028ab9640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40140047d0 0x7f40240717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f40292ba640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4024148570 0x7f4024153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 -- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40140047d0 msgr2=0x7f40240717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40140047d0 0x7f40240717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 -- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4024148570 msgr2=0x7f4024153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4024148570 0x7f4024153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 -- 172.21.15.31:0/84529679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4024150ba0 con 0x7f40241f2920 2023-10-30T13:43:25.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f40292ba640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4024148570 0x7f4024153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:25.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f4023fff640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40241f2920 0x7f4024153740 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f401001c880 tx=0x7f40100197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:25.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f40217fa640 1 -- 172.21.15.31:0/84529679 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010004020 con 0x7f40241f2920 2023-10-30T13:43:25.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f40217fa640 1 -- 172.21.15.31:0/84529679 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40100041c0 con 0x7f40241f2920 2023-10-30T13:43:25.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f40217fa640 1 -- 172.21.15.31:0/84529679 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010003620 con 0x7f40241f2920 2023-10-30T13:43:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4024150e80 con 0x7f40241f2920 2023-10-30T13:43:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4024071d20 con 0x7f40241f2920 2023-10-30T13:43:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.904+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4024153080 con 0x7f40241f2920 2023-10-30T13:43:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.908+0000 7f40217fa640 1 -- 172.21.15.31:0/84529679 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40100b4070 con 0x7f40241f2920 2023-10-30T13:43:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.908+0000 7f40217fa640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40080664a0 0x7f4008068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.908+0000 7f40217fa640 1 -- 172.21.15.31:0/84529679 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f401011e8d0 con 0x7f40241f2920 2023-10-30T13:43:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.908+0000 7f4028ab9640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40080664a0 0x7f4008068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.908+0000 7f4028ab9640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40080664a0 0x7f4008068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4018000c10 tx=0x7f4018009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:25.912+0000 7f40217fa640 1 -- 172.21.15.31:0/84529679 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40100ebc80 con 0x7f40241f2920 2023-10-30T13:43:26.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f402404e170 con 0x7f40241f2920 2023-10-30T13:43:26.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f40217fa640 1 -- 172.21.15.31:0/84529679 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7f40100b2090 con 0x7f40241f2920 2023-10-30T13:43:26.105 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:26.106 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:26.106 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:26.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40080664a0 msgr2=0x7f4008068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:26.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40080664a0 0x7f4008068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4018000c10 tx=0x7f4018009040 comp rx=0 tx=0).stop 2023-10-30T13:43:26.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40241f2920 msgr2=0x7f4024153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:26.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40241f2920 0x7f4024153740 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f401001c880 tx=0x7f40100197f0 comp rx=0 tx=0).stop 2023-10-30T13:43:26.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 shutdown_connections 2023-10-30T13:43:26.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40080664a0 0x7f4008068960 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:26.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4024148570 0x7f4024153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40241f2920 0x7f4024153740 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 --2- 172.21.15.31:0/84529679 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40140047d0 0x7f40240717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 >> 172.21.15.31:0/84529679 conn(0x7f40240cac70 msgr2=0x7f4024072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 shutdown_connections 2023-10-30T13:43:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:26.104+0000 7f402ad1d640 1 -- 172.21.15.31:0/84529679 wait complete. 2023-10-30T13:43:26.126 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:26.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:26.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:28.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:28.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.280+0000 7f11441c4640 1 Processor -- start 2023-10-30T13:43:28.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f11441c4640 1 -- start start 2023-10-30T13:43:28.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f11441c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 0x7f113c0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f11441c4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0c2aa0 con 0x7f113c06b2f0 2023-10-30T13:43:28.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f11441c4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0708e0 con 0x7f113c15a2e0 2023-10-30T13:43:28.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f11441c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0c3ab0 con 0x7f113c1537f0 2023-10-30T13:43:28.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1142761640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f113c06b2f0 0x7f113c1641e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55546/0 (socket says 172.21.15.31:55546) 2023-10-30T13:43:28.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1142761640 1 -- 172.21.15.31:0/2170393041 learned_addr learned my addr 172.21.15.31:0/2170393041 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:28.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1141f60640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 0x7f113c0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324574781 0 0) 0x7f113c0c2aa0 con 0x7f113c06b2f0 2023-10-30T13:43:28.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1128003530 con 0x7f113c06b2f0 2023-10-30T13:43:28.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1656572778 0 0) 0x7f1128003530 con 0x7f113c06b2f0 2023-10-30T13:43:28.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 msgr2=0x7f113c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:28.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 0x7f113c0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 >> v1:172.21.15.33:6789/0 conn(0x7f113c15a2e0 legacy=0x7f113c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:28.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0c1eb0 con 0x7f113c06b2f0 2023-10-30T13:43:28.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f1134001a00 con 0x7f113c06b2f0 2023-10-30T13:43:28.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 >> v1:172.21.15.31:6789/0 conn(0x7f113c06b2f0 legacy=0x7f113c1641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:28.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11280039b0 0x7f1128003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1128004360 0x7f1128004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 0x7f1128005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112800b990 con 0x7f1128004360 2023-10-30T13:43:28.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112800bad0 con 0x7f11280039b0 2023-10-30T13:43:28.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112800bc90 con 0x7f113c1537f0 2023-10-30T13:43:28.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f114175f640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1128004360 0x7f1128004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1142761640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 0x7f1128005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f1141f60640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11280039b0 0x7f1128003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f114175f640 1 -- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 msgr2=0x7f1128005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:28.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f114175f640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c1537f0 0x7f1128005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f114175f640 1 -- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11280039b0 msgr2=0x7f1128003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:28.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f114175f640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11280039b0 0x7f1128003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.284+0000 7f114175f640 1 -- 172.21.15.31:0/2170393041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1128003530 con 0x7f1128004360 2023-10-30T13:43:28.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f114175f640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1128004360 0x7f1128004830 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f112c001860 tx=0x7f112c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d32c7c0ccef4d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:28.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0ac640 con 0x7f1128004360 2023-10-30T13:43:28.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c0ac7a0 con 0x7f1128004360 2023-10-30T13:43:28.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f1140f5e640 1 -- 172.21.15.31:0/2170393041 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0acb50 con 0x7f1128004360 2023-10-30T13:43:28.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f11441c4640 1 -- 172.21.15.31:0/2170393041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1128004360 msgr2=0x7f1128004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f11441c4640 1 --2- 172.21.15.31:0/2170393041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1128004360 0x7f1128004830 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f112c001860 tx=0x7f112c01c4e0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f1142761640 1 -- 172.21.15.31:0/2170393041 reap_dead start 2023-10-30T13:43:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f11441c4640 1 -- 172.21.15.31:0/2170393041 shutdown_connections 2023-10-30T13:43:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f11441c4640 1 -- 172.21.15.31:0/2170393041 >> 172.21.15.31:0/2170393041 conn(0x7f113c071fb0 msgr2=0x7f113c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f11441c4640 1 -- 172.21.15.31:0/2170393041 shutdown_connections 2023-10-30T13:43:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.288+0000 7f11441c4640 1 -- 172.21.15.31:0/2170393041 wait complete. 2023-10-30T13:43:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 Processor -- start 2023-10-30T13:43:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 -- start start 2023-10-30T13:43:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1128004360 0x7f113c159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c151a80 0x7f113c1ef200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f113c156040 0x7f113c0c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c155cf0 con 0x7f113c156040 2023-10-30T13:43:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c1568e0 con 0x7f1128004360 2023-10-30T13:43:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c156bc0 con 0x7f113c151a80 2023-10-30T13:43:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f113c156040 0x7f113c0c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f113c156040 0x7f113c0c5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43712/0 (socket says 172.21.15.31:43712) 2023-10-30T13:43:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 -- 172.21.15.31:0/3796422527 learned_addr learned my addr 172.21.15.31:0/3796422527 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f114175f640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c151a80 0x7f113c1ef200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1141f60640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1128004360 0x7f113c159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 -- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c151a80 msgr2=0x7f113c1ef200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c151a80 0x7f113c1ef200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 -- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1128004360 msgr2=0x7f113c159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1128004360 0x7f113c159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 -- 172.21.15.31:0/3796422527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c156ea0 con 0x7f113c156040 2023-10-30T13:43:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f1142761640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f113c156040 0x7f113c0c5830 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f113c06d180 tx=0x7f1134011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11327fc640 1 -- 172.21.15.31:0/3796422527 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113400f920 con 0x7f113c156040 2023-10-30T13:43:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11327fc640 1 -- 172.21.15.31:0/3796422527 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113400faa0 con 0x7f113c156040 2023-10-30T13:43:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11327fc640 1 -- 172.21.15.31:0/3796422527 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113400fe50 con 0x7f113c156040 2023-10-30T13:43:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c1eece0 con 0x7f113c156040 2023-10-30T13:43:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.292+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c159fa0 con 0x7f113c156040 2023-10-30T13:43:28.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.296+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c04e170 con 0x7f113c156040 2023-10-30T13:43:28.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.296+0000 7f11327fc640 1 -- 172.21.15.31:0/3796422527 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f113401f460 con 0x7f113c156040 2023-10-30T13:43:28.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.296+0000 7f11327fc640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f11200664d0 0x7f1120068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:28.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.296+0000 7f11327fc640 1 -- 172.21.15.31:0/3796422527 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f1134087090 con 0x7f113c156040 2023-10-30T13:43:28.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.296+0000 7f1141f60640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f11200664d0 0x7f1120068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:28.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.296+0000 7f1141f60640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f11200664d0 0x7f1120068990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1138009a30 tx=0x7f1138008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:28.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.300+0000 7f11327fc640 1 -- 172.21.15.31:0/3796422527 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1134057e20 con 0x7f113c156040 2023-10-30T13:43:28.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f113c0c2700 con 0x7f113c156040 2023-10-30T13:43:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11327fc640 1 -- 172.21.15.31:0/3796422527 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7f11340579b0 con 0x7f113c156040 2023-10-30T13:43:28.493 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:28.494 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:28.494 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f11200664d0 msgr2=0x7f1120068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f11200664d0 0x7f1120068990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1138009a30 tx=0x7f1138008040 comp rx=0 tx=0).stop 2023-10-30T13:43:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f113c156040 msgr2=0x7f113c0c5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f113c156040 0x7f113c0c5830 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f113c06d180 tx=0x7f1134011c20 comp rx=0 tx=0).stop 2023-10-30T13:43:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 shutdown_connections 2023-10-30T13:43:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f11200664d0 0x7f1120068990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f113c156040 0x7f113c0c5830 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f113c151a80 0x7f113c1ef200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 --2- 172.21.15.31:0/3796422527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1128004360 0x7f113c159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:28.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 >> 172.21.15.31:0/3796422527 conn(0x7f113c071fb0 msgr2=0x7f113c151f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:28.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 shutdown_connections 2023-10-30T13:43:28.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:28.492+0000 7f11441c4640 1 -- 172.21.15.31:0/3796422527 wait complete. 2023-10-30T13:43:28.515 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:30.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:30.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f7945eea640 1 Processor -- start 2023-10-30T13:43:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f7945eea640 1 -- start start 2023-10-30T13:43:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f7945eea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f79401537f0 0x7f79400701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f7945eea640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f794015a2e0 0x7f79400ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f7945eea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79400c2aa0 con 0x7f79400cd4a0 2023-10-30T13:43:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f7945eea640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79400cd8b0 con 0x7f794015a2e0 2023-10-30T13:43:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f7945eea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79400c3ab0 con 0x7f79401537f0 2023-10-30T13:43:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f793effd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f794015a2e0 0x7f79400ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f793ffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f79400cd4a0 0x7f794006b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55562/0 (socket says 172.21.15.31:55562) 2023-10-30T13:43:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.668+0000 7f793ffff640 1 -- 172.21.15.31:0/3525945222 learned_addr learned my addr 172.21.15.31:0/3525945222 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793f7fe640 1 --2- 172.21.15.31:0/3525945222 >> v2:172.21.15.33:3300/0 conn(0x7f79401537f0 0x7f79400701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793effd640 1 -- 172.21.15.31:0/3525945222 >> v2:172.21.15.33:3300/0 conn(0x7f79401537f0 msgr2=0x7f79400701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793effd640 1 --2- 172.21.15.31:0/3525945222 >> v2:172.21.15.33:3300/0 conn(0x7f79401537f0 0x7f79400701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793effd640 1 -- 172.21.15.31:0/3525945222 >> v1:172.21.15.31:6789/0 conn(0x7f79400cd4a0 legacy=0x7f794006b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793effd640 1 -- 172.21.15.31:0/3525945222 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79400708e0 con 0x7f794015a2e0 2023-10-30T13:43:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793e7fc640 1 -- 172.21.15.31:0/3525945222 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208177650 0 0) 0x7f79400c2aa0 con 0x7f79400cd4a0 2023-10-30T13:43:30.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793effd640 1 --2- 172.21.15.31:0/3525945222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794015a2e0 0x7f79400ccf60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f79400cdb70 tx=0x7f793401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6142e18856bf7552 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:30.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793e7fc640 1 -- 172.21.15.31:0/3525945222 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79340a8950 con 0x7f794015a2e0 2023-10-30T13:43:30.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793e7fc640 1 -- 172.21.15.31:0/3525945222 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79340a8ab0 con 0x7f794015a2e0 2023-10-30T13:43:30.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f793e7fc640 1 -- 172.21.15.31:0/3525945222 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79340a8e40 con 0x7f794015a2e0 2023-10-30T13:43:30.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f7945eea640 1 -- 172.21.15.31:0/3525945222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794015a2e0 msgr2=0x7f79400ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:30.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.672+0000 7f7945eea640 1 --2- 172.21.15.31:0/3525945222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794015a2e0 0x7f79400ccf60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f79400cdb70 tx=0x7f793401ec50 comp rx=0 tx=0).stop 2023-10-30T13:43:30.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- 172.21.15.31:0/3525945222 shutdown_connections 2023-10-30T13:43:30.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 --2- 172.21.15.31:0/3525945222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794015a2e0 0x7f79400ccf60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 --2- 172.21.15.31:0/3525945222 >> v2:172.21.15.33:3300/0 conn(0x7f79401537f0 0x7f79400701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- 172.21.15.31:0/3525945222 >> 172.21.15.31:0/3525945222 conn(0x7f7940071fb0 msgr2=0x7f79400723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:30.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- 172.21.15.31:0/3525945222 shutdown_connections 2023-10-30T13:43:30.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- 172.21.15.31:0/3525945222 wait complete. 2023-10-30T13:43:30.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 Processor -- start 2023-10-30T13:43:30.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- start start 2023-10-30T13:43:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79400cd4a0 0x7f794015aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79401537f0 0x7f794015ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f794015a2e0 0x7f794014c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794015aa90 con 0x7f79401537f0 2023-10-30T13:43:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794015b970 con 0x7f79400cd4a0 2023-10-30T13:43:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f7945eea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79400c6c20 con 0x7f794015a2e0 2023-10-30T13:43:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79401537f0 0x7f794015ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79401537f0 0x7f794015ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43732/0 (socket says 172.21.15.31:43732) 2023-10-30T13:43:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 -- 172.21.15.31:0/1995956800 learned_addr learned my addr 172.21.15.31:0/1995956800 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793f7fe640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79400cd4a0 0x7f794015aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793ffff640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f794015a2e0 0x7f794014c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 -- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f794015a2e0 msgr2=0x7f794014c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:30.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f794015a2e0 0x7f794014c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 -- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79400cd4a0 msgr2=0x7f794015aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79400cd4a0 0x7f794015aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 -- 172.21.15.31:0/1995956800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7934009420 con 0x7f79401537f0 2023-10-30T13:43:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793f7fe640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79400cd4a0 0x7f794015aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.676+0000 7f793effd640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79401537f0 0x7f794015ccc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f79340b3040 tx=0x7f793401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7927fff640 1 -- 172.21.15.31:0/1995956800 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79340b9630 con 0x7f79401537f0 2023-10-30T13:43:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7927fff640 1 -- 172.21.15.31:0/1995956800 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79340b97b0 con 0x7f79401537f0 2023-10-30T13:43:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7927fff640 1 -- 172.21.15.31:0/1995956800 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79340b9b60 con 0x7f79401537f0 2023-10-30T13:43:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79400c6f00 con 0x7f79401537f0 2023-10-30T13:43:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79400c5bb0 con 0x7f79401537f0 2023-10-30T13:43:30.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794004e170 con 0x7f79401537f0 2023-10-30T13:43:30.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7927fff640 1 -- 172.21.15.31:0/1995956800 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79340b7040 con 0x7f79401537f0 2023-10-30T13:43:30.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7927fff640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f792c0664a0 0x7f792c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:30.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f7927fff640 1 -- 172.21.15.31:0/1995956800 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f7934122020 con 0x7f79401537f0 2023-10-30T13:43:30.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.680+0000 7f793f7fe640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f792c0664a0 0x7f792c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:30.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.684+0000 7f793f7fe640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f792c0664a0 0x7f792c068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7930007a70 tx=0x7f7930008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:30.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.684+0000 7f7927fff640 1 -- 172.21.15.31:0/1995956800 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f79340ef320 con 0x7f79401537f0 2023-10-30T13:43:30.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7940073de0 con 0x7f79401537f0 2023-10-30T13:43:30.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7927fff640 1 -- 172.21.15.31:0/1995956800 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6488 (secure 0 0 0) 0x7f79340b5090 con 0x7f79401537f0 2023-10-30T13:43:30.877 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:30.878 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:12.145642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.33:6839/2419808617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":2419808617},{"type":"v1","addr":"172.21.15.33:6839","nonce":2419808617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2023-10-30T13:43:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f792c0664a0 msgr2=0x7f792c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f792c0664a0 0x7f792c068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7930007a70 tx=0x7f7930008040 comp rx=0 tx=0).stop 2023-10-30T13:43:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79401537f0 msgr2=0x7f794015ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:30.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79401537f0 0x7f794015ccc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f79340b3040 tx=0x7f793401b9f0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 shutdown_connections 2023-10-30T13:43:30.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f792c0664a0 0x7f792c068960 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f794015a2e0 0x7f794014c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79401537f0 0x7f794015ccc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 --2- 172.21.15.31:0/1995956800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79400cd4a0 0x7f794015aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 >> 172.21.15.31:0/1995956800 conn(0x7f7940071fb0 msgr2=0x7f794006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 shutdown_connections 2023-10-30T13:43:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:30.876+0000 7f7945eea640 1 -- 172.21.15.31:0/1995956800 wait complete. 2023-10-30T13:43:30.899 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:43:32.899 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:33.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d40d6640 1 Processor -- start 2023-10-30T13:43:33.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d40d6640 1 -- start start 2023-10-30T13:43:33.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d40d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc0c6320 0x7fb4cc06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d40d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb4cc155c40 0x7fb4cc1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d40d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0c55f0 con 0x7fb4cc14fc50 2023-10-30T13:43:33.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d40d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc0c5f00 con 0x7fb4cc155c40 2023-10-30T13:43:33.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d40d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc1567b0 con 0x7fb4cc0c6320 2023-10-30T13:43:33.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d1e72640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc0c6320 0x7fb4cc06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.051+0000 7fb4d2673640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb4cc155c40 0x7fb4cc1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d2673640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb4cc155c40 0x7fb4cc1560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:48262/0 (socket says 172.21.15.31:48262) 2023-10-30T13:43:33.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d1e72640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc0c6320 0x7fb4cc06fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:55186/0 (socket says 172.21.15.31:55186) 2023-10-30T13:43:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d2673640 1 -- 172.21.15.31:0/1832239232 learned_addr learned my addr 172.21.15.31:0/1832239232 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d1e72640 1 -- 172.21.15.31:0/1832239232 >> v2:172.21.15.33:3300/0 conn(0x7fb4cc155c40 msgr2=0x7fb4cc1560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:33.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d1e72640 1 --2- 172.21.15.31:0/1832239232 >> v2:172.21.15.33:3300/0 conn(0x7fb4cc155c40 0x7fb4cc1560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d1e72640 1 -- 172.21.15.31:0/1832239232 >> v1:172.21.15.33:6789/0 conn(0x7fb4cc14fc50 legacy=0x7fb4cc070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d1e72640 1 -- 172.21.15.31:0/1832239232 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc156a70 con 0x7fb4cc0c6320 2023-10-30T13:43:33.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d2673640 1 --2- 172.21.15.31:0/1832239232 >> v2:172.21.15.33:3300/0 conn(0x7fb4cc155c40 0x7fb4cc1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:43:33.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.055+0000 7fb4d1e72640 1 --2- 172.21.15.31:0/1832239232 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc0c6320 0x7fb4cc06fdf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80097c0 tx=0x7fb4c801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dda0928f3b76398 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:33.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d0e70640 1 -- 172.21.15.31:0/1832239232 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80ab5f0 con 0x7fb4cc0c6320 2023-10-30T13:43:33.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d0e70640 1 -- 172.21.15.31:0/1832239232 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c80ab750 con 0x7fb4cc0c6320 2023-10-30T13:43:33.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d0e70640 1 -- 172.21.15.31:0/1832239232 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80abae0 con 0x7fb4cc0c6320 2023-10-30T13:43:33.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- 172.21.15.31:0/1832239232 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc0c6320 msgr2=0x7fb4cc06fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/1832239232 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc0c6320 0x7fb4cc06fdf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80097c0 tx=0x7fb4c801c3f0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- 172.21.15.31:0/1832239232 shutdown_connections 2023-10-30T13:43:33.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/1832239232 >> v2:172.21.15.33:3300/0 conn(0x7fb4cc155c40 0x7fb4cc1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/1832239232 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc0c6320 0x7fb4cc06fdf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- 172.21.15.31:0/1832239232 >> 172.21.15.31:0/1832239232 conn(0x7fb4cc0cac70 msgr2=0x7fb4cc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:33.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- 172.21.15.31:0/1832239232 shutdown_connections 2023-10-30T13:43:33.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- 172.21.15.31:0/1832239232 wait complete. 2023-10-30T13:43:33.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 Processor -- start 2023-10-30T13:43:33.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- start start 2023-10-30T13:43:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4cc0c6320 0x7fb4cc158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc14fc50 0x7fb4cc158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4cc155c40 0x7fb4cc0c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc158150 con 0x7fb4cc0c6320 2023-10-30T13:43:33.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc141c60 con 0x7fb4cc155c40 2023-10-30T13:43:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.059+0000 7fb4d40d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc142ac0 con 0x7fb4cc14fc50 2023-10-30T13:43:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4cc0c6320 0x7fb4cc158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4cc0c6320 0x7fb4cc158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43744/0 (socket says 172.21.15.31:43744) 2023-10-30T13:43:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 -- 172.21.15.31:0/4079489654 learned_addr learned my addr 172.21.15.31:0/4079489654 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:33.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d2673640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4cc155c40 0x7fb4cc0c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1671640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc14fc50 0x7fb4cc158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 -- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc14fc50 msgr2=0x7fb4cc158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc14fc50 0x7fb4cc158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 -- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4cc155c40 msgr2=0x7fb4cc0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4cc155c40 0x7fb4cc0c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 -- 172.21.15.31:0/4079489654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c4009b30 con 0x7fb4cc0c6320 2023-10-30T13:43:33.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1671640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc14fc50 0x7fb4cc158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:33.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d1e72640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4cc0c6320 0x7fb4cc158670 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80ad040 tx=0x7fb4c801ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:33.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4c27fc640 1 -- 172.21.15.31:0/4079489654 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80bf610 con 0x7fb4cc0c6320 2023-10-30T13:43:33.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4c27fc640 1 -- 172.21.15.31:0/4079489654 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c80bf790 con 0x7fb4cc0c6320 2023-10-30T13:43:33.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c8009420 con 0x7fb4cc0c6320 2023-10-30T13:43:33.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4c27fc640 1 -- 172.21.15.31:0/4079489654 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80bfb40 con 0x7fb4cc0c6320 2023-10-30T13:43:33.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc142190 con 0x7fb4cc0c6320 2023-10-30T13:43:33.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc04e170 con 0x7fb4cc0c6320 2023-10-30T13:43:33.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4c27fc640 1 -- 172.21.15.31:0/4079489654 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4c80b5400 con 0x7fb4cc0c6320 2023-10-30T13:43:33.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.063+0000 7fb4c27fc640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4b00664a0 0x7fb4b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.067+0000 7fb4c27fc640 1 -- 172.21.15.31:0/4079489654 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fb4c80a8050 con 0x7fb4cc0c6320 2023-10-30T13:43:33.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.067+0000 7fb4d1671640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4b00664a0 0x7fb4b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.067+0000 7fb4d1671640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4b00664a0 0x7fb4b0068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc155bb0 tx=0x7fb4bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.067+0000 7fb4c27fc640 1 -- 172.21.15.31:0/4079489654 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4c80ebde0 con 0x7fb4cc0c6320 2023-10-30T13:43:33.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.259+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4cc14d260 con 0x7fb4cc0c6320 2023-10-30T13:43:33.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.259+0000 7fb4c27fc640 1 -- 172.21.15.31:0/4079489654 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5698 (secure 0 0 0) 0x7fb4c80c4090 con 0x7fb4cc0c6320 2023-10-30T13:43:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:33.264 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:31.895650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:33.266 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2023-10-30T13:43:33.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4b00664a0 msgr2=0x7fb4b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4b00664a0 0x7fb4b0068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc155bb0 tx=0x7fb4bc009040 comp rx=0 tx=0).stop 2023-10-30T13:43:33.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4cc0c6320 msgr2=0x7fb4cc158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4cc0c6320 0x7fb4cc158670 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80ad040 tx=0x7fb4c801ac30 comp rx=0 tx=0).stop 2023-10-30T13:43:33.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 shutdown_connections 2023-10-30T13:43:33.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb4b00664a0 0x7fb4b0068960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4cc155c40 0x7fb4cc0c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb4cc14fc50 0x7fb4cc158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 --2- 172.21.15.31:0/4079489654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4cc0c6320 0x7fb4cc158670 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 >> 172.21.15.31:0/4079489654 conn(0x7fb4cc0cac70 msgr2=0x7fb4cc140800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:33.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 shutdown_connections 2023-10-30T13:43:33.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.263+0000 7fb4d40d6640 1 -- 172.21.15.31:0/4079489654 wait complete. 2023-10-30T13:43:33.284 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-30T13:43:33.284 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:43:33.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.423+0000 7f6761c17640 1 Processor -- start 2023-10-30T13:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.423+0000 7f6761c17640 1 -- start start 2023-10-30T13:43:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f6761c17640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f675c153aa0 0x7f675c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f6761c17640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675c0c7b00 con 0x7f675c0701e0 2023-10-30T13:43:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f6761c17640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675c0c2050 con 0x7f675c06b3b0 2023-10-30T13:43:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f6761c17640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675c070a50 con 0x7f675c153aa0 2023-10-30T13:43:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f675c153aa0 0x7f675c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f675c0701e0 0x7f675c0ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:55566/0 (socket says 172.21.15.31:55566) 2023-10-30T13:43:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f675c153aa0 0x7f675c15d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43760/0 (socket says 172.21.15.31:43760) 2023-10-30T13:43:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675affd640 1 -- 172.21.15.31:0/1325556480 learned_addr learned my addr 172.21.15.31:0/1325556480 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:33.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675b7fe640 1 -- 172.21.15.31:0/1325556480 >> v1:172.21.15.33:6789/0 conn(0x7f675c06b3b0 legacy=0x7f675c06b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:33.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675b7fe640 1 -- 172.21.15.31:0/1325556480 >> v1:172.21.15.31:6789/0 conn(0x7f675c0701e0 legacy=0x7f675c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675a7fc640 1 -- 172.21.15.31:0/1325556480 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616221509 0 0) 0x7f675c0c7b00 con 0x7f675c0701e0 2023-10-30T13:43:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.427+0000 7f675b7fe640 1 -- 172.21.15.31:0/1325556480 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c0c2330 con 0x7f675c153aa0 2023-10-30T13:43:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f675b7fe640 1 --2- 172.21.15.31:0/1325556480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c15d840 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f674c0097c0 tx=0x7f674c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2e4946d2e44b00 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f675a7fc640 1 -- 172.21.15.31:0/1325556480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c0a8970 con 0x7f675c153aa0 2023-10-30T13:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f675a7fc640 1 -- 172.21.15.31:0/1325556480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c0a8ad0 con 0x7f675c153aa0 2023-10-30T13:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f675a7fc640 1 -- 172.21.15.31:0/1325556480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c0a8e60 con 0x7f675c153aa0 2023-10-30T13:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 -- 172.21.15.31:0/1325556480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 msgr2=0x7f675c15d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 --2- 172.21.15.31:0/1325556480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c15d840 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f674c0097c0 tx=0x7f674c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 -- 172.21.15.31:0/1325556480 shutdown_connections 2023-10-30T13:43:33.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 --2- 172.21.15.31:0/1325556480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c15d840 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 -- 172.21.15.31:0/1325556480 >> 172.21.15.31:0/1325556480 conn(0x7f675c071fb0 msgr2=0x7f675c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:33.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 -- 172.21.15.31:0/1325556480 shutdown_connections 2023-10-30T13:43:33.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 -- 172.21.15.31:0/1325556480 wait complete. 2023-10-30T13:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 Processor -- start 2023-10-30T13:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.431+0000 7f6761c17640 1 -- start start 2023-10-30T13:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f675c06b3b0 0x7f675c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f675c0701e0 0x7f675c15d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0c4650 con 0x7f675c153aa0 2023-10-30T13:43:33.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0c66f0 con 0x7f675c0701e0 2023-10-30T13:43:33.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0c4370 con 0x7f675c06b3b0 2023-10-30T13:43:33.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43776/0 (socket says 172.21.15.31:43776) 2023-10-30T13:43:33.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 -- 172.21.15.31:0/3488548941 learned_addr learned my addr 172.21.15.31:0/3488548941 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675b7fe640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f675c06b3b0 0x7f675c15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675affd640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f675c0701e0 0x7f675c15d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 -- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f675c06b3b0 msgr2=0x7f675c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f675c06b3b0 0x7f675c15d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 -- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f675c0701e0 msgr2=0x7f675c15d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f675c0701e0 0x7f675c15d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 -- 172.21.15.31:0/3488548941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c009420 con 0x7f675c153aa0 2023-10-30T13:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675affd640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f675c0701e0 0x7f675c15d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f675bfff640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c158ac0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f675400c770 tx=0x7f675400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:33.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6743fff640 1 -- 172.21.15.31:0/3488548941 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754003b80 con 0x7f675c153aa0 2023-10-30T13:43:33.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6743fff640 1 -- 172.21.15.31:0/3488548941 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6754003ce0 con 0x7f675c153aa0 2023-10-30T13:43:33.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6743fff640 1 -- 172.21.15.31:0/3488548941 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675400aa90 con 0x7f675c153aa0 2023-10-30T13:43:33.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c0c68e0 con 0x7f675c153aa0 2023-10-30T13:43:33.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c0c6ea0 con 0x7f675c153aa0 2023-10-30T13:43:33.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.435+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c04e170 con 0x7f675c153aa0 2023-10-30T13:43:33.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.439+0000 7f6743fff640 1 -- 172.21.15.31:0/3488548941 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f675400abf0 con 0x7f675c153aa0 2023-10-30T13:43:33.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.439+0000 7f6743fff640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67480664d0 0x7f6748068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:33.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.439+0000 7f6743fff640 1 -- 172.21.15.31:0/3488548941 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f6754083d40 con 0x7f675c153aa0 2023-10-30T13:43:33.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.439+0000 7f675b7fe640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67480664d0 0x7f6748068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:33.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.439+0000 7f675b7fe640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67480664d0 0x7f6748068990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f674c0b2000 tx=0x7f674c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:33.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.443+0000 7f6743fff640 1 -- 172.21.15.31:0/3488548941 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6754051060 con 0x7f675c153aa0 2023-10-30T13:43:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.631+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f675c1501b0 con 0x7f675c153aa0 2023-10-30T13:43:33.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6743fff640 1 -- 172.21.15.31:0/3488548941 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5698 (secure 0 0 0) 0x7f6754054680 con 0x7f675c153aa0 2023-10-30T13:43:33.636 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:33.637 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:31.895650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2023-10-30T13:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67480664d0 msgr2=0x7f6748068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67480664d0 0x7f6748068990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f674c0b2000 tx=0x7f674c008040 comp rx=0 tx=0).stop 2023-10-30T13:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 msgr2=0x7f675c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c158ac0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f675400c770 tx=0x7f675400cc30 comp rx=0 tx=0).stop 2023-10-30T13:43:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 shutdown_connections 2023-10-30T13:43:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67480664d0 0x7f6748068990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c153aa0 0x7f675c158ac0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f675c0701e0 0x7f675c15d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 --2- 172.21.15.31:0/3488548941 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f675c06b3b0 0x7f675c15d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:33.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 >> 172.21.15.31:0/3488548941 conn(0x7f675c071fb0 msgr2=0x7f675c1506d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:33.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 shutdown_connections 2023-10-30T13:43:33.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:33.635+0000 7f6761c17640 1 -- 172.21.15.31:0/3488548941 wait complete. 2023-10-30T13:43:33.658 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5470": { "addr": "172.21.15.33:6836/210349556", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 210349556, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 210349556, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5470, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:43:31.895650+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5470 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6837/349618961", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 349618961, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 349618961, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 47, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5448, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:43:35.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.811+0000 7faf86254640 1 Processor -- start 2023-10-30T13:43:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.811+0000 7faf86254640 1 -- start start 2023-10-30T13:43:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.811+0000 7faf86254640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faf80070520 0x7faf80070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.811+0000 7faf86254640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf800c2310 con 0x7faf800cb7d0 2023-10-30T13:43:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.811+0000 7faf86254640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf80156710 con 0x7faf80150830 2023-10-30T13:43:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.811+0000 7faf86254640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8006bc10 con 0x7faf80070520 2023-10-30T13:43:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faf80070520 0x7faf80070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faf80070520 0x7faf80070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50042/0 (socket says 172.21.15.31:50042) 2023-10-30T13:43:35.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7effd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7faf800cb7d0 0x7faf800c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46894/0 (socket says 172.21.15.31:46894) 2023-10-30T13:43:35.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7ffff640 1 -- 172.21.15.31:0/3218788291 learned_addr learned my addr 172.21.15.31:0/3218788291 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:35.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7ffff640 1 -- 172.21.15.31:0/3218788291 >> v1:172.21.15.33:6789/0 conn(0x7faf80150830 legacy=0x7faf8006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7ffff640 1 -- 172.21.15.31:0/3218788291 >> v1:172.21.15.31:6789/0 conn(0x7faf800cb7d0 legacy=0x7faf800c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7ffff640 1 -- 172.21.15.31:0/3218788291 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf800c4360 con 0x7faf80070520 2023-10-30T13:43:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7e7fc640 1 -- 172.21.15.31:0/3218788291 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973518855 0 0) 0x7faf800c2310 con 0x7faf800cb7d0 2023-10-30T13:43:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7ffff640 1 --2- 172.21.15.31:0/3218788291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf80070990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7faf78009e50 tx=0x7faf7801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd4648fa2de9a2e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7e7fc640 1 -- 172.21.15.31:0/3218788291 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf780af400 con 0x7faf80070520 2023-10-30T13:43:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7e7fc640 1 -- 172.21.15.31:0/3218788291 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf780ab050 con 0x7faf80070520 2023-10-30T13:43:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.815+0000 7faf7e7fc640 1 -- 172.21.15.31:0/3218788291 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf780af780 con 0x7faf80070520 2023-10-30T13:43:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- 172.21.15.31:0/3218788291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 msgr2=0x7faf80070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 --2- 172.21.15.31:0/3218788291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf80070990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7faf78009e50 tx=0x7faf7801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:43:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- 172.21.15.31:0/3218788291 shutdown_connections 2023-10-30T13:43:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 --2- 172.21.15.31:0/3218788291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf80070990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- 172.21.15.31:0/3218788291 >> 172.21.15.31:0/3218788291 conn(0x7faf800cac70 msgr2=0x7faf800cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- 172.21.15.31:0/3218788291 shutdown_connections 2023-10-30T13:43:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- 172.21.15.31:0/3218788291 wait complete. 2023-10-30T13:43:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 Processor -- start 2023-10-30T13:43:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- start start 2023-10-30T13:43:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf8006bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf800cb7d0 0x7faf80154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf80150830 0x7faf80154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf801eeb90 con 0x7faf80070520 2023-10-30T13:43:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf800c5800 con 0x7faf800cb7d0 2023-10-30T13:43:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf86254640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf801ef9b0 con 0x7faf80150830 2023-10-30T13:43:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf7f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf8006bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf7f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf8006bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50044/0 (socket says 172.21.15.31:50044) 2023-10-30T13:43:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf7f7fe640 1 -- 172.21.15.31:0/2174124733 learned_addr learned my addr 172.21.15.31:0/2174124733 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf7effd640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf800cb7d0 0x7faf80154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.819+0000 7faf7ffff640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf80150830 0x7faf80154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf7f7fe640 1 -- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf80150830 msgr2=0x7faf80154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf7f7fe640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf80150830 0x7faf80154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf7f7fe640 1 -- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf800cb7d0 msgr2=0x7faf80154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf7f7fe640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf800cb7d0 0x7faf80154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf7f7fe640 1 -- 172.21.15.31:0/2174124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf78009b30 con 0x7faf80070520 2023-10-30T13:43:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf7effd640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf800cb7d0 0x7faf80154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf7f7fe640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf8006bfb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faf7000c3c0 tx=0x7faf7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf67fff640 1 -- 172.21.15.31:0/2174124733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf700052d0 con 0x7faf80070520 2023-10-30T13:43:35.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf67fff640 1 -- 172.21.15.31:0/2174124733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf70005430 con 0x7faf80070520 2023-10-30T13:43:35.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf801eeda0 con 0x7faf80070520 2023-10-30T13:43:35.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf67fff640 1 -- 172.21.15.31:0/2174124733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70003840 con 0x7faf80070520 2023-10-30T13:43:35.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf801ef360 con 0x7faf80070520 2023-10-30T13:43:35.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8004e170 con 0x7faf80070520 2023-10-30T13:43:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf67fff640 1 -- 172.21.15.31:0/2174124733 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf700039a0 con 0x7faf80070520 2023-10-30T13:43:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf67fff640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf6c0664a0 0x7faf6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.823+0000 7faf67fff640 1 -- 172.21.15.31:0/2174124733 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7faf700836f0 con 0x7faf80070520 2023-10-30T13:43:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.827+0000 7faf7effd640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf6c0664a0 0x7faf6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.827+0000 7faf7effd640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf6c0664a0 0x7faf6c068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf8014e5d0 tx=0x7faf74007ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:35.827+0000 7faf67fff640 1 -- 172.21.15.31:0/2174124733 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf70050850 con 0x7faf80070520 2023-10-30T13:43:36.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.019+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf800c61c0 con 0x7faf80070520 2023-10-30T13:43:36.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.019+0000 7faf67fff640 1 -- 172.21.15.31:0/2174124733 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5698 (secure 0 0 0) 0x7faf70053e70 con 0x7faf80070520 2023-10-30T13:43:36.023 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:36.023 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:31.895650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:36.023 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2023-10-30T13:43:36.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf6c0664a0 msgr2=0x7faf6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:36.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf6c0664a0 0x7faf6c068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf8014e5d0 tx=0x7faf74007ca0 comp rx=0 tx=0).stop 2023-10-30T13:43:36.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 msgr2=0x7faf8006bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:36.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf8006bfb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faf7000c3c0 tx=0x7faf7000c880 comp rx=0 tx=0).stop 2023-10-30T13:43:36.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 shutdown_connections 2023-10-30T13:43:36.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf6c0664a0 0x7faf6c068960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:36.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf80150830 0x7faf80154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:36.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf800cb7d0 0x7faf80154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:36.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 --2- 172.21.15.31:0/2174124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf80070520 0x7faf8006bfb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:36.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 >> 172.21.15.31:0/2174124733 conn(0x7faf800cac70 msgr2=0x7faf800733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:36.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 shutdown_connections 2023-10-30T13:43:36.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:36.023+0000 7faf86254640 1 -- 172.21.15.31:0/2174124733 wait complete. 2023-10-30T13:43:36.044 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:38.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:38.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.195+0000 7f4d6cdf5640 1 Processor -- start 2023-10-30T13:43:38.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.195+0000 7f4d6cdf5640 1 -- start start 2023-10-30T13:43:38.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6cdf5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f4d68150830 0x7f4d6806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:38.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6cdf5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680cb7d0 0x7f4d680c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:38.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6cdf5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d680c2310 con 0x7f4d680c3e40 2023-10-30T13:43:38.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6cdf5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d68156710 con 0x7f4d68150830 2023-10-30T13:43:38.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6cdf5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d680c4360 con 0x7f4d680cb7d0 2023-10-30T13:43:38.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d5ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680cb7d0 0x7f4d680c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:38.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6659c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f4d68150830 0x7f4d6806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d5ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680cb7d0 0x7f4d680c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:42534/0 (socket says 172.21.15.31:42534) 2023-10-30T13:43:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d5ffff640 1 -- 172.21.15.31:0/3282327921 learned_addr learned my addr 172.21.15.31:0/3282327921 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6659c640 1 -- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680cb7d0 msgr2=0x7f4d680c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6659c640 1 --2- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680cb7d0 0x7f4d680c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6659c640 1 -- 172.21.15.31:0/3282327921 >> v1:172.21.15.33:6789/0 conn(0x7f4d680c3e40 legacy=0x7f4d68070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:38.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d6659c640 1 -- 172.21.15.31:0/3282327921 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d681569b0 con 0x7f4d68150830 2023-10-30T13:43:38.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.199+0000 7f4d5ffff640 1 --2- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680cb7d0 0x7f4d680c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:43:38.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6659c640 1 --2- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d68150830 0x7f4d6806b500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d580097c0 tx=0x7f4d5801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64387c29e007d75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d65d9b640 1 -- 172.21.15.31:0/3282327921 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d580a8970 con 0x7f4d68150830 2023-10-30T13:43:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d65d9b640 1 -- 172.21.15.31:0/3282327921 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d580a8ad0 con 0x7f4d68150830 2023-10-30T13:43:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d65d9b640 1 -- 172.21.15.31:0/3282327921 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d580a8e60 con 0x7f4d68150830 2023-10-30T13:43:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d68150830 msgr2=0x7f4d6806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d68150830 0x7f4d6806b500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d580097c0 tx=0x7f4d5801ec70 comp rx=0 tx=0).stop 2023-10-30T13:43:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/3282327921 shutdown_connections 2023-10-30T13:43:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680cb7d0 0x7f4d680c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/3282327921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d68150830 0x7f4d6806b500 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/3282327921 >> 172.21.15.31:0/3282327921 conn(0x7f4d680cac70 msgr2=0x7f4d680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/3282327921 shutdown_connections 2023-10-30T13:43:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.203+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/3282327921 wait complete. 2023-10-30T13:43:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 Processor -- start 2023-10-30T13:43:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 -- start start 2023-10-30T13:43:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680c3e40 0x7f4d681ef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d680cb7d0 0x7f4d6814f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68150830 0x7f4d68159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6814de10 con 0x7f4d68150830 2023-10-30T13:43:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681efa00 con 0x7f4d680cb7d0 2023-10-30T13:43:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681f23c0 con 0x7f4d680c3e40 2023-10-30T13:43:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68150830 0x7f4d68159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68150830 0x7f4d68159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50056/0 (socket says 172.21.15.31:50056) 2023-10-30T13:43:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 -- 172.21.15.31:0/1633140397 learned_addr learned my addr 172.21.15.31:0/1633140397 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6659c640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680c3e40 0x7f4d681ef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d5ffff640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d680cb7d0 0x7f4d6814f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 -- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680c3e40 msgr2=0x7f4d681ef4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680c3e40 0x7f4d681ef4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 -- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d680cb7d0 msgr2=0x7f4d6814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d680cb7d0 0x7f4d6814f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 -- 172.21.15.31:0/1633140397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094d0 con 0x7f4d68150830 2023-10-30T13:43:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d5ffff640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d680cb7d0 0x7f4d6814f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d66d9d640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68150830 0x7f4d68159ea0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000c7d0 tx=0x7f4d6000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d5effd640 1 -- 172.21.15.31:0/1633140397 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60003b80 con 0x7f4d68150830 2023-10-30T13:43:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d5effd640 1 -- 172.21.15.31:0/1633140397 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d60003ce0 con 0x7f4d68150830 2023-10-30T13:43:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d58009420 con 0x7f4d68150830 2023-10-30T13:43:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d5effd640 1 -- 172.21.15.31:0/1633140397 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6000aa30 con 0x7f4d68150830 2023-10-30T13:43:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.207+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d681f2980 con 0x7f4d68150830 2023-10-30T13:43:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.211+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6804e170 con 0x7f4d68150830 2023-10-30T13:43:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.211+0000 7f4d5effd640 1 -- 172.21.15.31:0/1633140397 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d6000ab90 con 0x7f4d68150830 2023-10-30T13:43:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.211+0000 7f4d5effd640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4d48066520 0x7f4d480689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.211+0000 7f4d5effd640 1 -- 172.21.15.31:0/1633140397 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f4d60083e10 con 0x7f4d68150830 2023-10-30T13:43:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.211+0000 7f4d6659c640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4d48066520 0x7f4d480689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.211+0000 7f4d6659c640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4d48066520 0x7f4d480689e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d580b2000 tx=0x7f4d580aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.215+0000 7f4d5effd640 1 -- 172.21.15.31:0/1633140397 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d60087030 con 0x7f4d68150830 2023-10-30T13:43:38.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.403+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d68070300 con 0x7f4d68150830 2023-10-30T13:43:38.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.403+0000 7f4d5effd640 1 -- 172.21.15.31:0/1633140397 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5698 (secure 0 0 0) 0x7f4d60051080 con 0x7f4d68150830 2023-10-30T13:43:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:38.408 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:31.895650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:38.408 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2023-10-30T13:43:38.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4d48066520 msgr2=0x7f4d480689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:38.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4d48066520 0x7f4d480689e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d580b2000 tx=0x7f4d580aa070 comp rx=0 tx=0).stop 2023-10-30T13:43:38.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68150830 msgr2=0x7f4d68159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:38.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68150830 0x7f4d68159ea0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000c7d0 tx=0x7f4d6000cc90 comp rx=0 tx=0).stop 2023-10-30T13:43:38.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 shutdown_connections 2023-10-30T13:43:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4d48066520 0x7f4d480689e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68150830 0x7f4d68159ea0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d680cb7d0 0x7f4d6814f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 --2- 172.21.15.31:0/1633140397 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4d680c3e40 0x7f4d681ef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:38.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 >> 172.21.15.31:0/1633140397 conn(0x7f4d680cac70 msgr2=0x7f4d68070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 shutdown_connections 2023-10-30T13:43:38.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:38.407+0000 7f4d6cdf5640 1 -- 172.21.15.31:0/1633140397 wait complete. 2023-10-30T13:43:38.429 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:40.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f680a009640 1 Processor -- start 2023-10-30T13:43:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f680a009640 1 -- start start 2023-10-30T13:43:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f680a009640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f680415cf90 0x7f68040c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:40.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f680a009640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6804159f20 con 0x7f6804153c80 2023-10-30T13:43:40.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f680a009640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68040c47e0 con 0x7f68040c4a10 2023-10-30T13:43:40.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f680a009640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680415dec0 con 0x7f680415cf90 2023-10-30T13:43:40.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f68037fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f6804153c80 0x7f680415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46904/0 (socket says 172.21.15.31:46904) 2023-10-30T13:43:40.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f68037fe640 1 -- 172.21.15.31:0/91008903 learned_addr learned my addr 172.21.15.31:0/91008903 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:40.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f6803fff640 1 --2- 172.21.15.31:0/91008903 >> v2:172.21.15.31:3300/0 conn(0x7f680415cf90 0x7f68040c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:40.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f6803fff640 1 -- 172.21.15.31:0/91008903 >> v1:172.21.15.33:6789/0 conn(0x7f68040c4a10 legacy=0x7f68040ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:40.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f6803fff640 1 -- 172.21.15.31:0/91008903 >> v1:172.21.15.31:6789/0 conn(0x7f6804153c80 legacy=0x7f680415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:40.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f6803fff640 1 -- 172.21.15.31:0/91008903 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680415d620 con 0x7f680415cf90 2023-10-30T13:43:40.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.583+0000 7f68027fc640 1 -- 172.21.15.31:0/91008903 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459771424 0 0) 0x7f6804159f20 con 0x7f6804153c80 2023-10-30T13:43:40.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f6803fff640 1 --2- 172.21.15.31:0/91008903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f680415cf90 0x7f68040c42a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009d90 tx=0x7f67fc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9fbaadf6b55e23 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:40.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f68027fc640 1 -- 172.21.15.31:0/91008903 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0af600 con 0x7f680415cf90 2023-10-30T13:43:40.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f68027fc640 1 -- 172.21.15.31:0/91008903 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc0ab050 con 0x7f680415cf90 2023-10-30T13:43:40.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f68027fc640 1 -- 172.21.15.31:0/91008903 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0af980 con 0x7f680415cf90 2023-10-30T13:43:40.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f680a009640 1 -- 172.21.15.31:0/91008903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f680415cf90 msgr2=0x7f68040c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:40.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f680a009640 1 --2- 172.21.15.31:0/91008903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f680415cf90 0x7f68040c42a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009d90 tx=0x7f67fc01f130 comp rx=0 tx=0).stop 2023-10-30T13:43:40.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f680a009640 1 -- 172.21.15.31:0/91008903 shutdown_connections 2023-10-30T13:43:40.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f680a009640 1 --2- 172.21.15.31:0/91008903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f680415cf90 0x7f68040c42a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:40.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f680a009640 1 -- 172.21.15.31:0/91008903 >> 172.21.15.31:0/91008903 conn(0x7f68040715c0 msgr2=0x7f68040719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:40.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f680a009640 1 -- 172.21.15.31:0/91008903 shutdown_connections 2023-10-30T13:43:40.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.587+0000 7f680a009640 1 -- 172.21.15.31:0/91008903 wait complete. 2023-10-30T13:43:40.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 Processor -- start 2023-10-30T13:43:40.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 -- start start 2023-10-30T13:43:40.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68040c4a10 0x7f6804156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:40.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6804153c80 0x7f6804158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:40.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f680415cf90 0x7f6804158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:40.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6804156ea0 con 0x7f68040c4a10 2023-10-30T13:43:40.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68040b1d50 con 0x7f680415cf90 2023-10-30T13:43:40.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68040cf590 con 0x7f6804153c80 2023-10-30T13:43:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68040c4a10 0x7f6804156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68040c4a10 0x7f6804156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50074/0 (socket says 172.21.15.31:50074) 2023-10-30T13:43:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 -- 172.21.15.31:0/2019278008 learned_addr learned my addr 172.21.15.31:0/2019278008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f6803fff640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f680415cf90 0x7f6804158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f6802ffd640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6804153c80 0x7f6804158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 -- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6804153c80 msgr2=0x7f6804158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:40.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6804153c80 0x7f6804158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:40.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 -- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f680415cf90 msgr2=0x7f6804158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:40.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f680415cf90 0x7f6804158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:40.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 -- 172.21.15.31:0/2019278008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc009a70 con 0x7f68040c4a10 2023-10-30T13:43:40.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f68037fe640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68040c4a10 0x7f6804156530 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c4a0 tx=0x7f67f400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:40.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f67ebfff640 1 -- 172.21.15.31:0/2019278008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f40074e0 con 0x7f68040c4a10 2023-10-30T13:43:40.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f67ebfff640 1 -- 172.21.15.31:0/2019278008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4007640 con 0x7f68040c4a10 2023-10-30T13:43:40.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68040cf830 con 0x7f68040c4a10 2023-10-30T13:43:40.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f67ebfff640 1 -- 172.21.15.31:0/2019278008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4004a30 con 0x7f68040c4a10 2023-10-30T13:43:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.591+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68040b1a00 con 0x7f68040c4a10 2023-10-30T13:43:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.595+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680404e170 con 0x7f68040c4a10 2023-10-30T13:43:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.595+0000 7f67ebfff640 1 -- 172.21.15.31:0/2019278008 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67f4004b90 con 0x7f68040c4a10 2023-10-30T13:43:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.595+0000 7f67ebfff640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67f00664d0 0x7f67f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.595+0000 7f67ebfff640 1 -- 172.21.15.31:0/2019278008 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f67f4083510 con 0x7f68040c4a10 2023-10-30T13:43:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.595+0000 7f6802ffd640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67f00664d0 0x7f67f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.599+0000 7f6802ffd640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67f00664d0 0x7f67f0068990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68040cf440 tx=0x7f67f8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.599+0000 7f67ebfff640 1 -- 172.21.15.31:0/2019278008 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67f4050790 con 0x7f68040c4a10 2023-10-30T13:43:40.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68040ceb90 con 0x7f68040c4a10 2023-10-30T13:43:40.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f67ebfff640 1 -- 172.21.15.31:0/2019278008 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f67f4053db0 con 0x7f68040c4a10 2023-10-30T13:43:40.797 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:40.797 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:38.030599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:40.797 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 49 2023-10-30T13:43:40.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67f00664d0 msgr2=0x7f67f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:40.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67f00664d0 0x7f67f0068990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68040cf440 tx=0x7f67f8009040 comp rx=0 tx=0).stop 2023-10-30T13:43:40.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68040c4a10 msgr2=0x7f6804156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:40.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68040c4a10 0x7f6804156530 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c4a0 tx=0x7f67f400c960 comp rx=0 tx=0).stop 2023-10-30T13:43:40.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 shutdown_connections 2023-10-30T13:43:40.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f67f00664d0 0x7f67f0068990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:40.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f680415cf90 0x7f6804158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:40.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6804153c80 0x7f6804158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:40.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 --2- 172.21.15.31:0/2019278008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68040c4a10 0x7f6804156530 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:40.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 >> 172.21.15.31:0/2019278008 conn(0x7f68040715c0 msgr2=0x7f68041520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:40.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 shutdown_connections 2023-10-30T13:43:40.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:40.795+0000 7f680a009640 1 -- 172.21.15.31:0/2019278008 wait complete. 2023-10-30T13:43:40.819 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:42.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:42.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.971+0000 7f5b1ae68640 1 Processor -- start 2023-10-30T13:43:42.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b1ae68640 1 -- start start 2023-10-30T13:43:42.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b1ae68640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1414e030 0x7f5b14072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:42.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b1ae68640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1406b500 con 0x7f5b1406b990 2023-10-30T13:43:42.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b1ae68640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1414e410 con 0x7f5b1406fdf0 2023-10-30T13:43:42.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b1ae68640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b141582e0 con 0x7f5b1414e030 2023-10-30T13:43:42.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b19405640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f5b1406b990 0x7f5b1406be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46914/0 (socket says 172.21.15.31:46914) 2023-10-30T13:43:42.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b19405640 1 -- 172.21.15.31:0/452669242 learned_addr learned my addr 172.21.15.31:0/452669242 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b18c04640 1 --2- 172.21.15.31:0/452669242 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1414e030 0x7f5b14072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984909252 0 0) 0x7f5b1406b500 con 0x7f5b1406b990 2023-10-30T13:43:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04003530 con 0x7f5b1406b990 2023-10-30T13:43:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773346303 0 0) 0x7f5b1414e410 con 0x7f5b1406fdf0 2023-10-30T13:43:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1406b500 con 0x7f5b1406fdf0 2023-10-30T13:43:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b18c04640 1 -- 172.21.15.31:0/452669242 >> v1:172.21.15.33:6789/0 conn(0x7f5b1406fdf0 legacy=0x7f5b140707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b18c04640 1 -- 172.21.15.31:0/452669242 >> v1:172.21.15.31:6789/0 conn(0x7f5b1406b990 legacy=0x7f5b1406be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3213202415 0 0) 0x7f5b04003530 con 0x7f5b1406b990 2023-10-30T13:43:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.975+0000 7f5b18c04640 1 -- 172.21.15.31:0/452669242 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b140c5820 con 0x7f5b1414e030 2023-10-30T13:43:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b18c04640 1 --2- 172.21.15.31:0/452669242 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1414e030 0x7f5b14072890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b080097f0 tx=0x7f5b0801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b5fc16ae9aa14d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c001a00 con 0x7f5b1414e030 2023-10-30T13:43:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b080a7050 con 0x7f5b1414e030 2023-10-30T13:43:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b137fe640 1 -- 172.21.15.31:0/452669242 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080ab600 con 0x7f5b1414e030 2023-10-30T13:43:42.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b1ae68640 1 -- 172.21.15.31:0/452669242 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1414e030 msgr2=0x7f5b14072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:42.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/452669242 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1414e030 0x7f5b14072890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b080097f0 tx=0x7f5b0801c3e0 comp rx=0 tx=0).stop 2023-10-30T13:43:42.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b1ae68640 1 -- 172.21.15.31:0/452669242 shutdown_connections 2023-10-30T13:43:42.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/452669242 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1414e030 0x7f5b14072890 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:42.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b1ae68640 1 -- 172.21.15.31:0/452669242 >> 172.21.15.31:0/452669242 conn(0x7f5b14069e90 msgr2=0x7f5b1406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b1ae68640 1 -- 172.21.15.31:0/452669242 shutdown_connections 2023-10-30T13:43:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.979+0000 7f5b1ae68640 1 -- 172.21.15.31:0/452669242 wait complete. 2023-10-30T13:43:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 Processor -- start 2023-10-30T13:43:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 -- start start 2023-10-30T13:43:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1406b990 0x7f5b14156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1406fdf0 0x7f5b141f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b1414e030 0x7f5b141490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140c9890 con 0x7f5b1406b990 2023-10-30T13:43:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b14151eb0 con 0x7f5b1414e030 2023-10-30T13:43:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b141562d0 con 0x7f5b1406fdf0 2023-10-30T13:43:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1406b990 0x7f5b14156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1406b990 0x7f5b14156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50082/0 (socket says 172.21.15.31:50082) 2023-10-30T13:43:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 -- 172.21.15.31:0/3588799210 learned_addr learned my addr 172.21.15.31:0/3588799210 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b13fff640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1406fdf0 0x7f5b141f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b19405640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b1414e030 0x7f5b141490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 -- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1406fdf0 msgr2=0x7f5b141f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1406fdf0 0x7f5b141f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 -- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b1414e030 msgr2=0x7f5b141490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b1414e030 0x7f5b141490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:42.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 -- 172.21.15.31:0/3588799210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08009420 con 0x7f5b1406b990 2023-10-30T13:43:42.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b18c04640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1406b990 0x7f5b14156d70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5b0801a010 tx=0x7f5b08019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b117fa640 1 -- 172.21.15.31:0/3588799210 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08003820 con 0x7f5b1406b990 2023-10-30T13:43:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b117fa640 1 -- 172.21.15.31:0/3588799210 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b08003980 con 0x7f5b1406b990 2023-10-30T13:43:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b117fa640 1 -- 172.21.15.31:0/3588799210 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08003cf0 con 0x7f5b1406b990 2023-10-30T13:43:42.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b14152100 con 0x7f5b1406b990 2023-10-30T13:43:42.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.983+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b14156570 con 0x7f5b1406b990 2023-10-30T13:43:42.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.987+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1404e170 con 0x7f5b1406b990 2023-10-30T13:43:42.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.987+0000 7f5b117fa640 1 -- 172.21.15.31:0/3588799210 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b080b2050 con 0x7f5b1406b990 2023-10-30T13:43:42.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.987+0000 7f5b117fa640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5af80664a0 0x7f5af8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:42.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.987+0000 7f5b117fa640 1 -- 172.21.15.31:0/3588799210 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f5b0811e110 con 0x7f5b1406b990 2023-10-30T13:43:42.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.987+0000 7f5b13fff640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5af80664a0 0x7f5af8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:42.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.987+0000 7f5b13fff640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5af80664a0 0x7f5af8068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5b140c8870 tx=0x7f5b00009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:42.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:42.991+0000 7f5b117fa640 1 -- 172.21.15.31:0/3588799210 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b080eb390 con 0x7f5b1406b990 2023-10-30T13:43:43.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.183+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b140c5ea0 con 0x7f5b1406b990 2023-10-30T13:43:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b117fa640 1 -- 172.21.15.31:0/3588799210 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f5b080ee9b0 con 0x7f5b1406b990 2023-10-30T13:43:43.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:43.188 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:38.030599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 49 2023-10-30T13:43:43.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5af80664a0 msgr2=0x7f5af8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:43.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5af80664a0 0x7f5af8068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5b140c8870 tx=0x7f5b00009040 comp rx=0 tx=0).stop 2023-10-30T13:43:43.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1406b990 msgr2=0x7f5b14156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:43.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1406b990 0x7f5b14156d70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5b0801a010 tx=0x7f5b08019180 comp rx=0 tx=0).stop 2023-10-30T13:43:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 shutdown_connections 2023-10-30T13:43:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5af80664a0 0x7f5af8068960 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b1414e030 0x7f5b141490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5b1406fdf0 0x7f5b141f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 --2- 172.21.15.31:0/3588799210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1406b990 0x7f5b14156d70 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:43.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 >> 172.21.15.31:0/3588799210 conn(0x7f5b14069e90 msgr2=0x7f5b14149ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:43.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 shutdown_connections 2023-10-30T13:43:43.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:43.187+0000 7f5b1ae68640 1 -- 172.21.15.31:0/3588799210 wait complete. 2023-10-30T13:43:43.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:45.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f8885a91640 1 Processor -- start 2023-10-30T13:43:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f8885a91640 1 -- start start 2023-10-30T13:43:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f8885a91640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8880153c80 0x7f888015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f8885a91640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88800c42a0 0x7f88800ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f8885a91640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8880159f20 con 0x7f88800c4a10 2023-10-30T13:43:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f8885a91640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888015dec0 con 0x7f8880153c80 2023-10-30T13:43:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f8885a91640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88801599a0 con 0x7f88800c42a0 2023-10-30T13:43:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887f7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8880153c80 0x7f888015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887effd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f88800c4a10 0x7f888015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:40954/0 (socket says 172.21.15.31:40954) 2023-10-30T13:43:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887effd640 1 -- 172.21.15.31:0/2693360117 learned_addr learned my addr 172.21.15.31:0/2693360117 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887ffff640 1 --2- 172.21.15.31:0/2693360117 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88800c42a0 0x7f88800ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887f7fe640 1 -- 172.21.15.31:0/2693360117 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88800c42a0 msgr2=0x7f88800ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887f7fe640 1 --2- 172.21.15.31:0/2693360117 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88800c42a0 0x7f88800ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887f7fe640 1 -- 172.21.15.31:0/2693360117 >> v1:172.21.15.31:6789/0 conn(0x7f88800c4a10 legacy=0x7f888015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887f7fe640 1 -- 172.21.15.31:0/2693360117 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888015d620 con 0x7f8880153c80 2023-10-30T13:43:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.363+0000 7f887e7fc640 1 -- 172.21.15.31:0/2693360117 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192617319 0 0) 0x7f8880159f20 con 0x7f88800c4a10 2023-10-30T13:43:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f887f7fe640 1 --2- 172.21.15.31:0/2693360117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8880153c80 0x7f888015cf90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f88700097c0 tx=0x7f887001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81ffd0057b9d1d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f887e7fc640 1 -- 172.21.15.31:0/2693360117 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88700a8970 con 0x7f8880153c80 2023-10-30T13:43:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f887e7fc640 1 -- 172.21.15.31:0/2693360117 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88700a8ad0 con 0x7f8880153c80 2023-10-30T13:43:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f887e7fc640 1 -- 172.21.15.31:0/2693360117 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88700a8e60 con 0x7f8880153c80 2023-10-30T13:43:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 -- 172.21.15.31:0/2693360117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8880153c80 msgr2=0x7f888015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 --2- 172.21.15.31:0/2693360117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8880153c80 0x7f888015cf90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f88700097c0 tx=0x7f887001ec70 comp rx=0 tx=0).stop 2023-10-30T13:43:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 -- 172.21.15.31:0/2693360117 shutdown_connections 2023-10-30T13:43:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 --2- 172.21.15.31:0/2693360117 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f88800c42a0 0x7f88800ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 --2- 172.21.15.31:0/2693360117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8880153c80 0x7f888015cf90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 -- 172.21.15.31:0/2693360117 >> 172.21.15.31:0/2693360117 conn(0x7f88800715c0 msgr2=0x7f88800719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 -- 172.21.15.31:0/2693360117 shutdown_connections 2023-10-30T13:43:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.367+0000 7f8885a91640 1 -- 172.21.15.31:0/2693360117 wait complete. 2023-10-30T13:43:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 Processor -- start 2023-10-30T13:43:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 -- start start 2023-10-30T13:43:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88800c42a0 0x7f88800c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88800c4a10 0x7f888015cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8880153c80 0x7f88800ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800ce1e0 con 0x7f88800c4a10 2023-10-30T13:43:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800cd2f0 con 0x7f88800c42a0 2023-10-30T13:43:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88800cd5d0 con 0x7f8880153c80 2023-10-30T13:43:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88800c4a10 0x7f888015cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88800c4a10 0x7f888015cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39554/0 (socket says 172.21.15.31:39554) 2023-10-30T13:43:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 -- 172.21.15.31:0/3134049356 learned_addr learned my addr 172.21.15.31:0/3134049356 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887f7fe640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88800c42a0 0x7f88800c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887ffff640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8880153c80 0x7f88800ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 -- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8880153c80 msgr2=0x7f88800ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8880153c80 0x7f88800ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 -- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88800c42a0 msgr2=0x7f88800c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88800c42a0 0x7f88800c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 -- 172.21.15.31:0/3134049356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8870009420 con 0x7f88800c4a10 2023-10-30T13:43:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887f7fe640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88800c42a0 0x7f88800c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887ffff640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8880153c80 0x7f88800ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f887effd640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88800c4a10 0x7f888015cb30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f887400c4b0 tx=0x7f887400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8867fff640 1 -- 172.21.15.31:0/3134049356 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740074e0 con 0x7f88800c4a10 2023-10-30T13:43:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8867fff640 1 -- 172.21.15.31:0/3134049356 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874007660 con 0x7f88800c4a10 2023-10-30T13:43:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88800cd8b0 con 0x7f88800c4a10 2023-10-30T13:43:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8867fff640 1 -- 172.21.15.31:0/3134049356 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874004a30 con 0x7f88800c4a10 2023-10-30T13:43:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.371+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88800cde70 con 0x7f88800c4a10 2023-10-30T13:43:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.375+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88800ce390 con 0x7f88800c4a10 2023-10-30T13:43:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.375+0000 7f8867fff640 1 -- 172.21.15.31:0/3134049356 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8874004b90 con 0x7f88800c4a10 2023-10-30T13:43:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.375+0000 7f8867fff640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f886c0664a0 0x7f886c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.375+0000 7f8867fff640 1 -- 172.21.15.31:0/3134049356 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f88740835e0 con 0x7f88800c4a10 2023-10-30T13:43:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.375+0000 7f887f7fe640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f886c0664a0 0x7f886c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.379+0000 7f887f7fe640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f886c0664a0 0x7f886c068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f88700b2000 tx=0x7f8870008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.379+0000 7f8867fff640 1 -- 172.21.15.31:0/3134049356 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88800ce390 con 0x7f88800c4a10 2023-10-30T13:43:45.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.567+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f888004e170 con 0x7f88800c4a10 2023-10-30T13:43:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.567+0000 7f8867fff640 1 -- 172.21.15.31:0/3134049356 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f8874018090 con 0x7f88800c4a10 2023-10-30T13:43:45.571 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:45.571 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:44.019147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 50 2023-10-30T13:43:45.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f886c0664a0 msgr2=0x7f886c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:45.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f886c0664a0 0x7f886c068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f88700b2000 tx=0x7f8870008040 comp rx=0 tx=0).stop 2023-10-30T13:43:45.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88800c4a10 msgr2=0x7f888015cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88800c4a10 0x7f888015cb30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f887400c4b0 tx=0x7f887400c970 comp rx=0 tx=0).stop 2023-10-30T13:43:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 shutdown_connections 2023-10-30T13:43:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f886c0664a0 0x7f886c068960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8880153c80 0x7f88800ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88800c4a10 0x7f888015cb30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 --2- 172.21.15.31:0/3134049356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88800c42a0 0x7f88800c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 >> 172.21.15.31:0/3134049356 conn(0x7f88800715c0 msgr2=0x7f88800719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 shutdown_connections 2023-10-30T13:43:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:45.571+0000 7f8885a91640 1 -- 172.21.15.31:0/3134049356 wait complete. 2023-10-30T13:43:45.592 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5470": { "addr": "172.21.15.33:6836/210349556", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 210349556, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 210349556, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5470, "incarnation": 48, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:43:44.019147+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5470 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6837/349618961", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 349618961, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 349618961, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 47, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5448, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:43:47.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:47.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5826c68640 1 Processor -- start 2023-10-30T13:43:47.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5826c68640 1 -- start start 2023-10-30T13:43:47.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5826c68640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f58200cb7d0 0x7f58200c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:47.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5826c68640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f5820070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:47.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5826c68640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58200c2310 con 0x7f5820150830 2023-10-30T13:43:47.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5826c68640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582006bc10 con 0x7f58200cb7d0 2023-10-30T13:43:47.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5826c68640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5820156710 con 0x7f58200c3e40 2023-10-30T13:43:47.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f581ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f58200cb7d0 0x7f58200c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:47.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f581ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f58200cb7d0 0x7f58200c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39568/0 (socket says 172.21.15.31:39568) 2023-10-30T13:43:47.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f581ffff640 1 -- 172.21.15.31:0/2876609149 learned_addr learned my addr 172.21.15.31:0/2876609149 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:47.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5825205640 1 --2- 172.21.15.31:0/2876609149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f5820070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:47.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f581ffff640 1 -- 172.21.15.31:0/2876609149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 msgr2=0x7f5820070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:47.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f581ffff640 1 --2- 172.21.15.31:0/2876609149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f5820070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f581ffff640 1 -- 172.21.15.31:0/2876609149 >> v1:172.21.15.33:6789/0 conn(0x7f5820150830 legacy=0x7f582006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:47.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f581ffff640 1 -- 172.21.15.31:0/2876609149 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58200c4360 con 0x7f58200cb7d0 2023-10-30T13:43:47.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.747+0000 7f5825205640 1 --2- 172.21.15.31:0/2876609149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f5820070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:47.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f581ffff640 1 --2- 172.21.15.31:0/2876609149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58200c3900 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f580c009870 tx=0x7f580c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b04c67ce0f4068f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:47.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f581f7fe640 1 -- 172.21.15.31:0/2876609149 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0a8a30 con 0x7f58200cb7d0 2023-10-30T13:43:47.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f581f7fe640 1 -- 172.21.15.31:0/2876609149 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c0a8b90 con 0x7f58200cb7d0 2023-10-30T13:43:47.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f581f7fe640 1 -- 172.21.15.31:0/2876609149 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0b0440 con 0x7f58200cb7d0 2023-10-30T13:43:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 -- 172.21.15.31:0/2876609149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 msgr2=0x7f58200c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 --2- 172.21.15.31:0/2876609149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58200c3900 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f580c009870 tx=0x7f580c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:43:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 -- 172.21.15.31:0/2876609149 shutdown_connections 2023-10-30T13:43:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 --2- 172.21.15.31:0/2876609149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f5820070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 --2- 172.21.15.31:0/2876609149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58200c3900 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 -- 172.21.15.31:0/2876609149 >> 172.21.15.31:0/2876609149 conn(0x7f58200cac70 msgr2=0x7f58200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 -- 172.21.15.31:0/2876609149 shutdown_connections 2023-10-30T13:43:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.751+0000 7f5826c68640 1 -- 172.21.15.31:0/2876609149 wait complete. 2023-10-30T13:43:47.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 Processor -- start 2023-10-30T13:43:47.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 -- start start 2023-10-30T13:43:47.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f58200ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:47.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58201ef160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:47.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5820150830 0x7f5820145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58200caae0 con 0x7f58200cb7d0 2023-10-30T13:43:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58201ef6a0 con 0x7f5820150830 2023-10-30T13:43:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5820145bb0 con 0x7f58200c3e40 2023-10-30T13:43:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58201ef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58201ef160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39580/0 (socket says 172.21.15.31:39580) 2023-10-30T13:43:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 -- 172.21.15.31:0/1272404203 learned_addr learned my addr 172.21.15.31:0/1272404203 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5824a04640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f58200ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5825205640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5820150830 0x7f5820145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 -- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 msgr2=0x7f58200ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f58200ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 -- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5820150830 msgr2=0x7f5820145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5820150830 0x7f5820145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 -- 172.21.15.31:0/1272404203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c0094d0 con 0x7f58200cb7d0 2023-10-30T13:43:47.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581ffff640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58201ef160 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f580c0b3040 tx=0x7f580c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:47.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581d7fa640 1 -- 172.21.15.31:0/1272404203 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0be070 con 0x7f58200cb7d0 2023-10-30T13:43:47.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581d7fa640 1 -- 172.21.15.31:0/1272404203 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c0ba400 con 0x7f58200cb7d0 2023-10-30T13:43:47.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f581d7fa640 1 -- 172.21.15.31:0/1272404203 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0ba7d0 con 0x7f58200cb7d0 2023-10-30T13:43:47.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582013d830 con 0x7f58200cb7d0 2023-10-30T13:43:47.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.755+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58201f1df0 con 0x7f58200cb7d0 2023-10-30T13:43:47.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.759+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58201f1c50 con 0x7f58200cb7d0 2023-10-30T13:43:47.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.759+0000 7f581d7fa640 1 -- 172.21.15.31:0/1272404203 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f580c0c3410 con 0x7f58200cb7d0 2023-10-30T13:43:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.759+0000 7f581d7fa640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f58040664a0 0x7f5804068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.759+0000 7f581d7fa640 1 -- 172.21.15.31:0/1272404203 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f580c121b30 con 0x7f58200cb7d0 2023-10-30T13:43:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.759+0000 7f5824a04640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f58040664a0 0x7f5804068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.759+0000 7f5824a04640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f58040664a0 0x7f5804068960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5814007fd0 tx=0x7f5814009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:47.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.763+0000 7f581d7fa640 1 -- 172.21.15.31:0/1272404203 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58201f1c50 con 0x7f58200cb7d0 2023-10-30T13:43:47.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.955+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58201f1c50 con 0x7f58200cb7d0 2023-10-30T13:43:47.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.955+0000 7f581d7fa640 1 -- 172.21.15.31:0/1272404203 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f58201f1c50 con 0x7f58200cb7d0 2023-10-30T13:43:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:47.959 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:44.019147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:47.959 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 50 2023-10-30T13:43:47.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f58040664a0 msgr2=0x7f5804068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:47.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f58040664a0 0x7f5804068960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5814007fd0 tx=0x7f5814009040 comp rx=0 tx=0).stop 2023-10-30T13:43:47.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 msgr2=0x7f58201ef160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:47.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58201ef160 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f580c0b3040 tx=0x7f580c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 shutdown_connections 2023-10-30T13:43:47.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f58040664a0 0x7f5804068960 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5820150830 0x7f5820145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58200cb7d0 0x7f58201ef160 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 --2- 172.21.15.31:0/1272404203 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f58200c3e40 0x7f58200ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:47.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 >> 172.21.15.31:0/1272404203 conn(0x7f58200cac70 msgr2=0x7f58200733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:47.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 shutdown_connections 2023-10-30T13:43:47.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:47.959+0000 7f5826c68640 1 -- 172.21.15.31:0/1272404203 wait complete. 2023-10-30T13:43:47.980 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:49.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:50.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4ba1ba1640 1 Processor -- start 2023-10-30T13:43:50.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4ba1ba1640 1 -- start start 2023-10-30T13:43:50.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4ba1ba1640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f4b9c151840 0x7f4b9c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:50.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4ba1ba1640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4b9c0cb8b0 0x7f4b9c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:50.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4ba1ba1640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0c5030 con 0x7f4b9c151c20 2023-10-30T13:43:50.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4ba1ba1640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c157830 con 0x7f4b9c0cb8b0 2023-10-30T13:43:50.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4ba1ba1640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c157b10 con 0x7f4b9c151840 2023-10-30T13:43:50.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4b9c0cb8b0 0x7f4b9c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:50.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f4b9c151c20 0x7f4b9c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:40964/0 (socket says 172.21.15.31:40964) 2023-10-30T13:43:50.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9affd640 1 -- 172.21.15.31:0/2469335296 learned_addr learned my addr 172.21.15.31:0/2469335296 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:50.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9b7fe640 1 --2- 172.21.15.31:0/2469335296 >> v2:172.21.15.33:3300/0 conn(0x7f4b9c151840 0x7f4b9c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:50.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9bfff640 1 -- 172.21.15.31:0/2469335296 >> v2:172.21.15.33:3300/0 conn(0x7f4b9c151840 msgr2=0x7f4b9c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:50.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9bfff640 1 --2- 172.21.15.31:0/2469335296 >> v2:172.21.15.33:3300/0 conn(0x7f4b9c151840 0x7f4b9c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9bfff640 1 -- 172.21.15.31:0/2469335296 >> v1:172.21.15.31:6789/0 conn(0x7f4b9c151c20 legacy=0x7f4b9c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:50.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9bfff640 1 -- 172.21.15.31:0/2469335296 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c157df0 con 0x7f4b9c0cb8b0 2023-10-30T13:43:50.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.135+0000 7f4b9a7fc640 1 -- 172.21.15.31:0/2469335296 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961747149 0 0) 0x7f4b9c0c5030 con 0x7f4b9c151c20 2023-10-30T13:43:50.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4b9bfff640 1 --2- 172.21.15.31:0/2469335296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c06b600 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c072b40 tx=0x7f4b9401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a6430e9e61824e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:50.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4b9a7fc640 1 -- 172.21.15.31:0/2469335296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940b0440 con 0x7f4b9c0cb8b0 2023-10-30T13:43:50.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4b9a7fc640 1 -- 172.21.15.31:0/2469335296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b940b05a0 con 0x7f4b9c0cb8b0 2023-10-30T13:43:50.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4b9a7fc640 1 -- 172.21.15.31:0/2469335296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940b0930 con 0x7f4b9c0cb8b0 2023-10-30T13:43:50.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/2469335296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c0cb8b0 msgr2=0x7f4b9c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:50.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/2469335296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c06b600 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c072b40 tx=0x7f4b9401f0f0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/2469335296 shutdown_connections 2023-10-30T13:43:50.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/2469335296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c06b600 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/2469335296 >> v2:172.21.15.33:3300/0 conn(0x7f4b9c151840 0x7f4b9c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/2469335296 >> 172.21.15.31:0/2469335296 conn(0x7f4b9c0cac70 msgr2=0x7f4b9c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:50.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/2469335296 shutdown_connections 2023-10-30T13:43:50.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.139+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/2469335296 wait complete. 2023-10-30T13:43:50.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 Processor -- start 2023-10-30T13:43:50.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 -- start start 2023-10-30T13:43:50.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c1e2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:50.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c151840 0x7f4b9c14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:50.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b9c151c20 0x7f4b9c1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:50.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c1e2690 con 0x7f4b9c151840 2023-10-30T13:43:50.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c1e2e40 con 0x7f4b9c0cb8b0 2023-10-30T13:43:50.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c1f29e0 con 0x7f4b9c151c20 2023-10-30T13:43:50.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c151840 0x7f4b9c14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:50.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c151840 0x7f4b9c14f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39592/0 (socket says 172.21.15.31:39592) 2023-10-30T13:43:50.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 -- 172.21.15.31:0/779908580 learned_addr learned my addr 172.21.15.31:0/779908580 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:50.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9b7fe640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c1e2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:50.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9bfff640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b9c151c20 0x7f4b9c1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:50.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 -- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b9c151c20 msgr2=0x7f4b9c1eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:50.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b9c151c20 0x7f4b9c1eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 -- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b9c0cb8b0 msgr2=0x7f4b9c1e2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:50.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c1e2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 -- 172.21.15.31:0/779908580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b94009a70 con 0x7f4b9c151840 2023-10-30T13:43:50.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9b7fe640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c1e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:43:50.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b9affd640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c151840 0x7f4b9c14f290 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c480 tx=0x7f4b9000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:50.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b83fff640 1 -- 172.21.15.31:0/779908580 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900074e0 con 0x7f4b9c151840 2023-10-30T13:43:50.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b83fff640 1 -- 172.21.15.31:0/779908580 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90007660 con 0x7f4b9c151840 2023-10-30T13:43:50.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c1f2cb0 con 0x7f4b9c151840 2023-10-30T13:43:50.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.143+0000 7f4b83fff640 1 -- 172.21.15.31:0/779908580 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90004a30 con 0x7f4b9c151840 2023-10-30T13:43:50.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.147+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c1e22b0 con 0x7f4b9c151840 2023-10-30T13:43:50.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.147+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c1e2440 con 0x7f4b9c151840 2023-10-30T13:43:50.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.147+0000 7f4b83fff640 1 -- 172.21.15.31:0/779908580 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b90004b90 con 0x7f4b9c151840 2023-10-30T13:43:50.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.147+0000 7f4b83fff640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b880664a0 0x7f4b88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:50.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.147+0000 7f4b83fff640 1 -- 172.21.15.31:0/779908580 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f4b90083460 con 0x7f4b9c151840 2023-10-30T13:43:50.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.147+0000 7f4b9b7fe640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b880664a0 0x7f4b88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:50.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.151+0000 7f4b9b7fe640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b880664a0 0x7f4b88068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c006aa0 tx=0x7f4b8c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:50.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.151+0000 7f4b83fff640 1 -- 172.21.15.31:0/779908580 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b90050830 con 0x7f4b9c151840 2023-10-30T13:43:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b9c04e170 con 0x7f4b9c151840 2023-10-30T13:43:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4b83fff640 1 -- 172.21.15.31:0/779908580 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f4b90053e50 con 0x7f4b9c151840 2023-10-30T13:43:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:50.346 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:49.125978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2023-10-30T13:43:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b880664a0 msgr2=0x7f4b88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b880664a0 0x7f4b88068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c006aa0 tx=0x7f4b8c008040 comp rx=0 tx=0).stop 2023-10-30T13:43:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c151840 msgr2=0x7f4b9c14f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c151840 0x7f4b9c14f290 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c480 tx=0x7f4b9000c940 comp rx=0 tx=0).stop 2023-10-30T13:43:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 shutdown_connections 2023-10-30T13:43:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4b880664a0 0x7f4b88068960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4b9c151c20 0x7f4b9c1eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b9c151840 0x7f4b9c14f290 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 --2- 172.21.15.31:0/779908580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b9c0cb8b0 0x7f4b9c1e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 >> 172.21.15.31:0/779908580 conn(0x7f4b9c0cac70 msgr2=0x7f4b9c147540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 shutdown_connections 2023-10-30T13:43:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:50.343+0000 7f4ba1ba1640 1 -- 172.21.15.31:0/779908580 wait complete. 2023-10-30T13:43:50.366 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:52.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:52.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f602053e640 1 Processor -- start 2023-10-30T13:43:52.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f602053e640 1 -- start start 2023-10-30T13:43:52.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f602053e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:52.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f602053e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6018070520 0x7f60180709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:52.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f602053e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60180c2310 con 0x7f60180cb7d0 2023-10-30T13:43:52.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f602053e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6018156710 con 0x7f6018070520 2023-10-30T13:43:52.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f602053e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60180c4360 con 0x7f6018150830 2023-10-30T13:43:52.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f601eadb640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6018070520 0x7f60180709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:52.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.519+0000 7f601eadb640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6018070520 0x7f60180709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39606/0 (socket says 172.21.15.31:39606) 2023-10-30T13:43:52.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601eadb640 1 -- 172.21.15.31:0/2515767872 learned_addr learned my addr 172.21.15.31:0/2515767872 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:52.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601e2da640 1 --2- 172.21.15.31:0/2515767872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:52.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601eadb640 1 -- 172.21.15.31:0/2515767872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 msgr2=0x7f601806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:52.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601eadb640 1 --2- 172.21.15.31:0/2515767872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601eadb640 1 -- 172.21.15.31:0/2515767872 >> v1:172.21.15.33:6789/0 conn(0x7f60180cb7d0 legacy=0x7f60180c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:52.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601eadb640 1 -- 172.21.15.31:0/2515767872 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60181569b0 con 0x7f6018070520 2023-10-30T13:43:52.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601eadb640 1 --2- 172.21.15.31:0/2515767872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f60180709b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f601000d150 tx=0x7f6010021400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a3dce8c76d65c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:52.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601d2d8640 1 -- 172.21.15.31:0/2515767872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601000b2d0 con 0x7f6018070520 2023-10-30T13:43:52.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601d2d8640 1 -- 172.21.15.31:0/2515767872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60100ad050 con 0x7f6018070520 2023-10-30T13:43:52.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.523+0000 7f601d2d8640 1 -- 172.21.15.31:0/2515767872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010009820 con 0x7f6018070520 2023-10-30T13:43:52.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- 172.21.15.31:0/2515767872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 msgr2=0x7f60180709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:52.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 --2- 172.21.15.31:0/2515767872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f60180709b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f601000d150 tx=0x7f6010021400 comp rx=0 tx=0).stop 2023-10-30T13:43:52.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- 172.21.15.31:0/2515767872 shutdown_connections 2023-10-30T13:43:52.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 --2- 172.21.15.31:0/2515767872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f60180709b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 --2- 172.21.15.31:0/2515767872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- 172.21.15.31:0/2515767872 >> 172.21.15.31:0/2515767872 conn(0x7f60180cac70 msgr2=0x7f60180cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:52.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- 172.21.15.31:0/2515767872 shutdown_connections 2023-10-30T13:43:52.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- 172.21.15.31:0/2515767872 wait complete. 2023-10-30T13:43:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 Processor -- start 2023-10-30T13:43:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- start start 2023-10-30T13:43:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f6018141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60180cb7d0 0x7f6018149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601814a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60181481f0 con 0x7f6018070520 2023-10-30T13:43:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601814ccb0 con 0x7f60180cb7d0 2023-10-30T13:43:52.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f602053e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601814cf70 con 0x7f6018150830 2023-10-30T13:43:52.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601e2da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f6018141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:52.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601e2da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f6018141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39618/0 (socket says 172.21.15.31:39618) 2023-10-30T13:43:52.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601e2da640 1 -- 172.21.15.31:0/4202284440 learned_addr learned my addr 172.21.15.31:0/4202284440 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:52.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601dad9640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60180cb7d0 0x7f6018149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:52.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601eadb640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601814a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601e2da640 1 -- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 msgr2=0x7f601814a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601e2da640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601814a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601e2da640 1 -- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60180cb7d0 msgr2=0x7f6018149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.527+0000 7f601e2da640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60180cb7d0 0x7f6018149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f601e2da640 1 -- 172.21.15.31:0/4202284440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601000bd40 con 0x7f6018070520 2023-10-30T13:43:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f601eadb640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601814a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f601e2da640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f6018141900 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f601400e870 tx=0x7f601400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f600effd640 1 -- 172.21.15.31:0/4202284440 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014009510 con 0x7f6018070520 2023-10-30T13:43:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f600effd640 1 -- 172.21.15.31:0/4202284440 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601400eec0 con 0x7f6018070520 2023-10-30T13:43:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f600effd640 1 -- 172.21.15.31:0/4202284440 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601400cb40 con 0x7f6018070520 2023-10-30T13:43:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6018148490 con 0x7f6018070520 2023-10-30T13:43:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601814d250 con 0x7f6018070520 2023-10-30T13:43:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601804e170 con 0x7f6018070520 2023-10-30T13:43:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f600effd640 1 -- 172.21.15.31:0/4202284440 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f601400cca0 con 0x7f6018070520 2023-10-30T13:43:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f600effd640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ffc0664a0 0x7f5ffc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f600effd640 1 -- 172.21.15.31:0/4202284440 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f60140842d0 con 0x7f6018070520 2023-10-30T13:43:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.531+0000 7f601dad9640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ffc0664a0 0x7f5ffc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.535+0000 7f601dad9640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ffc0664a0 0x7f5ffc068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f60180c6bf0 tx=0x7f6008009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:52.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.535+0000 7f600effd640 1 -- 172.21.15.31:0/4202284440 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6014051430 con 0x7f6018070520 2023-10-30T13:43:52.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.723+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f601806c7a0 con 0x7f6018070520 2023-10-30T13:43:52.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f600effd640 1 -- 172.21.15.31:0/4202284440 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f6014054a50 con 0x7f6018070520 2023-10-30T13:43:52.728 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:52.728 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:49.125978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:52.729 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2023-10-30T13:43:52.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ffc0664a0 msgr2=0x7f5ffc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:52.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ffc0664a0 0x7f5ffc068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f60180c6bf0 tx=0x7f6008009040 comp rx=0 tx=0).stop 2023-10-30T13:43:52.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 msgr2=0x7f6018141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:52.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f6018141900 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f601400e870 tx=0x7f601400ec70 comp rx=0 tx=0).stop 2023-10-30T13:43:52.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 shutdown_connections 2023-10-30T13:43:52.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ffc0664a0 0x7f5ffc068960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6018150830 0x7f601814a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60180cb7d0 0x7f6018149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 --2- 172.21.15.31:0/4202284440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6018070520 0x7f6018141900 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:52.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 >> 172.21.15.31:0/4202284440 conn(0x7f60180cac70 msgr2=0x7f60180c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:52.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 shutdown_connections 2023-10-30T13:43:52.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:52.727+0000 7f602053e640 1 -- 172.21.15.31:0/4202284440 wait complete. 2023-10-30T13:43:52.749 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:54.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9c66b640 1 Processor -- start 2023-10-30T13:43:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9c66b640 1 -- start start 2023-10-30T13:43:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9c66b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0e94153c80 0x7f0e9415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9c66b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e940c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9c66b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e94159f20 con 0x7f0e9415a3b0 2023-10-30T13:43:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9c66b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9415dec0 con 0x7f0e94153c80 2023-10-30T13:43:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9c66b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e941599a0 con 0x7f0e940c4a10 2023-10-30T13:43:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9a407640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0e94153c80 0x7f0e9415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9a407640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0e94153c80 0x7f0e9415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44420/0 (socket says 172.21.15.31:44420) 2023-10-30T13:43:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9a407640 1 -- 172.21.15.31:0/158705415 learned_addr learned my addr 172.21.15.31:0/158705415 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9ac08640 1 --1- 172.21.15.31:0/158705415 >> v1:172.21.15.31:6789/0 conn(0x7f0e9415a3b0 0x7f0e940ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37158/0 (socket says 172.21.15.31:37158) 2023-10-30T13:43:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e99c06640 1 --2- 172.21.15.31:0/158705415 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e940c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9a407640 1 -- 172.21.15.31:0/158705415 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 msgr2=0x7f0e940c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9a407640 1 --2- 172.21.15.31:0/158705415 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e940c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9a407640 1 -- 172.21.15.31:0/158705415 >> v1:172.21.15.31:6789/0 conn(0x7f0e9415a3b0 legacy=0x7f0e940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e9a407640 1 -- 172.21.15.31:0/158705415 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e940c47e0 con 0x7f0e94153c80 2023-10-30T13:43:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.903+0000 7f0e99405640 1 -- 172.21.15.31:0/158705415 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180776474 0 0) 0x7f0e94159f20 con 0x7f0e9415a3b0 2023-10-30T13:43:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9a407640 1 --2- 172.21.15.31:0/158705415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e9415cf90 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009870 tx=0x7f0e9001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4659b1e53bf6e9f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e99405640 1 -- 172.21.15.31:0/158705415 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900a8a30 con 0x7f0e94153c80 2023-10-30T13:43:54.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e99405640 1 -- 172.21.15.31:0/158705415 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e900a8b90 con 0x7f0e94153c80 2023-10-30T13:43:54.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e99405640 1 -- 172.21.15.31:0/158705415 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900b0440 con 0x7f0e94153c80 2023-10-30T13:43:54.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 -- 172.21.15.31:0/158705415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 msgr2=0x7f0e9415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:54.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/158705415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e9415cf90 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009870 tx=0x7f0e9001ed30 comp rx=0 tx=0).stop 2023-10-30T13:43:54.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 -- 172.21.15.31:0/158705415 shutdown_connections 2023-10-30T13:43:54.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/158705415 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e940c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:54.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/158705415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e9415cf90 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:54.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 -- 172.21.15.31:0/158705415 >> 172.21.15.31:0/158705415 conn(0x7f0e940715c0 msgr2=0x7f0e940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:54.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 -- 172.21.15.31:0/158705415 shutdown_connections 2023-10-30T13:43:54.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.907+0000 7f0e9c66b640 1 -- 172.21.15.31:0/158705415 wait complete. 2023-10-30T13:43:54.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 Processor -- start 2023-10-30T13:43:54.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 -- start start 2023-10-30T13:43:54.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e9415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:54.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e940c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:54.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9415a3b0 0x7f0e9414ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:54.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9414b290 con 0x7f0e94153c80 2023-10-30T13:43:54.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9415b050 con 0x7f0e9415a3b0 2023-10-30T13:43:54.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e94151ee0 con 0x7f0e940c4a10 2023-10-30T13:43:54.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e940c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:54.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e940c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44434/0 (socket says 172.21.15.31:44434) 2023-10-30T13:43:54.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 -- 172.21.15.31:0/1458395323 learned_addr learned my addr 172.21.15.31:0/1458395323 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:54.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9a407640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e9415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:54.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9ac08640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9415a3b0 0x7f0e9414ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:54.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 -- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 msgr2=0x7f0e9415b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e9415b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 -- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9415a3b0 msgr2=0x7f0e9414ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9415a3b0 0x7f0e9414ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 -- 172.21.15.31:0/1458395323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e900094d0 con 0x7f0e94153c80 2023-10-30T13:43:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e99c06640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e940c8b50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400d800 tx=0x7f0e8400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e8affd640 1 -- 172.21.15.31:0/1458395323 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400b6c0 con 0x7f0e94153c80 2023-10-30T13:43:54.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e8affd640 1 -- 172.21.15.31:0/1458395323 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8400b840 con 0x7f0e94153c80 2023-10-30T13:43:54.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e8affd640 1 -- 172.21.15.31:0/1458395323 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400bbf0 con 0x7f0e94153c80 2023-10-30T13:43:54.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e941521c0 con 0x7f0e94153c80 2023-10-30T13:43:54.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.911+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e94152780 con 0x7f0e94153c80 2023-10-30T13:43:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.915+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9415b4e0 con 0x7f0e94153c80 2023-10-30T13:43:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.915+0000 7f0e8affd640 1 -- 172.21.15.31:0/1458395323 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e8400bd50 con 0x7f0e94153c80 2023-10-30T13:43:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.915+0000 7f0e8affd640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e780664a0 0x7f0e78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.915+0000 7f0e8affd640 1 -- 172.21.15.31:0/1458395323 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f0e84084680 con 0x7f0e94153c80 2023-10-30T13:43:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.915+0000 7f0e9a407640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e780664a0 0x7f0e78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.915+0000 7f0e9a407640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e780664a0 0x7f0e78068960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e900b2000 tx=0x7f0e90008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:54.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:54.919+0000 7f0e8affd640 1 -- 172.21.15.31:0/1458395323 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e9415b4e0 con 0x7f0e94153c80 2023-10-30T13:43:55.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e9415b190 con 0x7f0e94153c80 2023-10-30T13:43:55.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e8affd640 1 -- 172.21.15.31:0/1458395323 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f0e9415b190 con 0x7f0e94153c80 2023-10-30T13:43:55.113 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:55.113 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:49.125978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:55.113 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2023-10-30T13:43:55.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e780664a0 msgr2=0x7f0e78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:55.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e780664a0 0x7f0e78068960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e900b2000 tx=0x7f0e90008040 comp rx=0 tx=0).stop 2023-10-30T13:43:55.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 msgr2=0x7f0e940c8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:55.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e940c8b50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400d800 tx=0x7f0e8400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:43:55.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 shutdown_connections 2023-10-30T13:43:55.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e780664a0 0x7f0e78068960 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:55.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9415a3b0 0x7f0e9414ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:55.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e94153c80 0x7f0e940c8b50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:55.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 --2- 172.21.15.31:0/1458395323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e940c4a10 0x7f0e9415b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:55.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 >> 172.21.15.31:0/1458395323 conn(0x7f0e940715c0 msgr2=0x7f0e9406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:55.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 shutdown_connections 2023-10-30T13:43:55.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:55.111+0000 7f0e9c66b640 1 -- 172.21.15.31:0/1458395323 wait complete. 2023-10-30T13:43:55.134 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:56.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:56.346 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:57.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:57.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f808359d640 1 Processor -- start 2023-10-30T13:43:57.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f808359d640 1 -- start start 2023-10-30T13:43:57.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f808359d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c153c80 0x7f807c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f808359d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f807c0c4a10 0x7f807c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f808359d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c159f20 con 0x7f807c15a3b0 2023-10-30T13:43:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f808359d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c15dec0 con 0x7f807c0c4a10 2023-10-30T13:43:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f808359d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c1599a0 con 0x7f807c153c80 2023-10-30T13:43:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8081339640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c153c80 0x7f807c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8081339640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c153c80 0x7f807c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:56834/0 (socket says 172.21.15.31:56834) 2023-10-30T13:43:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8081339640 1 -- 172.21.15.31:0/1289450328 learned_addr learned my addr 172.21.15.31:0/1289450328 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8080b38640 1 --2- 172.21.15.31:0/1289450328 >> v2:172.21.15.33:3300/0 conn(0x7f807c0c4a10 0x7f807c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8081339640 1 -- 172.21.15.31:0/1289450328 >> v2:172.21.15.33:3300/0 conn(0x7f807c0c4a10 msgr2=0x7f807c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8081339640 1 --2- 172.21.15.31:0/1289450328 >> v2:172.21.15.33:3300/0 conn(0x7f807c0c4a10 0x7f807c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8081339640 1 -- 172.21.15.31:0/1289450328 >> v1:172.21.15.33:6789/0 conn(0x7f807c15a3b0 legacy=0x7f807c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.287+0000 7f8081339640 1 -- 172.21.15.31:0/1289450328 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0c47e0 con 0x7f807c153c80 2023-10-30T13:43:57.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f8081339640 1 --2- 172.21.15.31:0/1289450328 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c153c80 0x7f807c15cf90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8078009870 tx=0x7f807801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c0607292f6cc75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f8073fff640 1 -- 172.21.15.31:0/1289450328 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780a8a30 con 0x7f807c153c80 2023-10-30T13:43:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f8073fff640 1 -- 172.21.15.31:0/1289450328 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80780a8b90 con 0x7f807c153c80 2023-10-30T13:43:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f8073fff640 1 -- 172.21.15.31:0/1289450328 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780b0440 con 0x7f807c153c80 2023-10-30T13:43:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 -- 172.21.15.31:0/1289450328 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c153c80 msgr2=0x7f807c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 --2- 172.21.15.31:0/1289450328 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c153c80 0x7f807c15cf90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8078009870 tx=0x7f807801ed30 comp rx=0 tx=0).stop 2023-10-30T13:43:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 -- 172.21.15.31:0/1289450328 shutdown_connections 2023-10-30T13:43:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 --2- 172.21.15.31:0/1289450328 >> v2:172.21.15.33:3300/0 conn(0x7f807c0c4a10 0x7f807c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 --2- 172.21.15.31:0/1289450328 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c153c80 0x7f807c15cf90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 -- 172.21.15.31:0/1289450328 >> 172.21.15.31:0/1289450328 conn(0x7f807c0715c0 msgr2=0x7f807c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 -- 172.21.15.31:0/1289450328 shutdown_connections 2023-10-30T13:43:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.291+0000 7f808359d640 1 -- 172.21.15.31:0/1289450328 wait complete. 2023-10-30T13:43:57.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 Processor -- start 2023-10-30T13:43:57.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 -- start start 2023-10-30T13:43:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0c4a10 0x7f807c15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c153c80 0x7f807c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c15a3b0 0x7f807c152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c154fd0 con 0x7f807c0c4a10 2023-10-30T13:43:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c152050 con 0x7f807c153c80 2023-10-30T13:43:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c1f25c0 con 0x7f807c15a3b0 2023-10-30T13:43:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0c4a10 0x7f807c15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0c4a10 0x7f807c15cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44442/0 (socket says 172.21.15.31:44442) 2023-10-30T13:43:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 -- 172.21.15.31:0/2595988710 learned_addr learned my addr 172.21.15.31:0/2595988710 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081b3a640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c15a3b0 0x7f807c152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8080b38640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c153c80 0x7f807c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 -- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c15a3b0 msgr2=0x7f807c152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c15a3b0 0x7f807c152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 -- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c153c80 msgr2=0x7f807c1527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c153c80 0x7f807c1527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 -- 172.21.15.31:0/2595988710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80780094d0 con 0x7f807c0c4a10 2023-10-30T13:43:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8081339640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0c4a10 0x7f807c15cc30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f80780097c0 tx=0x7f807801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:57.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8071ffb640 1 -- 172.21.15.31:0/2595988710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780bf070 con 0x7f807c0c4a10 2023-10-30T13:43:57.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8071ffb640 1 -- 172.21.15.31:0/2595988710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80780b1b60 con 0x7f807c0c4a10 2023-10-30T13:43:57.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f8071ffb640 1 -- 172.21.15.31:0/2595988710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780b9410 con 0x7f807c0c4a10 2023-10-30T13:43:57.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c1f28a0 con 0x7f807c0c4a10 2023-10-30T13:43:57.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.295+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c1f2e60 con 0x7f807c0c4a10 2023-10-30T13:43:57.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.299+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c04e170 con 0x7f807c0c4a10 2023-10-30T13:43:57.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.299+0000 7f8071ffb640 1 -- 172.21.15.31:0/2595988710 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f80780b7040 con 0x7f807c0c4a10 2023-10-30T13:43:57.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.299+0000 7f8071ffb640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80600664a0 0x7f8060068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.299+0000 7f8071ffb640 1 -- 172.21.15.31:0/2595988710 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8078121360 con 0x7f807c0c4a10 2023-10-30T13:43:57.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.299+0000 7f8080b38640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80600664a0 0x7f8060068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.299+0000 7f8080b38640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80600664a0 0x7f8060068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f807c0c5f70 tx=0x7f806c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:57.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.303+0000 7f8071ffb640 1 -- 172.21.15.31:0/2595988710 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80780ee550 con 0x7f807c0c4a10 2023-10-30T13:43:57.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.491+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807c0cad40 con 0x7f807c0c4a10 2023-10-30T13:43:57.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.491+0000 7f8071ffb640 1 -- 172.21.15.31:0/2595988710 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f80780f1b70 con 0x7f807c0c4a10 2023-10-30T13:43:57.495 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:57.495 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:57.496 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 52 2023-10-30T13:43:57.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80600664a0 msgr2=0x7f8060068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80600664a0 0x7f8060068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f807c0c5f70 tx=0x7f806c008040 comp rx=0 tx=0).stop 2023-10-30T13:43:57.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0c4a10 msgr2=0x7f807c15cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0c4a10 0x7f807c15cc30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f80780097c0 tx=0x7f807801bad0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 shutdown_connections 2023-10-30T13:43:57.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80600664a0 0x7f8060068960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c15a3b0 0x7f807c152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c153c80 0x7f807c1527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 --2- 172.21.15.31:0/2595988710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0c4a10 0x7f807c15cc30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 >> 172.21.15.31:0/2595988710 conn(0x7f807c0715c0 msgr2=0x7f807c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 shutdown_connections 2023-10-30T13:43:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.495+0000 7f808359d640 1 -- 172.21.15.31:0/2595988710 wait complete. 2023-10-30T13:43:57.516 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4282 2023-10-30T13:43:57.517 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2023-10-30T13:43:57.517 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-30T13:43:57.517 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T13:43:57.517 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T13:43:57.520 INFO:tasks.ceph.mds.b:Started 2023-10-30T13:43:57.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:57.602 INFO:tasks.ceph.mds.b.smithi033.stdout:starting mds.b at 2023-10-30T13:43:57.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.659+0000 7fb2069c5640 1 Processor -- start 2023-10-30T13:43:57.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.659+0000 7fb2069c5640 1 -- start start 2023-10-30T13:43:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb2069c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb2000701a0 0x7fb20015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb2069c5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb20006b2f0 0x7fb20006b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb2069c5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2000c2aa0 con 0x7fb2001537f0 2023-10-30T13:43:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb2069c5640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2000c3ab0 con 0x7fb20006b2f0 2023-10-30T13:43:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb2069c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20015a820 con 0x7fb2000701a0 2023-10-30T13:43:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb1fffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb2001537f0 0x7fb2000ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37164/0 (socket says 172.21.15.31:37164) 2023-10-30T13:43:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb1fffff640 1 -- 172.21.15.31:0/2806280323 learned_addr learned my addr 172.21.15.31:0/2806280323 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:43:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb204f62640 1 --2- 172.21.15.31:0/2806280323 >> v2:172.21.15.31:3300/0 conn(0x7fb20006b2f0 0x7fb20006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb1ff7fe640 1 --2- 172.21.15.31:0/2806280323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb2000701a0 0x7fb20015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb204f62640 1 -- 172.21.15.31:0/2806280323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb2000701a0 msgr2=0x7fb20015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb1feffd640 1 -- 172.21.15.31:0/2806280323 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091463782 0 0) 0x7fb2000c2aa0 con 0x7fb2001537f0 2023-10-30T13:43:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb204f62640 1 --2- 172.21.15.31:0/2806280323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb2000701a0 0x7fb20015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb204f62640 1 -- 172.21.15.31:0/2806280323 >> v1:172.21.15.31:6789/0 conn(0x7fb2001537f0 legacy=0x7fb2000ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.663+0000 7fb204f62640 1 -- 172.21.15.31:0/2806280323 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20006bca0 con 0x7fb20006b2f0 2023-10-30T13:43:57.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb204f62640 1 --2- 172.21.15.31:0/2806280323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb20006b2f0 0x7fb20006b760 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009d90 tx=0x7fb1f801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c746e48ae195612 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb1feffd640 1 -- 172.21.15.31:0/2806280323 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f80af600 con 0x7fb20006b2f0 2023-10-30T13:43:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb1feffd640 1 -- 172.21.15.31:0/2806280323 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f80ab050 con 0x7fb20006b2f0 2023-10-30T13:43:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb1feffd640 1 -- 172.21.15.31:0/2806280323 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f80af980 con 0x7fb20006b2f0 2023-10-30T13:43:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 -- 172.21.15.31:0/2806280323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb20006b2f0 msgr2=0x7fb20006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 --2- 172.21.15.31:0/2806280323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb20006b2f0 0x7fb20006b760 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009d90 tx=0x7fb1f801f130 comp rx=0 tx=0).stop 2023-10-30T13:43:57.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 -- 172.21.15.31:0/2806280323 shutdown_connections 2023-10-30T13:43:57.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 --2- 172.21.15.31:0/2806280323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb20006b2f0 0x7fb20006b760 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 --2- 172.21.15.31:0/2806280323 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb2000701a0 0x7fb20015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 -- 172.21.15.31:0/2806280323 >> 172.21.15.31:0/2806280323 conn(0x7fb200071fb0 msgr2=0x7fb2000723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:57.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 -- 172.21.15.31:0/2806280323 shutdown_connections 2023-10-30T13:43:57.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 -- 172.21.15.31:0/2806280323 wait complete. 2023-10-30T13:43:57.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 Processor -- start 2023-10-30T13:43:57.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.667+0000 7fb2069c5640 1 -- start start 2023-10-30T13:43:57.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb20006b2f0 0x7fb200150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2000701a0 0x7fb20014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2001537f0 0x7fb20014e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000c5e60 con 0x7fb2001537f0 2023-10-30T13:43:57.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000c68b0 con 0x7fb2000701a0 2023-10-30T13:43:57.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb200152820 con 0x7fb20006b2f0 2023-10-30T13:43:57.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2001537f0 0x7fb20014e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2001537f0 0x7fb20014e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44452/0 (socket says 172.21.15.31:44452) 2023-10-30T13:43:57.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 -- 172.21.15.31:0/635477094 learned_addr learned my addr 172.21.15.31:0/635477094 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:43:57.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb1fffff640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb20006b2f0 0x7fb200150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb1ff7fe640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2000701a0 0x7fb20014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 -- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb20006b2f0 msgr2=0x7fb200150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb20006b2f0 0x7fb200150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 -- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2000701a0 msgr2=0x7fb20014bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2000701a0 0x7fb20014bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 -- 172.21.15.31:0/635477094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8009a70 con 0x7fb2001537f0 2023-10-30T13:43:57.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb1ff7fe640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2000701a0 0x7fb20014bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:57.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb204f62640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2001537f0 0x7fb20014e430 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb1f801d7e0 tx=0x7fb1f801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:57.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb1fcff9640 1 -- 172.21.15.31:0/635477094 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8009450 con 0x7fb2001537f0 2023-10-30T13:43:57.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb1fcff9640 1 -- 172.21.15.31:0/635477094 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f80039a0 con 0x7fb2001537f0 2023-10-30T13:43:57.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb1fcff9640 1 -- 172.21.15.31:0/635477094 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8003d10 con 0x7fb2001537f0 2023-10-30T13:43:57.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2000c7230 con 0x7fb2001537f0 2023-10-30T13:43:57.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2000c93f0 con 0x7fb2001537f0 2023-10-30T13:43:57.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.671+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20004e170 con 0x7fb2001537f0 2023-10-30T13:43:57.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.675+0000 7fb1fcff9640 1 -- 172.21.15.31:0/635477094 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1f8004030 con 0x7fb2001537f0 2023-10-30T13:43:57.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.675+0000 7fb1fcff9640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1ec0664a0 0x7fb1ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:57.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.675+0000 7fb1fcff9640 1 -- 172.21.15.31:0/635477094 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fb1f8122980 con 0x7fb2001537f0 2023-10-30T13:43:57.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.675+0000 7fb1fffff640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1ec0664a0 0x7fb1ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:57.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.675+0000 7fb1fffff640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1ec0664a0 0x7fb1ec068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0007fd0 tx=0x7fb1f0007d60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:57.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.679+0000 7fb1fcff9640 1 -- 172.21.15.31:0/635477094 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1f80efd40 con 0x7fb2001537f0 2023-10-30T13:43:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.867+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2000c4ac0 con 0x7fb2001537f0 2023-10-30T13:43:57.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.867+0000 7fb1fcff9640 1 -- 172.21.15.31:0/635477094 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5699 (secure 0 0 0) 0x7fb1f80b4070 con 0x7fb2001537f0 2023-10-30T13:43:57.871 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:43:57.871 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:57.871 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 52 2023-10-30T13:43:57.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1ec0664a0 msgr2=0x7fb1ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1ec0664a0 0x7fb1ec068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0007fd0 tx=0x7fb1f0007d60 comp rx=0 tx=0).stop 2023-10-30T13:43:57.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2001537f0 msgr2=0x7fb20014e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:57.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2001537f0 0x7fb20014e430 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb1f801d7e0 tx=0x7fb1f801bed0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 shutdown_connections 2023-10-30T13:43:57.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb1ec0664a0 0x7fb1ec068960 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2001537f0 0x7fb20014e430 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2000701a0 0x7fb20014bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 --2- 172.21.15.31:0/635477094 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb20006b2f0 0x7fb200150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:57.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 >> 172.21.15.31:0/635477094 conn(0x7fb200071fb0 msgr2=0x7fb20015a070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:57.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 shutdown_connections 2023-10-30T13:43:57.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:43:57.871+0000 7fb2069c5640 1 -- 172.21.15.31:0/635477094 wait complete. 2023-10-30T13:43:57.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-30T13:43:59.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf346b640 1 Processor -- start 2023-10-30T13:44:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf346b640 1 -- start start 2023-10-30T13:44:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf346b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4bec153c80 0x7f4bec15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf346b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:00.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf346b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec159f20 con 0x7f4bec0c4a10 2023-10-30T13:44:00.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf346b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec15dec0 con 0x7f4bec153c80 2023-10-30T13:44:00.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf346b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec1599a0 con 0x7f4bec0c42a0 2023-10-30T13:44:00.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf1207640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4bec153c80 0x7f4bec15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:00.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf1207640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4bec153c80 0x7f4bec15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44466/0 (socket says 172.21.15.31:44466) 2023-10-30T13:44:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf0a06640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f4bec0c4a10 0x7f4bec15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37178/0 (socket says 172.21.15.31:37178) 2023-10-30T13:44:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf1207640 1 -- 172.21.15.31:0/1662389764 learned_addr learned my addr 172.21.15.31:0/1662389764 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.047+0000 7f4bf1a08640 1 --2- 172.21.15.31:0/1662389764 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf1207640 1 -- 172.21.15.31:0/1662389764 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 msgr2=0x7f4bec0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf1207640 1 --2- 172.21.15.31:0/1662389764 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf1207640 1 -- 172.21.15.31:0/1662389764 >> v1:172.21.15.31:6789/0 conn(0x7f4bec0c4a10 legacy=0x7f4bec15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:00.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf1207640 1 -- 172.21.15.31:0/1662389764 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec15d620 con 0x7f4bec153c80 2023-10-30T13:44:00.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf1207640 1 --2- 172.21.15.31:0/1662389764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec153c80 0x7f4bec15cf90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009870 tx=0x7f4be801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d009f5d43baff12a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:00.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4be3fff640 1 -- 172.21.15.31:0/1662389764 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be80a8a30 con 0x7f4bec153c80 2023-10-30T13:44:00.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4be3fff640 1 -- 172.21.15.31:0/1662389764 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be80a8b90 con 0x7f4bec153c80 2023-10-30T13:44:00.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4be3fff640 1 -- 172.21.15.31:0/1662389764 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be80b0440 con 0x7f4bec153c80 2023-10-30T13:44:00.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf346b640 1 -- 172.21.15.31:0/1662389764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec153c80 msgr2=0x7f4bec15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:00.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf346b640 1 --2- 172.21.15.31:0/1662389764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec153c80 0x7f4bec15cf90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009870 tx=0x7f4be801ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf346b640 1 -- 172.21.15.31:0/1662389764 shutdown_connections 2023-10-30T13:44:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf346b640 1 --2- 172.21.15.31:0/1662389764 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf346b640 1 --2- 172.21.15.31:0/1662389764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec153c80 0x7f4bec15cf90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.051+0000 7f4bf346b640 1 -- 172.21.15.31:0/1662389764 >> 172.21.15.31:0/1662389764 conn(0x7f4bec0715c0 msgr2=0x7f4bec0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 -- 172.21.15.31:0/1662389764 shutdown_connections 2023-10-30T13:44:00.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 -- 172.21.15.31:0/1662389764 wait complete. 2023-10-30T13:44:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 Processor -- start 2023-10-30T13:44:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 -- start start 2023-10-30T13:44:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec0c4a10 0x7f4bec14f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bec153c80 0x7f4bec1511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:00.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec1f3320 con 0x7f4bec0c4a10 2023-10-30T13:44:00.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec1556d0 con 0x7f4bec153c80 2023-10-30T13:44:00.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf346b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec157ea0 con 0x7f4bec0c42a0 2023-10-30T13:44:00.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec0c4a10 0x7f4bec14f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:00.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec0c4a10 0x7f4bec14f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44476/0 (socket says 172.21.15.31:44476) 2023-10-30T13:44:00.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 -- 172.21.15.31:0/4239410366 learned_addr learned my addr 172.21.15.31:0/4239410366 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:00.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf1207640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:00.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf1a08640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bec153c80 0x7f4bec1511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:00.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 -- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 msgr2=0x7f4bec155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:00.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 -- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bec153c80 msgr2=0x7f4bec1511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bec153c80 0x7f4bec1511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 -- 172.21.15.31:0/4239410366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be80094d0 con 0x7f4bec0c4a10 2023-10-30T13:44:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.055+0000 7f4bf0a06640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec0c4a10 0x7f4bec14f780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00c500 tx=0x7f4bdc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4be1ffb640 1 -- 172.21.15.31:0/4239410366 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc007500 con 0x7f4bec0c4a10 2023-10-30T13:44:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4be1ffb640 1 -- 172.21.15.31:0/4239410366 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc007660 con 0x7f4bec0c4a10 2023-10-30T13:44:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4be1ffb640 1 -- 172.21.15.31:0/4239410366 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc004a10 con 0x7f4bec0c4a10 2023-10-30T13:44:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec1f2430 con 0x7f4bec0c4a10 2023-10-30T13:44:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec1f29f0 con 0x7f4bec0c4a10 2023-10-30T13:44:00.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec04e170 con 0x7f4bec0c4a10 2023-10-30T13:44:00.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4be1ffb640 1 -- 172.21.15.31:0/4239410366 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bdc004b70 con 0x7f4bec0c4a10 2023-10-30T13:44:00.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4be1ffb640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bd00664a0 0x7f4bd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:00.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4be1ffb640 1 -- 172.21.15.31:0/4239410366 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f4bdc083210 con 0x7f4bec0c4a10 2023-10-30T13:44:00.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.059+0000 7f4bf1207640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bd00664a0 0x7f4bd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:00.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.063+0000 7f4bf1207640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bd00664a0 0x7f4bd0068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4be80b2000 tx=0x7f4be8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:00.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.067+0000 7f4be1ffb640 1 -- 172.21.15.31:0/4239410366 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bdc0505e0 con 0x7f4bec0c4a10 2023-10-30T13:44:00.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bec150f10 con 0x7f4bec0c4a10 2023-10-30T13:44:00.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4be1ffb640 1 -- 172.21.15.31:0/4239410366 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7f4bdc053c00 con 0x7f4bec0c4a10 2023-10-30T13:44:00.258 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:00.258 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:00.259 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:00.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bd00664a0 msgr2=0x7f4bd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:00.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bd00664a0 0x7f4bd0068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4be80b2000 tx=0x7f4be8009040 comp rx=0 tx=0).stop 2023-10-30T13:44:00.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec0c4a10 msgr2=0x7f4bec14f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:00.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec0c4a10 0x7f4bec14f780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00c500 tx=0x7f4bdc00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 shutdown_connections 2023-10-30T13:44:00.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bd00664a0 0x7f4bd0068960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bec153c80 0x7f4bec1511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bec0c4a10 0x7f4bec14f780 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 --2- 172.21.15.31:0/4239410366 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bec0c42a0 0x7f4bec155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:00.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 >> 172.21.15.31:0/4239410366 conn(0x7f4bec0715c0 msgr2=0x7f4bec157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:00.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 shutdown_connections 2023-10-30T13:44:00.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:00.255+0000 7f4bf346b640 1 -- 172.21.15.31:0/4239410366 wait complete. 2023-10-30T13:44:00.279 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-30T13:44:00.279 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-10-30T13:44:02.034 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/29' 2023-10-30T13:44:02.034 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/32' 2023-10-30T13:44:02.034 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/22' 2023-10-30T13:44:02.035 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/25' 2023-10-30T13:44:02.035 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/30' 2023-10-30T13:44:02.035 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/33' 2023-10-30T13:44:02.035 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/23' 2023-10-30T13:44:02.035 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/35' 2023-10-30T13:44:02.036 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/26' 2023-10-30T13:44:02.036 INFO:tasks.workunit.client.0.smithi031.stdout:removed './226/37' 2023-10-30T13:44:02.036 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './226' 2023-10-30T13:44:02.036 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/21' 2023-10-30T13:44:02.036 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/27' 2023-10-30T13:44:02.037 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/31' 2023-10-30T13:44:02.037 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/36' 2023-10-30T13:44:02.037 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/20' 2023-10-30T13:44:02.037 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/39' 2023-10-30T13:44:02.037 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/34' 2023-10-30T13:44:02.037 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/38' 2023-10-30T13:44:02.038 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/28' 2023-10-30T13:44:02.038 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/24' 2023-10-30T13:44:02.038 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/29' 2023-10-30T13:44:02.038 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/32' 2023-10-30T13:44:02.038 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/22' 2023-10-30T13:44:02.039 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/25' 2023-10-30T13:44:02.039 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/30' 2023-10-30T13:44:02.039 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/33' 2023-10-30T13:44:02.039 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/23' 2023-10-30T13:44:02.039 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/35' 2023-10-30T13:44:02.039 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/26' 2023-10-30T13:44:02.040 INFO:tasks.workunit.client.0.smithi031.stdout:removed './227/37' 2023-10-30T13:44:02.040 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './227' 2023-10-30T13:44:02.040 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/21' 2023-10-30T13:44:02.040 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/27' 2023-10-30T13:44:02.040 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/31' 2023-10-30T13:44:02.041 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/36' 2023-10-30T13:44:02.041 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/20' 2023-10-30T13:44:02.041 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/39' 2023-10-30T13:44:02.041 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/34' 2023-10-30T13:44:02.041 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/38' 2023-10-30T13:44:02.041 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/28' 2023-10-30T13:44:02.042 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/24' 2023-10-30T13:44:02.042 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/29' 2023-10-30T13:44:02.042 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/32' 2023-10-30T13:44:02.042 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/22' 2023-10-30T13:44:02.042 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/25' 2023-10-30T13:44:02.043 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/30' 2023-10-30T13:44:02.043 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/33' 2023-10-30T13:44:02.043 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/23' 2023-10-30T13:44:02.043 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/35' 2023-10-30T13:44:02.043 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/26' 2023-10-30T13:44:02.043 INFO:tasks.workunit.client.0.smithi031.stdout:removed './228/37' 2023-10-30T13:44:02.044 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './228' 2023-10-30T13:44:02.044 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/21' 2023-10-30T13:44:02.044 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/27' 2023-10-30T13:44:02.044 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/31' 2023-10-30T13:44:02.044 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/36' 2023-10-30T13:44:02.045 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/20' 2023-10-30T13:44:02.045 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/39' 2023-10-30T13:44:02.045 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/34' 2023-10-30T13:44:02.045 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/38' 2023-10-30T13:44:02.045 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/28' 2023-10-30T13:44:02.045 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/24' 2023-10-30T13:44:02.046 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/29' 2023-10-30T13:44:02.046 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/32' 2023-10-30T13:44:02.046 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/22' 2023-10-30T13:44:02.046 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/25' 2023-10-30T13:44:02.046 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/30' 2023-10-30T13:44:02.047 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/33' 2023-10-30T13:44:02.047 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/23' 2023-10-30T13:44:02.047 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/35' 2023-10-30T13:44:02.047 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/26' 2023-10-30T13:44:02.047 INFO:tasks.workunit.client.0.smithi031.stdout:removed './229/37' 2023-10-30T13:44:02.048 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './229' 2023-10-30T13:44:02.048 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/21' 2023-10-30T13:44:02.048 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/27' 2023-10-30T13:44:02.048 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/31' 2023-10-30T13:44:02.048 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/36' 2023-10-30T13:44:02.049 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/20' 2023-10-30T13:44:02.049 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/39' 2023-10-30T13:44:02.049 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/34' 2023-10-30T13:44:02.049 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/38' 2023-10-30T13:44:02.049 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/28' 2023-10-30T13:44:02.049 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/24' 2023-10-30T13:44:02.050 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/29' 2023-10-30T13:44:02.050 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/32' 2023-10-30T13:44:02.050 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/22' 2023-10-30T13:44:02.050 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/25' 2023-10-30T13:44:02.050 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/30' 2023-10-30T13:44:02.051 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/33' 2023-10-30T13:44:02.051 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/23' 2023-10-30T13:44:02.051 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/35' 2023-10-30T13:44:02.051 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/26' 2023-10-30T13:44:02.051 INFO:tasks.workunit.client.0.smithi031.stdout:removed './230/37' 2023-10-30T13:44:02.052 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './230' 2023-10-30T13:44:02.052 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/21' 2023-10-30T13:44:02.052 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/27' 2023-10-30T13:44:02.052 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/31' 2023-10-30T13:44:02.052 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/36' 2023-10-30T13:44:02.052 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/20' 2023-10-30T13:44:02.053 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/39' 2023-10-30T13:44:02.053 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/34' 2023-10-30T13:44:02.053 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/38' 2023-10-30T13:44:02.053 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/28' 2023-10-30T13:44:02.053 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/24' 2023-10-30T13:44:02.053 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/29' 2023-10-30T13:44:02.054 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/32' 2023-10-30T13:44:02.054 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/22' 2023-10-30T13:44:02.054 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/25' 2023-10-30T13:44:02.054 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/30' 2023-10-30T13:44:02.054 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/33' 2023-10-30T13:44:02.055 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/23' 2023-10-30T13:44:02.055 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/35' 2023-10-30T13:44:02.055 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/26' 2023-10-30T13:44:02.055 INFO:tasks.workunit.client.0.smithi031.stdout:removed './231/37' 2023-10-30T13:44:02.055 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './231' 2023-10-30T13:44:02.056 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/21' 2023-10-30T13:44:02.056 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/27' 2023-10-30T13:44:02.056 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/31' 2023-10-30T13:44:02.056 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/36' 2023-10-30T13:44:02.056 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/20' 2023-10-30T13:44:02.057 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/39' 2023-10-30T13:44:02.057 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/34' 2023-10-30T13:44:02.057 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/38' 2023-10-30T13:44:02.057 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/28' 2023-10-30T13:44:02.057 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/24' 2023-10-30T13:44:02.058 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/29' 2023-10-30T13:44:02.058 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/32' 2023-10-30T13:44:02.058 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/22' 2023-10-30T13:44:02.058 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/25' 2023-10-30T13:44:02.058 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/30' 2023-10-30T13:44:02.058 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/33' 2023-10-30T13:44:02.059 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/23' 2023-10-30T13:44:02.059 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/35' 2023-10-30T13:44:02.059 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/26' 2023-10-30T13:44:02.059 INFO:tasks.workunit.client.0.smithi031.stdout:removed './232/37' 2023-10-30T13:44:02.059 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './232' 2023-10-30T13:44:02.060 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/21' 2023-10-30T13:44:02.060 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/27' 2023-10-30T13:44:02.060 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/31' 2023-10-30T13:44:02.060 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/36' 2023-10-30T13:44:02.060 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/20' 2023-10-30T13:44:02.060 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/39' 2023-10-30T13:44:02.061 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/34' 2023-10-30T13:44:02.061 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/38' 2023-10-30T13:44:02.061 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/28' 2023-10-30T13:44:02.061 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/24' 2023-10-30T13:44:02.061 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/29' 2023-10-30T13:44:02.061 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/32' 2023-10-30T13:44:02.062 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/22' 2023-10-30T13:44:02.062 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/25' 2023-10-30T13:44:02.062 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/30' 2023-10-30T13:44:02.062 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/33' 2023-10-30T13:44:02.063 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/23' 2023-10-30T13:44:02.063 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/35' 2023-10-30T13:44:02.063 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/26' 2023-10-30T13:44:02.063 INFO:tasks.workunit.client.0.smithi031.stdout:removed './233/37' 2023-10-30T13:44:02.063 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './233' 2023-10-30T13:44:02.064 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/21' 2023-10-30T13:44:02.064 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/27' 2023-10-30T13:44:02.064 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/31' 2023-10-30T13:44:02.064 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/36' 2023-10-30T13:44:02.064 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/20' 2023-10-30T13:44:02.064 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/39' 2023-10-30T13:44:02.065 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/34' 2023-10-30T13:44:02.065 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/38' 2023-10-30T13:44:02.065 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/28' 2023-10-30T13:44:02.065 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/24' 2023-10-30T13:44:02.065 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/29' 2023-10-30T13:44:02.065 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/32' 2023-10-30T13:44:02.066 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/22' 2023-10-30T13:44:02.066 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/25' 2023-10-30T13:44:02.066 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/30' 2023-10-30T13:44:02.066 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/33' 2023-10-30T13:44:02.066 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/23' 2023-10-30T13:44:02.067 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/35' 2023-10-30T13:44:02.067 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/26' 2023-10-30T13:44:02.067 INFO:tasks.workunit.client.0.smithi031.stdout:removed './234/37' 2023-10-30T13:44:02.067 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './234' 2023-10-30T13:44:02.067 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/21' 2023-10-30T13:44:02.067 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/27' 2023-10-30T13:44:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/31' 2023-10-30T13:44:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/36' 2023-10-30T13:44:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/20' 2023-10-30T13:44:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/39' 2023-10-30T13:44:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/34' 2023-10-30T13:44:02.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/38' 2023-10-30T13:44:02.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/28' 2023-10-30T13:44:02.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/24' 2023-10-30T13:44:02.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/29' 2023-10-30T13:44:02.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/32' 2023-10-30T13:44:02.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/22' 2023-10-30T13:44:02.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/25' 2023-10-30T13:44:02.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/30' 2023-10-30T13:44:02.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/33' 2023-10-30T13:44:02.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/23' 2023-10-30T13:44:02.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/35' 2023-10-30T13:44:02.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/26' 2023-10-30T13:44:02.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './235/37' 2023-10-30T13:44:02.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './235' 2023-10-30T13:44:02.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/21' 2023-10-30T13:44:02.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/27' 2023-10-30T13:44:02.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/31' 2023-10-30T13:44:02.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/36' 2023-10-30T13:44:02.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/20' 2023-10-30T13:44:02.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/39' 2023-10-30T13:44:02.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/34' 2023-10-30T13:44:02.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/38' 2023-10-30T13:44:02.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/28' 2023-10-30T13:44:02.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/24' 2023-10-30T13:44:02.073 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/29' 2023-10-30T13:44:02.073 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/32' 2023-10-30T13:44:03.165 INFO:tasks.ceph.mds.d.smithi033.stderr:2023-10-30T13:44:03.162+0000 7f5d368d0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:44:04.170 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/22' 2023-10-30T13:44:04.170 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/25' 2023-10-30T13:44:04.170 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/30' 2023-10-30T13:44:04.170 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/33' 2023-10-30T13:44:04.171 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/23' 2023-10-30T13:44:04.171 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/35' 2023-10-30T13:44:04.171 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/26' 2023-10-30T13:44:04.171 INFO:tasks.workunit.client.0.smithi031.stdout:removed './236/37' 2023-10-30T13:44:04.171 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './236' 2023-10-30T13:44:04.172 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/21' 2023-10-30T13:44:04.172 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/27' 2023-10-30T13:44:04.172 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/31' 2023-10-30T13:44:04.172 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/36' 2023-10-30T13:44:04.172 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/20' 2023-10-30T13:44:04.173 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/39' 2023-10-30T13:44:04.173 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/34' 2023-10-30T13:44:04.173 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/38' 2023-10-30T13:44:04.173 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/28' 2023-10-30T13:44:04.173 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/24' 2023-10-30T13:44:04.173 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/29' 2023-10-30T13:44:04.174 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/32' 2023-10-30T13:44:04.174 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/22' 2023-10-30T13:44:04.174 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/25' 2023-10-30T13:44:04.174 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/30' 2023-10-30T13:44:04.174 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/33' 2023-10-30T13:44:04.175 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/23' 2023-10-30T13:44:04.175 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/35' 2023-10-30T13:44:04.175 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/26' 2023-10-30T13:44:04.175 INFO:tasks.workunit.client.0.smithi031.stdout:removed './237/37' 2023-10-30T13:44:04.175 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './237' 2023-10-30T13:44:04.175 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/21' 2023-10-30T13:44:04.176 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/27' 2023-10-30T13:44:04.176 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/31' 2023-10-30T13:44:04.176 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/36' 2023-10-30T13:44:04.176 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/20' 2023-10-30T13:44:04.176 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/39' 2023-10-30T13:44:04.177 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/34' 2023-10-30T13:44:04.177 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/38' 2023-10-30T13:44:04.177 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/28' 2023-10-30T13:44:04.177 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/24' 2023-10-30T13:44:04.177 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/29' 2023-10-30T13:44:04.177 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/32' 2023-10-30T13:44:04.178 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/22' 2023-10-30T13:44:04.178 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/25' 2023-10-30T13:44:04.178 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/30' 2023-10-30T13:44:04.178 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/33' 2023-10-30T13:44:04.178 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/23' 2023-10-30T13:44:04.179 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/35' 2023-10-30T13:44:04.179 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/26' 2023-10-30T13:44:04.179 INFO:tasks.workunit.client.0.smithi031.stdout:removed './238/37' 2023-10-30T13:44:04.179 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './238' 2023-10-30T13:44:04.179 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/21' 2023-10-30T13:44:04.179 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/27' 2023-10-30T13:44:04.180 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/31' 2023-10-30T13:44:04.180 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/36' 2023-10-30T13:44:04.180 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/20' 2023-10-30T13:44:04.180 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/39' 2023-10-30T13:44:04.180 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/34' 2023-10-30T13:44:04.180 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/38' 2023-10-30T13:44:04.181 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/28' 2023-10-30T13:44:04.181 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/24' 2023-10-30T13:44:04.181 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/29' 2023-10-30T13:44:04.181 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/32' 2023-10-30T13:44:04.181 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/22' 2023-10-30T13:44:04.181 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/25' 2023-10-30T13:44:04.182 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/30' 2023-10-30T13:44:04.182 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/33' 2023-10-30T13:44:04.182 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/23' 2023-10-30T13:44:04.182 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/35' 2023-10-30T13:44:04.182 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/26' 2023-10-30T13:44:04.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed './239/37' 2023-10-30T13:44:04.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './239' 2023-10-30T13:44:04.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/21' 2023-10-30T13:44:04.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/27' 2023-10-30T13:44:04.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/31' 2023-10-30T13:44:04.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/36' 2023-10-30T13:44:04.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/20' 2023-10-30T13:44:04.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/39' 2023-10-30T13:44:04.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/34' 2023-10-30T13:44:04.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/38' 2023-10-30T13:44:04.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/28' 2023-10-30T13:44:04.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/24' 2023-10-30T13:44:04.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/29' 2023-10-30T13:44:04.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/32' 2023-10-30T13:44:04.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/22' 2023-10-30T13:44:04.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/25' 2023-10-30T13:44:04.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/30' 2023-10-30T13:44:04.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/33' 2023-10-30T13:44:04.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/23' 2023-10-30T13:44:04.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/35' 2023-10-30T13:44:04.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/26' 2023-10-30T13:44:04.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './240/37' 2023-10-30T13:44:04.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './240' 2023-10-30T13:44:04.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/21' 2023-10-30T13:44:04.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/27' 2023-10-30T13:44:04.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/31' 2023-10-30T13:44:04.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/36' 2023-10-30T13:44:04.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/20' 2023-10-30T13:44:04.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/39' 2023-10-30T13:44:04.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/34' 2023-10-30T13:44:04.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/38' 2023-10-30T13:44:04.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/28' 2023-10-30T13:44:04.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/24' 2023-10-30T13:44:04.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/29' 2023-10-30T13:44:04.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/32' 2023-10-30T13:44:04.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/22' 2023-10-30T13:44:04.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/25' 2023-10-30T13:44:04.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/30' 2023-10-30T13:44:04.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/33' 2023-10-30T13:44:04.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/23' 2023-10-30T13:44:04.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/35' 2023-10-30T13:44:04.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/26' 2023-10-30T13:44:04.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './241/37' 2023-10-30T13:44:04.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './241' 2023-10-30T13:44:04.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/21' 2023-10-30T13:44:04.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/27' 2023-10-30T13:44:04.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/31' 2023-10-30T13:44:04.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/36' 2023-10-30T13:44:04.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/20' 2023-10-30T13:44:04.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/39' 2023-10-30T13:44:04.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/34' 2023-10-30T13:44:04.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/38' 2023-10-30T13:44:04.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/28' 2023-10-30T13:44:04.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/24' 2023-10-30T13:44:04.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/29' 2023-10-30T13:44:04.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/32' 2023-10-30T13:44:04.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/22' 2023-10-30T13:44:04.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/25' 2023-10-30T13:44:04.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/30' 2023-10-30T13:44:04.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/33' 2023-10-30T13:44:04.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/23' 2023-10-30T13:44:04.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/35' 2023-10-30T13:44:04.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/26' 2023-10-30T13:44:04.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './242/37' 2023-10-30T13:44:04.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './242' 2023-10-30T13:44:04.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/21' 2023-10-30T13:44:04.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/27' 2023-10-30T13:44:04.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/31' 2023-10-30T13:44:04.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/36' 2023-10-30T13:44:04.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/20' 2023-10-30T13:44:04.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/39' 2023-10-30T13:44:04.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/34' 2023-10-30T13:44:04.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/38' 2023-10-30T13:44:04.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/28' 2023-10-30T13:44:04.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/24' 2023-10-30T13:44:04.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/29' 2023-10-30T13:44:04.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/32' 2023-10-30T13:44:04.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/22' 2023-10-30T13:44:04.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/25' 2023-10-30T13:44:04.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/30' 2023-10-30T13:44:04.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/33' 2023-10-30T13:44:04.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/23' 2023-10-30T13:44:04.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/35' 2023-10-30T13:44:04.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/26' 2023-10-30T13:44:04.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './243/37' 2023-10-30T13:44:04.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './243' 2023-10-30T13:44:04.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/21' 2023-10-30T13:44:04.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/27' 2023-10-30T13:44:04.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/31' 2023-10-30T13:44:04.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/36' 2023-10-30T13:44:04.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/20' 2023-10-30T13:44:04.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/39' 2023-10-30T13:44:04.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/34' 2023-10-30T13:44:04.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/38' 2023-10-30T13:44:04.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/28' 2023-10-30T13:44:04.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/24' 2023-10-30T13:44:04.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/29' 2023-10-30T13:44:04.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/32' 2023-10-30T13:44:04.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/22' 2023-10-30T13:44:04.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/25' 2023-10-30T13:44:04.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/30' 2023-10-30T13:44:04.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/33' 2023-10-30T13:44:04.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/23' 2023-10-30T13:44:04.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/35' 2023-10-30T13:44:04.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/26' 2023-10-30T13:44:04.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './244/37' 2023-10-30T13:44:04.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './244' 2023-10-30T13:44:04.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/21' 2023-10-30T13:44:04.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/27' 2023-10-30T13:44:04.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/31' 2023-10-30T13:44:04.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/36' 2023-10-30T13:44:04.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/20' 2023-10-30T13:44:04.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/39' 2023-10-30T13:44:04.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/34' 2023-10-30T13:44:04.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/38' 2023-10-30T13:44:04.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/28' 2023-10-30T13:44:04.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/24' 2023-10-30T13:44:04.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/29' 2023-10-30T13:44:04.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/32' 2023-10-30T13:44:04.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/22' 2023-10-30T13:44:04.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/25' 2023-10-30T13:44:04.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/30' 2023-10-30T13:44:04.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/33' 2023-10-30T13:44:04.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/23' 2023-10-30T13:44:04.205 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/35' 2023-10-30T13:44:04.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/26' 2023-10-30T13:44:04.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './245/37' 2023-10-30T13:44:04.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './245' 2023-10-30T13:44:04.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/21' 2023-10-30T13:44:04.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/27' 2023-10-30T13:44:04.206 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/31' 2023-10-30T13:44:04.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/36' 2023-10-30T13:44:04.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/20' 2023-10-30T13:44:04.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/39' 2023-10-30T13:44:04.207 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/34' 2023-10-30T13:44:04.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/38' 2023-10-30T13:44:04.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/28' 2023-10-30T13:44:04.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/24' 2023-10-30T13:44:04.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/29' 2023-10-30T13:44:04.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/32' 2023-10-30T13:44:04.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/22' 2023-10-30T13:44:07.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:07.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c14f3e640 1 Processor -- start 2023-10-30T13:44:07.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c14f3e640 1 -- start start 2023-10-30T13:44:07.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c14f3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4c100cb7d0 0x7f4c100c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c14f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c14f3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100c2310 con 0x7f4c10150830 2023-10-30T13:44:07.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c14f3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1006bc10 con 0x7f4c100cb7d0 2023-10-30T13:44:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c14f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10156710 con 0x7f4c100c3e40 2023-10-30T13:44:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0e59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f4c10150830 0x7f4c1006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:39168/0 (socket says 172.21.15.31:39168) 2023-10-30T13:44:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0e59c640 1 -- 172.21.15.31:0/2687791506 learned_addr learned my addr 172.21.15.31:0/2687791506 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0dd9b640 1 --2- 172.21.15.31:0/2687791506 >> v2:172.21.15.31:3300/0 conn(0x7f4c100cb7d0 0x7f4c100c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0ed9d640 1 --2- 172.21.15.31:0/2687791506 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0d59a640 1 -- 172.21.15.31:0/2687791506 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525266851 0 0) 0x7f4c100c2310 con 0x7f4c10150830 2023-10-30T13:44:07.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0d59a640 1 -- 172.21.15.31:0/2687791506 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf8005740 con 0x7f4c10150830 2023-10-30T13:44:07.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0dd9b640 1 -- 172.21.15.31:0/2687791506 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 msgr2=0x7f4c10070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:07.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0dd9b640 1 --2- 172.21.15.31:0/2687791506 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0dd9b640 1 -- 172.21.15.31:0/2687791506 >> v1:172.21.15.31:6789/0 conn(0x7f4c10150830 legacy=0x7f4c1006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0dd9b640 1 -- 172.21.15.31:0/2687791506 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100c4360 con 0x7f4c100cb7d0 2023-10-30T13:44:07.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.435+0000 7f4c0ed9d640 1 --2- 172.21.15.31:0/2687791506 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c0dd9b640 1 --2- 172.21.15.31:0/2687791506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c100c3900 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4c040097c0 tx=0x7f4c0401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e9bc8dcf0d8641 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c0d59a640 1 -- 172.21.15.31:0/2687791506 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040a8970 con 0x7f4c100cb7d0 2023-10-30T13:44:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c0d59a640 1 -- 172.21.15.31:0/2687791506 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c040a8ad0 con 0x7f4c100cb7d0 2023-10-30T13:44:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c0d59a640 1 -- 172.21.15.31:0/2687791506 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040a8e60 con 0x7f4c100cb7d0 2023-10-30T13:44:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2687791506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 msgr2=0x7f4c100c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2687791506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c100c3900 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4c040097c0 tx=0x7f4c0401ec70 comp rx=0 tx=0).stop 2023-10-30T13:44:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2687791506 shutdown_connections 2023-10-30T13:44:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2687791506 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2687791506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c100c3900 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2687791506 >> 172.21.15.31:0/2687791506 conn(0x7f4c100cac70 msgr2=0x7f4c100cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2687791506 shutdown_connections 2023-10-30T13:44:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.439+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2687791506 wait complete. 2023-10-30T13:44:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 Processor -- start 2023-10-30T13:44:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 -- start start 2023-10-30T13:44:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c10154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c10150830 0x7f4c100c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101ee560 con 0x7f4c100cb7d0 2023-10-30T13:44:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101eea70 con 0x7f4c10150830 2023-10-30T13:44:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c14f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101eb520 con 0x7f4c100c3e40 2023-10-30T13:44:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c10154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c10154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46020/0 (socket says 172.21.15.31:46020) 2023-10-30T13:44:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 -- 172.21.15.31:0/2706784830 learned_addr learned my addr 172.21.15.31:0/2706784830 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0e59c640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0ed9d640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c10150830 0x7f4c100c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 -- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 msgr2=0x7f4c10159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 -- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c10150830 msgr2=0x7f4c100c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c10150830 0x7f4c100c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 -- 172.21.15.31:0/2706784830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04009420 con 0x7f4c100cb7d0 2023-10-30T13:44:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0ed9d640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c10150830 0x7f4c100c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.443+0000 7f4c0dd9b640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c10154a60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4c040b3040 tx=0x7f4c0401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4bfeffd640 1 -- 172.21.15.31:0/2706784830 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040b9630 con 0x7f4c100cb7d0 2023-10-30T13:44:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4bfeffd640 1 -- 172.21.15.31:0/2706784830 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c040b97b0 con 0x7f4c100cb7d0 2023-10-30T13:44:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4bfeffd640 1 -- 172.21.15.31:0/2706784830 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040b9b60 con 0x7f4c100cb7d0 2023-10-30T13:44:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c101ee790 con 0x7f4c100cb7d0 2023-10-30T13:44:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c101eaa40 con 0x7f4c100cb7d0 2023-10-30T13:44:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c10066510 con 0x7f4c100cb7d0 2023-10-30T13:44:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4bfeffd640 1 -- 172.21.15.31:0/2706784830 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c040b7040 con 0x7f4c100cb7d0 2023-10-30T13:44:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4bfeffd640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bf00664d0 0x7f4bf0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4bfeffd640 1 -- 172.21.15.31:0/2706784830 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f4c04121fc0 con 0x7f4c100cb7d0 2023-10-30T13:44:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.447+0000 7f4c0e59c640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bf00664d0 0x7f4bf0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.451+0000 7f4c0e59c640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bf00664d0 0x7f4bf0068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4c000011f0 tx=0x7f4c00007d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.451+0000 7f4bfeffd640 1 -- 172.21.15.31:0/2706784830 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c040ef1b0 con 0x7f4c100cb7d0 2023-10-30T13:44:07.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.639+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c10156570 con 0x7f4c100cb7d0 2023-10-30T13:44:07.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.639+0000 7f4bfeffd640 1 -- 172.21.15.31:0/2706784830 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7f4c10156570 con 0x7f4c100cb7d0 2023-10-30T13:44:07.643 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:07.643 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:07.644 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:07.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bf00664d0 msgr2=0x7f4bf0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bf00664d0 0x7f4bf0068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4c000011f0 tx=0x7f4c00007d40 comp rx=0 tx=0).stop 2023-10-30T13:44:07.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 msgr2=0x7f4c10154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c10154a60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4c040b3040 tx=0x7f4c0401ba10 comp rx=0 tx=0).stop 2023-10-30T13:44:07.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 shutdown_connections 2023-10-30T13:44:07.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bf00664d0 0x7f4bf0068990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c10150830 0x7f4c100c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c100cb7d0 0x7f4c10154a60 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 --2- 172.21.15.31:0/2706784830 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c100c3e40 0x7f4c10159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 >> 172.21.15.31:0/2706784830 conn(0x7f4c100cac70 msgr2=0x7f4c1006c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:07.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 shutdown_connections 2023-10-30T13:44:07.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.643+0000 7f4c14f3e640 1 -- 172.21.15.31:0/2706784830 wait complete. 2023-10-30T13:44:07.664 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:44:07.665 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:44:07.665 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:44:07.682 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:44:07.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb283394640 1 Processor -- start 2023-10-30T13:44:07.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb283394640 1 -- start start 2023-10-30T13:44:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb283394640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb283394640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb27c0c4a10 0x7fb27c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb283394640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c159f20 con 0x7fb27c15a3b0 2023-10-30T13:44:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb283394640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c15dec0 con 0x7fb27c0c4a10 2023-10-30T13:44:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb283394640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1599a0 con 0x7fb27c153c80 2023-10-30T13:44:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb28092f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb27c0c4a10 0x7fb27c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb28092f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb27c0c4a10 0x7fb27c0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46028/0 (socket says 172.21.15.31:46028) 2023-10-30T13:44:07.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb28092f640 1 -- 172.21.15.31:0/2763084771 learned_addr learned my addr 172.21.15.31:0/2763084771 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:07.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb281130640 1 --2- 172.21.15.31:0/2763084771 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb28092f640 1 -- 172.21.15.31:0/2763084771 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 msgr2=0x7fb27c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb28092f640 1 --2- 172.21.15.31:0/2763084771 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb28092f640 1 -- 172.21.15.31:0/2763084771 >> v1:172.21.15.33:6789/0 conn(0x7fb27c15a3b0 legacy=0x7fb27c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.823+0000 7fb28092f640 1 -- 172.21.15.31:0/2763084771 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c0c47e0 con 0x7fb27c0c4a10 2023-10-30T13:44:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb28092f640 1 --2- 172.21.15.31:0/2763084771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c0c4a10 0x7fb27c0c42a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009870 tx=0x7fb26c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bd9cf4a42280e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:07.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb273fff640 1 -- 172.21.15.31:0/2763084771 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0a8a30 con 0x7fb27c0c4a10 2023-10-30T13:44:07.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb273fff640 1 -- 172.21.15.31:0/2763084771 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c0a8b90 con 0x7fb27c0c4a10 2023-10-30T13:44:07.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb273fff640 1 -- 172.21.15.31:0/2763084771 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0b0440 con 0x7fb27c0c4a10 2023-10-30T13:44:07.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 -- 172.21.15.31:0/2763084771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c0c4a10 msgr2=0x7fb27c0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 --2- 172.21.15.31:0/2763084771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c0c4a10 0x7fb27c0c42a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009870 tx=0x7fb26c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:07.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 -- 172.21.15.31:0/2763084771 shutdown_connections 2023-10-30T13:44:07.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 --2- 172.21.15.31:0/2763084771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c0c4a10 0x7fb27c0c42a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 --2- 172.21.15.31:0/2763084771 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 -- 172.21.15.31:0/2763084771 >> 172.21.15.31:0/2763084771 conn(0x7fb27c0715c0 msgr2=0x7fb27c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:07.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 -- 172.21.15.31:0/2763084771 shutdown_connections 2023-10-30T13:44:07.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.827+0000 7fb283394640 1 -- 172.21.15.31:0/2763084771 wait complete. 2023-10-30T13:44:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 Processor -- start 2023-10-30T13:44:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 -- start start 2023-10-30T13:44:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c0c4a10 0x7fb27c1f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c0c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c15a3b0 0x7fb27c14eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c15b330 con 0x7fb27c15a3b0 2023-10-30T13:44:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c15c930 con 0x7fb27c0c4a10 2023-10-30T13:44:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1f31b0 con 0x7fb27c153c80 2023-10-30T13:44:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c15a3b0 0x7fb27c14eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c15a3b0 0x7fb27c14eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46038/0 (socket says 172.21.15.31:46038) 2023-10-30T13:44:07.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 -- 172.21.15.31:0/3699179961 learned_addr learned my addr 172.21.15.31:0/3699179961 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:07.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb28092f640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c0c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281130640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c0c4a10 0x7fb27c1f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 -- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 msgr2=0x7fb27c0c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c0c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 -- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c0c4a10 msgr2=0x7fb27c1f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:07.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c0c4a10 0x7fb27c1f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:07.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 -- 172.21.15.31:0/3699179961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0094d0 con 0x7fb27c15a3b0 2023-10-30T13:44:07.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb281931640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c15a3b0 0x7fb27c14eab0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb27400c760 tx=0x7fb27400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:07.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb271ffb640 1 -- 172.21.15.31:0/3699179961 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274003b80 con 0x7fb27c15a3b0 2023-10-30T13:44:07.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb271ffb640 1 -- 172.21.15.31:0/3699179961 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb274003ce0 con 0x7fb27c15a3b0 2023-10-30T13:44:07.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb271ffb640 1 -- 172.21.15.31:0/3699179961 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27400aa30 con 0x7fb27c15a3b0 2023-10-30T13:44:07.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c0b1ae0 con 0x7fb27c15a3b0 2023-10-30T13:44:07.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.831+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c15c000 con 0x7fb27c15a3b0 2023-10-30T13:44:07.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.835+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c04e170 con 0x7fb27c15a3b0 2023-10-30T13:44:07.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.835+0000 7fb271ffb640 1 -- 172.21.15.31:0/3699179961 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb27400ab90 con 0x7fb27c15a3b0 2023-10-30T13:44:07.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.835+0000 7fb271ffb640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb2600664d0 0x7fb260068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:07.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.835+0000 7fb271ffb640 1 -- 172.21.15.31:0/3699179961 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fb274083e30 con 0x7fb27c15a3b0 2023-10-30T13:44:07.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.835+0000 7fb281130640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb2600664d0 0x7fb260068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:07.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.835+0000 7fb281130640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb2600664d0 0x7fb260068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb278009af0 tx=0x7fb278008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:07.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:07.839+0000 7fb271ffb640 1 -- 172.21.15.31:0/3699179961 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb274051020 con 0x7fb27c15a3b0 2023-10-30T13:44:08.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.031+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb27c15ca70 con 0x7fb27c15a3b0 2023-10-30T13:44:08.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.031+0000 7fb271ffb640 1 -- 172.21.15.31:0/3699179961 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7fb274054640 con 0x7fb27c15a3b0 2023-10-30T13:44:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:08.035 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:08.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.031+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb2600664d0 msgr2=0x7fb260068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:08.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb2600664d0 0x7fb260068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb278009af0 tx=0x7fb278008040 comp rx=0 tx=0).stop 2023-10-30T13:44:08.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c15a3b0 msgr2=0x7fb27c14eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:08.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c15a3b0 0x7fb27c14eab0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb27400c760 tx=0x7fb27400cc20 comp rx=0 tx=0).stop 2023-10-30T13:44:08.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 shutdown_connections 2023-10-30T13:44:08.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb2600664d0 0x7fb260068990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:08.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27c15a3b0 0x7fb27c14eab0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:08.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb27c153c80 0x7fb27c0c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:08.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 --2- 172.21.15.31:0/3699179961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c0c4a10 0x7fb27c1f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:08.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 >> 172.21.15.31:0/3699179961 conn(0x7fb27c0715c0 msgr2=0x7fb27c1586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:08.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 shutdown_connections 2023-10-30T13:44:08.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:08.035+0000 7fb283394640 1 -- 172.21.15.31:0/3699179961 wait complete. 2023-10-30T13:44:08.054 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:10.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.207+0000 7faf3d69a640 1 Processor -- start 2023-10-30T13:44:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.207+0000 7faf3d69a640 1 -- start start 2023-10-30T13:44:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf3d69a640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faf38153c80 0x7faf3815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf3d69a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c42a0 0x7faf380ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf3d69a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf38159f20 con 0x7faf380c4a10 2023-10-30T13:44:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf3d69a640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3815dec0 con 0x7faf38153c80 2023-10-30T13:44:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf3d69a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf381599a0 con 0x7faf380c42a0 2023-10-30T13:44:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faf38153c80 0x7faf3815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faf38153c80 0x7faf3815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46040/0 (socket says 172.21.15.31:46040) 2023-10-30T13:44:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 -- 172.21.15.31:0/3282808008 learned_addr learned my addr 172.21.15.31:0/3282808008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf377fe640 1 --2- 172.21.15.31:0/3282808008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c42a0 0x7faf380ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 -- 172.21.15.31:0/3282808008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c42a0 msgr2=0x7faf380ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 --2- 172.21.15.31:0/3282808008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c42a0 0x7faf380ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 -- 172.21.15.31:0/3282808008 >> v1:172.21.15.33:6789/0 conn(0x7faf380c4a10 legacy=0x7faf3815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 -- 172.21.15.31:0/3282808008 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3815d620 con 0x7faf38153c80 2023-10-30T13:44:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.211+0000 7faf36ffd640 1 --2- 172.21.15.31:0/3282808008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf3815cf90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7faf28009870 tx=0x7faf2801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec5fd2eb5ca9dce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf367fc640 1 -- 172.21.15.31:0/3282808008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf280a8a30 con 0x7faf38153c80 2023-10-30T13:44:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf367fc640 1 -- 172.21.15.31:0/3282808008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf280a8b90 con 0x7faf38153c80 2023-10-30T13:44:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf367fc640 1 -- 172.21.15.31:0/3282808008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf280b0440 con 0x7faf38153c80 2023-10-30T13:44:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- 172.21.15.31:0/3282808008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 msgr2=0x7faf3815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 --2- 172.21.15.31:0/3282808008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf3815cf90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7faf28009870 tx=0x7faf2801ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:10.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- 172.21.15.31:0/3282808008 shutdown_connections 2023-10-30T13:44:10.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 --2- 172.21.15.31:0/3282808008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c42a0 0x7faf380ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 --2- 172.21.15.31:0/3282808008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf3815cf90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- 172.21.15.31:0/3282808008 >> 172.21.15.31:0/3282808008 conn(0x7faf380715c0 msgr2=0x7faf380719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:10.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- 172.21.15.31:0/3282808008 shutdown_connections 2023-10-30T13:44:10.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- 172.21.15.31:0/3282808008 wait complete. 2023-10-30T13:44:10.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 Processor -- start 2023-10-30T13:44:10.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- start start 2023-10-30T13:44:10.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf380c42a0 0x7faf38155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c4a10 0x7faf3814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf381511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf381f3320 con 0x7faf38153c80 2023-10-30T13:44:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf381556d0 con 0x7faf380c42a0 2023-10-30T13:44:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf3d69a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf38157ea0 con 0x7faf380c4a10 2023-10-30T13:44:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf377fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf381511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf377fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf381511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46046/0 (socket says 172.21.15.31:46046) 2023-10-30T13:44:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.215+0000 7faf377fe640 1 -- 172.21.15.31:0/1121010848 learned_addr learned my addr 172.21.15.31:0/1121010848 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf2ffff640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c4a10 0x7faf3814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf36ffd640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf380c42a0 0x7faf38155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf377fe640 1 -- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c4a10 msgr2=0x7faf3814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf377fe640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c4a10 0x7faf3814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf377fe640 1 -- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf380c42a0 msgr2=0x7faf38155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf377fe640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf380c42a0 0x7faf38155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf377fe640 1 -- 172.21.15.31:0/1121010848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280094d0 con 0x7faf38153c80 2023-10-30T13:44:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf377fe640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf381511b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7faf30007e40 tx=0x7faf3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf2f7fe640 1 -- 172.21.15.31:0/1121010848 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30014620 con 0x7faf38153c80 2023-10-30T13:44:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf2f7fe640 1 -- 172.21.15.31:0/1121010848 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3000ee90 con 0x7faf38153c80 2023-10-30T13:44:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf2f7fe640 1 -- 172.21.15.31:0/1121010848 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf300149e0 con 0x7faf38153c80 2023-10-30T13:44:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf381f2430 con 0x7faf38153c80 2023-10-30T13:44:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf381f29f0 con 0x7faf38153c80 2023-10-30T13:44:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3804e170 con 0x7faf38153c80 2023-10-30T13:44:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf2f7fe640 1 -- 172.21.15.31:0/1121010848 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf30014b40 con 0x7faf38153c80 2023-10-30T13:44:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf2f7fe640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf180664a0 0x7faf18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.219+0000 7faf2f7fe640 1 -- 172.21.15.31:0/1121010848 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7faf30084ab0 con 0x7faf38153c80 2023-10-30T13:44:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.223+0000 7faf36ffd640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf180664a0 0x7faf18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.223+0000 7faf36ffd640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf180664a0 0x7faf18068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf280b2000 tx=0x7faf28008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.223+0000 7faf2f7fe640 1 -- 172.21.15.31:0/1121010848 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf30051ca0 con 0x7faf38153c80 2023-10-30T13:44:10.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.415+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf38150f10 con 0x7faf38153c80 2023-10-30T13:44:10.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.415+0000 7faf2f7fe640 1 -- 172.21.15.31:0/1121010848 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7faf30084020 con 0x7faf38153c80 2023-10-30T13:44:10.419 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:10.419 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:10.419 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:10.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf180664a0 msgr2=0x7faf18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:10.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf180664a0 0x7faf18068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf280b2000 tx=0x7faf28008040 comp rx=0 tx=0).stop 2023-10-30T13:44:10.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 msgr2=0x7faf381511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:10.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf381511b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7faf30007e40 tx=0x7faf3000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 shutdown_connections 2023-10-30T13:44:10.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faf180664a0 0x7faf18068960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf38153c80 0x7faf381511b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faf380c4a10 0x7faf3814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 --2- 172.21.15.31:0/1121010848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf380c42a0 0x7faf38155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 >> 172.21.15.31:0/1121010848 conn(0x7faf380715c0 msgr2=0x7faf38157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 shutdown_connections 2023-10-30T13:44:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:10.419+0000 7faf3d69a640 1 -- 172.21.15.31:0/1121010848 wait complete. 2023-10-30T13:44:10.440 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:12.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc63c609640 1 Processor -- start 2023-10-30T13:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc63c609640 1 -- start start 2023-10-30T13:44:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc63c609640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6340c4a10 0x7fc63415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc63c609640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc634159f20 con 0x7fc6340c42a0 2023-10-30T13:44:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc63c609640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63415d620 con 0x7fc634153c80 2023-10-30T13:44:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc63c609640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63415dec0 con 0x7fc6340c4a10 2023-10-30T13:44:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc639ba4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6340c4a10 0x7fc63415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc639ba4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6340c4a10 0x7fc63415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46050/0 (socket says 172.21.15.31:46050) 2023-10-30T13:44:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc639ba4640 1 -- 172.21.15.31:0/713439207 learned_addr learned my addr 172.21.15.31:0/713439207 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:12.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc639ba4640 1 -- 172.21.15.31:0/713439207 >> v1:172.21.15.33:6789/0 conn(0x7fc634153c80 legacy=0x7fc63415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:12.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc639ba4640 1 -- 172.21.15.31:0/713439207 >> v1:172.21.15.31:6789/0 conn(0x7fc6340c42a0 legacy=0x7fc6340ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:12.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.611+0000 7fc639ba4640 1 -- 172.21.15.31:0/713439207 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63415d900 con 0x7fc6340c4a10 2023-10-30T13:44:12.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc639ba4640 1 --2- 172.21.15.31:0/713439207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c4a10 0x7fc63415cf90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc624009870 tx=0x7fc62401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3ee813faf40fa7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc6393a3640 1 -- 172.21.15.31:0/713439207 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6240a8a30 con 0x7fc6340c4a10 2023-10-30T13:44:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc6393a3640 1 -- 172.21.15.31:0/713439207 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6240a8b90 con 0x7fc6340c4a10 2023-10-30T13:44:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc6393a3640 1 -- 172.21.15.31:0/713439207 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6240b0440 con 0x7fc6340c4a10 2023-10-30T13:44:12.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc63c609640 1 -- 172.21.15.31:0/713439207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c4a10 msgr2=0x7fc63415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:12.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc63c609640 1 --2- 172.21.15.31:0/713439207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c4a10 0x7fc63415cf90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc624009870 tx=0x7fc62401ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:12.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc63c609640 1 -- 172.21.15.31:0/713439207 shutdown_connections 2023-10-30T13:44:12.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc63c609640 1 --2- 172.21.15.31:0/713439207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c4a10 0x7fc63415cf90 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:12.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc63c609640 1 -- 172.21.15.31:0/713439207 >> 172.21.15.31:0/713439207 conn(0x7fc6340715c0 msgr2=0x7fc6340719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:12.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc63c609640 1 -- 172.21.15.31:0/713439207 shutdown_connections 2023-10-30T13:44:12.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.615+0000 7fc63c609640 1 -- 172.21.15.31:0/713439207 wait complete. 2023-10-30T13:44:12.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 Processor -- start 2023-10-30T13:44:12.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 -- start start 2023-10-30T13:44:12.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c42a0 0x7fc63415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:12.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6340c4a10 0x7fc63415b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc634153c80 0x7fc6341f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc634158f60 con 0x7fc6340c42a0 2023-10-30T13:44:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6341e32d0 con 0x7fc634153c80 2023-10-30T13:44:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63414fd80 con 0x7fc6340c4a10 2023-10-30T13:44:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c42a0 0x7fc63415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c42a0 0x7fc63415ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46058/0 (socket says 172.21.15.31:46058) 2023-10-30T13:44:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 -- 172.21.15.31:0/629920307 learned_addr learned my addr 172.21.15.31:0/629920307 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc639ba4640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6340c4a10 0x7fc63415b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63aba6640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc634153c80 0x7fc6341f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:12.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 -- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6340c4a10 msgr2=0x7fc63415b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:12.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6340c4a10 0x7fc63415b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:12.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 -- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc634153c80 msgr2=0x7fc6341f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:12.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc634153c80 0x7fc6341f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:12.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 -- 172.21.15.31:0/629920307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6240094d0 con 0x7fc6340c42a0 2023-10-30T13:44:12.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc639ba4640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6340c4a10 0x7fc63415b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:12.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63aba6640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc634153c80 0x7fc6341f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:12.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63a3a5640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c42a0 0x7fc63415ac60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c3c0 tx=0x7fc63000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:12.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc62affd640 1 -- 172.21.15.31:0/629920307 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6300052d0 con 0x7fc6340c42a0 2023-10-30T13:44:12.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc62affd640 1 -- 172.21.15.31:0/629920307 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630005430 con 0x7fc6340c42a0 2023-10-30T13:44:12.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc62affd640 1 -- 172.21.15.31:0/629920307 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630003840 con 0x7fc6340c42a0 2023-10-30T13:44:12.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc634158350 con 0x7fc6340c42a0 2023-10-30T13:44:12.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.619+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc634158910 con 0x7fc6340c42a0 2023-10-30T13:44:12.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.623+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6340c4710 con 0x7fc6340c42a0 2023-10-30T13:44:12.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.623+0000 7fc62affd640 1 -- 172.21.15.31:0/629920307 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6300039a0 con 0x7fc6340c42a0 2023-10-30T13:44:12.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.623+0000 7fc62affd640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6180664d0 0x7fc618068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:12.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.623+0000 7fc62affd640 1 -- 172.21.15.31:0/629920307 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fc630083600 con 0x7fc6340c42a0 2023-10-30T13:44:12.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.623+0000 7fc639ba4640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6180664d0 0x7fc618068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:12.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.627+0000 7fc639ba4640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6180664d0 0x7fc618068990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc6240b2000 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:12.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.627+0000 7fc62affd640 1 -- 172.21.15.31:0/629920307 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6340c4710 con 0x7fc6340c42a0 2023-10-30T13:44:12.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.819+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6340c4710 con 0x7fc6340c42a0 2023-10-30T13:44:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.819+0000 7fc62affd640 1 -- 172.21.15.31:0/629920307 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7fc6340c4710 con 0x7fc6340c42a0 2023-10-30T13:44:12.822 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:12.823 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:12.824 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:12.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6180664d0 msgr2=0x7fc618068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:12.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6180664d0 0x7fc618068990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc6240b2000 tx=0x7fc624008040 comp rx=0 tx=0).stop 2023-10-30T13:44:12.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c42a0 msgr2=0x7fc63415ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c42a0 0x7fc63415ac60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c3c0 tx=0x7fc63000c880 comp rx=0 tx=0).stop 2023-10-30T13:44:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 shutdown_connections 2023-10-30T13:44:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6180664d0 0x7fc618068990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc634153c80 0x7fc6341f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6340c4a10 0x7fc63415b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 --2- 172.21.15.31:0/629920307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6340c42a0 0x7fc63415ac60 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 >> 172.21.15.31:0/629920307 conn(0x7fc6340715c0 msgr2=0x7fc634151630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 shutdown_connections 2023-10-30T13:44:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:12.823+0000 7fc63c609640 1 -- 172.21.15.31:0/629920307 wait complete. 2023-10-30T13:44:12.843 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:14.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:14.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.995+0000 7fa936291640 1 Processor -- start 2023-10-30T13:44:14.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.995+0000 7fa936291640 1 -- start start 2023-10-30T13:44:15.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa936291640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa930151840 0x7fa9300c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:15.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa936291640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa9300cb8b0 0x7fa93006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:15.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa936291640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9300c5030 con 0x7fa930151c20 2023-10-30T13:44:15.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa936291640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa930157830 con 0x7fa9300cb8b0 2023-10-30T13:44:15.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa936291640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa930157b10 con 0x7fa930151840 2023-10-30T13:44:15.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa93482e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa9300cb8b0 0x7fa93006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:15.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa93482e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa9300cb8b0 0x7fa93006b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58122/0 (socket says 172.21.15.31:58122) 2023-10-30T13:44:15.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa93482e640 1 -- 172.21.15.31:0/380173071 learned_addr learned my addr 172.21.15.31:0/380173071 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:15.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa92ffff640 1 --2- 172.21.15.31:0/380173071 >> v2:172.21.15.33:3300/0 conn(0x7fa930151840 0x7fa9300c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:15.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa93482e640 1 -- 172.21.15.31:0/380173071 >> v2:172.21.15.33:3300/0 conn(0x7fa930151840 msgr2=0x7fa9300c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:15.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa93482e640 1 --2- 172.21.15.31:0/380173071 >> v2:172.21.15.33:3300/0 conn(0x7fa930151840 0x7fa9300c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa93482e640 1 -- 172.21.15.31:0/380173071 >> v1:172.21.15.33:6789/0 conn(0x7fa930151c20 legacy=0x7fa930072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:15.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:14.999+0000 7fa93482e640 1 -- 172.21.15.31:0/380173071 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930157df0 con 0x7fa9300cb8b0 2023-10-30T13:44:15.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa93482e640 1 --2- 172.21.15.31:0/380173071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9300cb8b0 0x7fa93006b600 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa930072b40 tx=0x7fa92801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1d3c58aa576a75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:15.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa92effd640 1 -- 172.21.15.31:0/380173071 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9280af400 con 0x7fa9300cb8b0 2023-10-30T13:44:15.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa92effd640 1 -- 172.21.15.31:0/380173071 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9280ab050 con 0x7fa9300cb8b0 2023-10-30T13:44:15.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa92effd640 1 -- 172.21.15.31:0/380173071 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9280af780 con 0x7fa9300cb8b0 2023-10-30T13:44:15.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 -- 172.21.15.31:0/380173071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9300cb8b0 msgr2=0x7fa93006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:15.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 --2- 172.21.15.31:0/380173071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9300cb8b0 0x7fa93006b600 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa930072b40 tx=0x7fa92801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 -- 172.21.15.31:0/380173071 shutdown_connections 2023-10-30T13:44:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 --2- 172.21.15.31:0/380173071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9300cb8b0 0x7fa93006b600 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 --2- 172.21.15.31:0/380173071 >> v2:172.21.15.33:3300/0 conn(0x7fa930151840 0x7fa9300c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 -- 172.21.15.31:0/380173071 >> 172.21.15.31:0/380173071 conn(0x7fa9300cac70 msgr2=0x7fa9300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 -- 172.21.15.31:0/380173071 shutdown_connections 2023-10-30T13:44:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 -- 172.21.15.31:0/380173071 wait complete. 2023-10-30T13:44:15.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 Processor -- start 2023-10-30T13:44:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.003+0000 7fa936291640 1 -- start start 2023-10-30T13:44:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9300cb8b0 0x7fa930155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa930151840 0x7fa930156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa930151c20 0x7fa930156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9301eed50 con 0x7fa930151840 2023-10-30T13:44:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9301eef40 con 0x7fa9300cb8b0 2023-10-30T13:44:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9301ef220 con 0x7fa930151c20 2023-10-30T13:44:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa930151840 0x7fa930156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:15.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa930151840 0x7fa930156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58138/0 (socket says 172.21.15.31:58138) 2023-10-30T13:44:15.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 -- 172.21.15.31:0/3544182838 learned_addr learned my addr 172.21.15.31:0/3544182838 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:15.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa93482e640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa930151c20 0x7fa930156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:15.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92ffff640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9300cb8b0 0x7fa930155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:15.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 -- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa930151c20 msgr2=0x7fa930156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:15.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa930151c20 0x7fa930156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 -- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9300cb8b0 msgr2=0x7fa930155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:15.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9300cb8b0 0x7fa930155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 -- 172.21.15.31:0/3544182838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928009b30 con 0x7fa930151840 2023-10-30T13:44:15.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92f7fe640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa930151840 0x7fa930156750 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c370 tx=0x7fa92400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:15.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92cff9640 1 -- 172.21.15.31:0/3544182838 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9240052d0 con 0x7fa930151840 2023-10-30T13:44:15.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92cff9640 1 -- 172.21.15.31:0/3544182838 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa924005430 con 0x7fa930151840 2023-10-30T13:44:15.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9301ef500 con 0x7fa930151840 2023-10-30T13:44:15.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92cff9640 1 -- 172.21.15.31:0/3544182838 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924007260 con 0x7fa930151840 2023-10-30T13:44:15.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9301efac0 con 0x7fa930151840 2023-10-30T13:44:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93004e170 con 0x7fa930151840 2023-10-30T13:44:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.007+0000 7fa92cff9640 1 -- 172.21.15.31:0/3544182838 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9240073c0 con 0x7fa930151840 2023-10-30T13:44:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.011+0000 7fa92cff9640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa91c0664d0 0x7fa91c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.011+0000 7fa92cff9640 1 -- 172.21.15.31:0/3544182838 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fa924083480 con 0x7fa930151840 2023-10-30T13:44:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.011+0000 7fa92ffff640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa91c0664d0 0x7fa91c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:15.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.011+0000 7fa92ffff640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa91c0664d0 0x7fa91c068990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa92000b9d0 tx=0x7fa920008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:15.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.015+0000 7fa92cff9640 1 -- 172.21.15.31:0/3544182838 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa924050670 con 0x7fa930151840 2023-10-30T13:44:15.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.203+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9300c66d0 con 0x7fa930151840 2023-10-30T13:44:15.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa92cff9640 1 -- 172.21.15.31:0/3544182838 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7fa924053c90 con 0x7fa930151840 2023-10-30T13:44:15.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:15.209 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:15.209 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:15.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa91c0664d0 msgr2=0x7fa91c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:15.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa91c0664d0 0x7fa91c068990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa92000b9d0 tx=0x7fa920008040 comp rx=0 tx=0).stop 2023-10-30T13:44:15.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa930151840 msgr2=0x7fa930156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:15.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa930151840 0x7fa930156750 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c370 tx=0x7fa92400c830 comp rx=0 tx=0).stop 2023-10-30T13:44:15.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 shutdown_connections 2023-10-30T13:44:15.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa91c0664d0 0x7fa91c068990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa930151c20 0x7fa930156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa930151840 0x7fa930156750 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 --2- 172.21.15.31:0/3544182838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9300cb8b0 0x7fa930155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 >> 172.21.15.31:0/3544182838 conn(0x7fa9300cac70 msgr2=0x7fa9301e26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 shutdown_connections 2023-10-30T13:44:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:15.207+0000 7fa936291640 1 -- 172.21.15.31:0/3544182838 wait complete. 2023-10-30T13:44:15.230 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:17.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:17.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b1c654640 1 Processor -- start 2023-10-30T13:44:17.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b1c654640 1 -- start start 2023-10-30T13:44:17.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b1c654640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3b14153aa0 0x7f3b1415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b1c654640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b140c7b00 con 0x7f3b140701e0 2023-10-30T13:44:17.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b1c654640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b140c2050 con 0x7f3b1406b3b0 2023-10-30T13:44:17.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b1c654640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b14070a50 con 0x7f3b14153aa0 2023-10-30T13:44:17.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b19bef640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f3b140701e0 0x7f3b140ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:33036/0 (socket says 172.21.15.31:33036) 2023-10-30T13:44:17.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b19bef640 1 -- 172.21.15.31:0/797504162 learned_addr learned my addr 172.21.15.31:0/797504162 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:17.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.383+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/797504162 >> v2:172.21.15.33:3300/0 conn(0x7f3b14153aa0 0x7f3b1415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097197948 0 0) 0x7f3b140c7b00 con 0x7f3b140701e0 2023-10-30T13:44:17.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b00003530 con 0x7f3b140701e0 2023-10-30T13:44:17.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1643974892 0 0) 0x7f3b00003530 con 0x7f3b140701e0 2023-10-30T13:44:17.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 >> v2:172.21.15.33:3300/0 conn(0x7f3b14153aa0 msgr2=0x7f3b1415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:17.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 --2- 172.21.15.31:0/797504162 >> v2:172.21.15.33:3300/0 conn(0x7f3b14153aa0 0x7f3b1415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 >> v1:172.21.15.33:6789/0 conn(0x7f3b1406b3b0 legacy=0x7f3b1406b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:17.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b140c2330 con 0x7f3b140701e0 2023-10-30T13:44:17.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f3b04001680 con 0x7f3b140701e0 2023-10-30T13:44:17.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 >> v1:172.21.15.31:6789/0 conn(0x7f3b140701e0 legacy=0x7f3b140ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:17.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b00003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b000047d0 0x7f3b00004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b00005190 0x7f3b0000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14070a50 con 0x7f3b00003980 2023-10-30T13:44:17.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0000c040 con 0x7f3b00005190 2023-10-30T13:44:17.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0000c200 con 0x7f3b000047d0 2023-10-30T13:44:17.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b00003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b19bef640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b000047d0 0x7f3b00004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1abf1640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b00005190 0x7f3b0000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1a3f0640 1 -- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b000047d0 msgr2=0x7f3b00004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:17.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b000047d0 0x7f3b00004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1a3f0640 1 -- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b00005190 msgr2=0x7f3b0000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:17.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b00005190 0x7f3b0000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1a3f0640 1 -- 172.21.15.31:0/797504162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00003530 con 0x7f3b00003980 2023-10-30T13:44:17.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b1abf1640 1 -- 172.21.15.31:0/797504162 reap_dead start 2023-10-30T13:44:17.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.387+0000 7f3b19bef640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b000047d0 0x7f3b00004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:17.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b00003da0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009ae0 tx=0x7f3b1001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f537ca5c6eb5bc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:17.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b100a5640 con 0x7f3b00003980 2023-10-30T13:44:17.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b100a57a0 con 0x7f3b00003980 2023-10-30T13:44:17.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b193ee640 1 -- 172.21.15.31:0/797504162 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b100a5b50 con 0x7f3b00003980 2023-10-30T13:44:17.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- 172.21.15.31:0/797504162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 msgr2=0x7f3b00003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:17.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b00003da0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009ae0 tx=0x7f3b1001ca50 comp rx=0 tx=0).stop 2023-10-30T13:44:17.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- 172.21.15.31:0/797504162 shutdown_connections 2023-10-30T13:44:17.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 --2- 172.21.15.31:0/797504162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b00003da0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- 172.21.15.31:0/797504162 >> 172.21.15.31:0/797504162 conn(0x7f3b14071fb0 msgr2=0x7f3b140723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:17.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- 172.21.15.31:0/797504162 shutdown_connections 2023-10-30T13:44:17.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- 172.21.15.31:0/797504162 wait complete. 2023-10-30T13:44:17.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 Processor -- start 2023-10-30T13:44:17.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- start start 2023-10-30T13:44:17.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b1415b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b14154b10 0x7f3b14158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1415aa00 0x7f3b14151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1415be90 con 0x7f3b00003980 2023-10-30T13:44:17.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14155270 con 0x7f3b1415aa00 2023-10-30T13:44:17.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.391+0000 7f3b1c654640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14158c60 con 0x7f3b14154b10 2023-10-30T13:44:17.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b1415b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b1415b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58152/0 (socket says 172.21.15.31:58152) 2023-10-30T13:44:17.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 -- 172.21.15.31:0/36171440 learned_addr learned my addr 172.21.15.31:0/36171440 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:17.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1abf1640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1415aa00 0x7f3b14151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b19bef640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b14154b10 0x7f3b14158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 -- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b14154b10 msgr2=0x7f3b14158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:17.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b14154b10 0x7f3b14158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 -- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1415aa00 msgr2=0x7f3b14151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:17.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1415aa00 0x7f3b14151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 -- 172.21.15.31:0/36171440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b141554a0 con 0x7f3b00003980 2023-10-30T13:44:17.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1a3f0640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b1415b950 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3b1001c880 tx=0x7f3b100197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:17.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b0affd640 1 -- 172.21.15.31:0/36171440 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b100bd070 con 0x7f3b00003980 2023-10-30T13:44:17.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b0affd640 1 -- 172.21.15.31:0/36171440 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b10004020 con 0x7f3b00003980 2023-10-30T13:44:17.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b0affd640 1 -- 172.21.15.31:0/36171440 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b100043d0 con 0x7f3b00003980 2023-10-30T13:44:17.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b14154330 con 0x7f3b00003980 2023-10-30T13:44:17.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b141545f0 con 0x7f3b00003980 2023-10-30T13:44:17.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1404e170 con 0x7f3b00003980 2023-10-30T13:44:17.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.395+0000 7f3b0affd640 1 -- 172.21.15.31:0/36171440 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b100b4070 con 0x7f3b00003980 2023-10-30T13:44:17.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.399+0000 7f3b0affd640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3af80664d0 0x7f3af8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:17.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.399+0000 7f3b0affd640 1 -- 172.21.15.31:0/36171440 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f3b1011e080 con 0x7f3b00003980 2023-10-30T13:44:17.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.399+0000 7f3b19bef640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3af80664d0 0x7f3af8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:17.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.399+0000 7f3b19bef640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3af80664d0 0x7f3af8068990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b1415dec0 tx=0x7f3b04009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:17.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.403+0000 7f3b0affd640 1 -- 172.21.15.31:0/36171440 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b100eb300 con 0x7f3b00003980 2023-10-30T13:44:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b140c8060 con 0x7f3b00003980 2023-10-30T13:44:17.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b0affd640 1 -- 172.21.15.31:0/36171440 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7f3b100ee920 con 0x7f3b00003980 2023-10-30T13:44:17.593 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:17.593 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:17.594 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:17.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3af80664d0 msgr2=0x7f3af8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:17.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3af80664d0 0x7f3af8068990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b1415dec0 tx=0x7f3b04009040 comp rx=0 tx=0).stop 2023-10-30T13:44:17.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 msgr2=0x7f3b1415b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:17.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b1415b950 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3b1001c880 tx=0x7f3b100197f0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 shutdown_connections 2023-10-30T13:44:17.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3af80664d0 0x7f3af8068990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1415aa00 0x7f3b14151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3b14154b10 0x7f3b14158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 --2- 172.21.15.31:0/36171440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b00003980 0x7f3b1415b950 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:17.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 >> 172.21.15.31:0/36171440 conn(0x7f3b14071fb0 msgr2=0x7f3b1406b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:17.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 shutdown_connections 2023-10-30T13:44:17.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:17.591+0000 7f3b1c654640 1 -- 172.21.15.31:0/36171440 wait complete. 2023-10-30T13:44:17.614 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:19.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:19.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.767+0000 7f759a5aa640 1 Processor -- start 2023-10-30T13:44:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.767+0000 7f759a5aa640 1 -- start start 2023-10-30T13:44:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f759a5aa640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7594153c80 0x7f759415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:19.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f759a5aa640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75940c4a10 0x7f75940c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:19.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f759a5aa640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7594159f20 con 0x7f759415a3b0 2023-10-30T13:44:19.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f759a5aa640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759415dec0 con 0x7f7594153c80 2023-10-30T13:44:19.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f759a5aa640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75941599a0 con 0x7f75940c4a10 2023-10-30T13:44:19.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f75937fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75940c4a10 0x7f75940c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:19.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f7593fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7594153c80 0x7f759415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:19.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f75937fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75940c4a10 0x7f75940c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:57612/0 (socket says 172.21.15.31:57612) 2023-10-30T13:44:19.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f75937fe640 1 -- 172.21.15.31:0/3235238876 learned_addr learned my addr 172.21.15.31:0/3235238876 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:19.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f7592ffd640 1 -- 172.21.15.31:0/3235238876 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900391672 0 0) 0x7f7594159f20 con 0x7f759415a3b0 2023-10-30T13:44:19.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f7592ffd640 1 -- 172.21.15.31:0/3235238876 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c003530 con 0x7f759415a3b0 2023-10-30T13:44:19.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f75937fe640 1 -- 172.21.15.31:0/3235238876 >> v2:172.21.15.33:3300/0 conn(0x7f7594153c80 msgr2=0x7f759415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:19.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f75937fe640 1 --2- 172.21.15.31:0/3235238876 >> v2:172.21.15.33:3300/0 conn(0x7f7594153c80 0x7f759415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f75937fe640 1 -- 172.21.15.31:0/3235238876 >> v1:172.21.15.33:6789/0 conn(0x7f759415a3b0 legacy=0x7f75940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:19.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f75937fe640 1 -- 172.21.15.31:0/3235238876 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940c47e0 con 0x7f75940c4a10 2023-10-30T13:44:19.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.771+0000 7f7593fff640 1 --2- 172.21.15.31:0/3235238876 >> v2:172.21.15.33:3300/0 conn(0x7f7594153c80 0x7f759415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:44:19.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f75937fe640 1 --2- 172.21.15.31:0/3235238876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75940c4a10 0x7f75940c42a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7588009870 tx=0x7f758801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e902d8b84fbc8e55 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:19.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f7592ffd640 1 -- 172.21.15.31:0/3235238876 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880a8a30 con 0x7f75940c4a10 2023-10-30T13:44:19.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f7592ffd640 1 -- 172.21.15.31:0/3235238876 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75880a8b90 con 0x7f75940c4a10 2023-10-30T13:44:19.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f7592ffd640 1 -- 172.21.15.31:0/3235238876 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880b0440 con 0x7f75940c4a10 2023-10-30T13:44:19.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 -- 172.21.15.31:0/3235238876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75940c4a10 msgr2=0x7f75940c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:19.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 --2- 172.21.15.31:0/3235238876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75940c4a10 0x7f75940c42a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7588009870 tx=0x7f758801ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:19.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 -- 172.21.15.31:0/3235238876 shutdown_connections 2023-10-30T13:44:19.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 --2- 172.21.15.31:0/3235238876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75940c4a10 0x7f75940c42a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 --2- 172.21.15.31:0/3235238876 >> v2:172.21.15.33:3300/0 conn(0x7f7594153c80 0x7f759415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 -- 172.21.15.31:0/3235238876 >> 172.21.15.31:0/3235238876 conn(0x7f75940715c0 msgr2=0x7f75940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:19.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 -- 172.21.15.31:0/3235238876 shutdown_connections 2023-10-30T13:44:19.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 -- 172.21.15.31:0/3235238876 wait complete. 2023-10-30T13:44:19.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 Processor -- start 2023-10-30T13:44:19.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.775+0000 7f759a5aa640 1 -- start start 2023-10-30T13:44:19.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75940c4a10 0x7f75940c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:19.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7594153c80 0x7f75940b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:19.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f759415a3b0 0x7f75940b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:19.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759406c8f0 con 0x7f75940c4a10 2023-10-30T13:44:19.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75940cfb80 con 0x7f759415a3b0 2023-10-30T13:44:19.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75940b2020 con 0x7f7594153c80 2023-10-30T13:44:19.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75940c4a10 0x7f75940c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:19.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75940c4a10 0x7f75940c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58154/0 (socket says 172.21.15.31:58154) 2023-10-30T13:44:19.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 -- 172.21.15.31:0/23753721 learned_addr learned my addr 172.21.15.31:0/23753721 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:19.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f75937fe640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7594153c80 0x7f75940b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:19.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7598b47640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f759415a3b0 0x7f75940b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:19.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 -- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7594153c80 msgr2=0x7f75940b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:19.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7594153c80 0x7f75940b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 -- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f759415a3b0 msgr2=0x7f75940b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:19.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f759415a3b0 0x7f75940b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 -- 172.21.15.31:0/23753721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584009420 con 0x7f75940c4a10 2023-10-30T13:44:19.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7593fff640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75940c4a10 0x7f75940c8b00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f758400ebe0 tx=0x7f758400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:19.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7590ff9640 1 -- 172.21.15.31:0/23753721 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758400eee0 con 0x7f75940c4a10 2023-10-30T13:44:19.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7590ff9640 1 -- 172.21.15.31:0/23753721 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758400cc30 con 0x7f75940c4a10 2023-10-30T13:44:19.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75880094d0 con 0x7f75940c4a10 2023-10-30T13:44:19.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f7590ff9640 1 -- 172.21.15.31:0/23753721 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584013470 con 0x7f75940c4a10 2023-10-30T13:44:19.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75940b22c0 con 0x7f75940c4a10 2023-10-30T13:44:19.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.779+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75940b3ab0 con 0x7f75940c4a10 2023-10-30T13:44:19.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.783+0000 7f7590ff9640 1 -- 172.21.15.31:0/23753721 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75840135d0 con 0x7f75940c4a10 2023-10-30T13:44:19.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.783+0000 7f7590ff9640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75800664a0 0x7f7580068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:19.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.783+0000 7f7590ff9640 1 -- 172.21.15.31:0/23753721 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f7584084700 con 0x7f75940c4a10 2023-10-30T13:44:19.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.783+0000 7f75937fe640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75800664a0 0x7f7580068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:19.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.783+0000 7f75937fe640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75800664a0 0x7f7580068960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f75880b2000 tx=0x7f7588009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:19.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.787+0000 7f7590ff9640 1 -- 172.21.15.31:0/23753721 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75840519a0 con 0x7f75940c4a10 2023-10-30T13:44:19.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.979+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75940b40b0 con 0x7f75940c4a10 2023-10-30T13:44:19.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.979+0000 7f7590ff9640 1 -- 172.21.15.31:0/23753721 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7f7584054fc0 con 0x7f75940c4a10 2023-10-30T13:44:19.984 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:19.984 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:19.984 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:19.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75800664a0 msgr2=0x7f7580068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:19.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75800664a0 0x7f7580068960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f75880b2000 tx=0x7f7588009040 comp rx=0 tx=0).stop 2023-10-30T13:44:19.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75940c4a10 msgr2=0x7f75940c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:19.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75940c4a10 0x7f75940c8b00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f758400ebe0 tx=0x7f758400c520 comp rx=0 tx=0).stop 2023-10-30T13:44:19.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 shutdown_connections 2023-10-30T13:44:19.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75800664a0 0x7f7580068960 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f759415a3b0 0x7f75940b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7594153c80 0x7f75940b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 --2- 172.21.15.31:0/23753721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75940c4a10 0x7f75940c8b00 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:19.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 >> 172.21.15.31:0/23753721 conn(0x7f75940715c0 msgr2=0x7f75940719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:19.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 shutdown_connections 2023-10-30T13:44:19.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:19.983+0000 7f759a5aa640 1 -- 172.21.15.31:0/23753721 wait complete. 2023-10-30T13:44:20.005 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:22.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e689a0640 1 Processor -- start 2023-10-30T13:44:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e689a0640 1 -- start start 2023-10-30T13:44:22.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e689a0640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0e640c8eb0 0x7f0e6415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e689a0640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0e6415ac70 0x7f0e6415a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e689a0640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6415dcc0 con 0x7f0e6406aeb0 2023-10-30T13:44:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e689a0640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e64159800 con 0x7f0e6415ac70 2023-10-30T13:44:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e689a0640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e64159d80 con 0x7f0e640c8eb0 2023-10-30T13:44:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e62ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0e6415ac70 0x7f0e6415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e62ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0e6415ac70 0x7f0e6415a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58170/0 (socket says 172.21.15.31:58170) 2023-10-30T13:44:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e62ffd640 1 -- 172.21.15.31:0/450585735 learned_addr learned my addr 172.21.15.31:0/450585735 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e637fe640 1 --2- 172.21.15.31:0/450585735 >> v2:172.21.15.33:3300/0 conn(0x7f0e640c8eb0 0x7f0e6415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e62ffd640 1 -- 172.21.15.31:0/450585735 >> v2:172.21.15.33:3300/0 conn(0x7f0e640c8eb0 msgr2=0x7f0e6415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e62ffd640 1 --2- 172.21.15.31:0/450585735 >> v2:172.21.15.33:3300/0 conn(0x7f0e640c8eb0 0x7f0e6415cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e62ffd640 1 -- 172.21.15.31:0/450585735 >> v1:172.21.15.33:6789/0 conn(0x7f0e6406aeb0 legacy=0x7f0e64069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.155+0000 7f0e62ffd640 1 -- 172.21.15.31:0/450585735 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6415a750 con 0x7f0e6415ac70 2023-10-30T13:44:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e62ffd640 1 --2- 172.21.15.31:0/450585735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6415ac70 0x7f0e6415a210 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0e6406aa40 tx=0x7f0e5801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbea8d15581c83c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e627fc640 1 -- 172.21.15.31:0/450585735 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e580a89b0 con 0x7f0e6415ac70 2023-10-30T13:44:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e627fc640 1 -- 172.21.15.31:0/450585735 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e580a8b10 con 0x7f0e6415ac70 2023-10-30T13:44:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e627fc640 1 -- 172.21.15.31:0/450585735 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e580a8ea0 con 0x7f0e6415ac70 2023-10-30T13:44:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 -- 172.21.15.31:0/450585735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6415ac70 msgr2=0x7f0e6415a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 --2- 172.21.15.31:0/450585735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6415ac70 0x7f0e6415a210 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0e6406aa40 tx=0x7f0e5801ecb0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 -- 172.21.15.31:0/450585735 shutdown_connections 2023-10-30T13:44:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 --2- 172.21.15.31:0/450585735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6415ac70 0x7f0e6415a210 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 --2- 172.21.15.31:0/450585735 >> v2:172.21.15.33:3300/0 conn(0x7f0e640c8eb0 0x7f0e6415cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 -- 172.21.15.31:0/450585735 >> 172.21.15.31:0/450585735 conn(0x7f0e640cef00 msgr2=0x7f0e640cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 -- 172.21.15.31:0/450585735 shutdown_connections 2023-10-30T13:44:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.159+0000 7f0e689a0640 1 -- 172.21.15.31:0/450585735 wait complete. 2023-10-30T13:44:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 Processor -- start 2023-10-30T13:44:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 -- start start 2023-10-30T13:44:22.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6406aeb0 0x7f0e641eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:22.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e640c8eb0 0x7f0e640c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:22.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e6415ac70 0x7f0e640c2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:22.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6414e140 con 0x7f0e6406aeb0 2023-10-30T13:44:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640c4aa0 con 0x7f0e6415ac70 2023-10-30T13:44:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6406b5d0 con 0x7f0e640c8eb0 2023-10-30T13:44:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6406aeb0 0x7f0e641eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6406aeb0 0x7f0e641eeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58180/0 (socket says 172.21.15.31:58180) 2023-10-30T13:44:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 -- 172.21.15.31:0/2061477396 learned_addr learned my addr 172.21.15.31:0/2061477396 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e62ffd640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e640c8eb0 0x7f0e640c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e63fff640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e6415ac70 0x7f0e640c2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 -- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e640c8eb0 msgr2=0x7f0e640c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e640c8eb0 0x7f0e640c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 -- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e6415ac70 msgr2=0x7f0e640c2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e6415ac70 0x7f0e640c2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 -- 172.21.15.31:0/2061477396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e580094d0 con 0x7f0e6406aeb0 2023-10-30T13:44:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e63fff640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e6415ac70 0x7f0e640c2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e637fe640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6406aeb0 0x7f0e641eeb00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400d830 tx=0x7f0e5400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e4bfff640 1 -- 172.21.15.31:0/2061477396 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400b6c0 con 0x7f0e6406aeb0 2023-10-30T13:44:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e4bfff640 1 -- 172.21.15.31:0/2061477396 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5400b820 con 0x7f0e6406aeb0 2023-10-30T13:44:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e4bfff640 1 -- 172.21.15.31:0/2061477396 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400bb90 con 0x7f0e6406aeb0 2023-10-30T13:44:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6414e350 con 0x7f0e6406aeb0 2023-10-30T13:44:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.163+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6414e910 con 0x7f0e6406aeb0 2023-10-30T13:44:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.167+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6404e170 con 0x7f0e6406aeb0 2023-10-30T13:44:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.167+0000 7f0e4bfff640 1 -- 172.21.15.31:0/2061477396 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e5401e460 con 0x7f0e6406aeb0 2023-10-30T13:44:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.167+0000 7f0e4bfff640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e500664d0 0x7f0e50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.167+0000 7f0e4bfff640 1 -- 172.21.15.31:0/2061477396 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f0e54084820 con 0x7f0e6406aeb0 2023-10-30T13:44:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.167+0000 7f0e62ffd640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e500664d0 0x7f0e50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.167+0000 7f0e62ffd640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e500664d0 0x7f0e50068990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0e580b2000 tx=0x7f0e580aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.171+0000 7f0e4bfff640 1 -- 172.21.15.31:0/2061477396 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e54051a20 con 0x7f0e6406aeb0 2023-10-30T13:44:22.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.359+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e640ca960 con 0x7f0e6406aeb0 2023-10-30T13:44:22.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.359+0000 7f0e4bfff640 1 -- 172.21.15.31:0/2061477396 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7f0e54055040 con 0x7f0e6406aeb0 2023-10-30T13:44:22.366 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:22.366 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:22.367 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:22.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e500664d0 msgr2=0x7f0e50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e500664d0 0x7f0e50068990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0e580b2000 tx=0x7f0e580aa070 comp rx=0 tx=0).stop 2023-10-30T13:44:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6406aeb0 msgr2=0x7f0e641eeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6406aeb0 0x7f0e641eeb00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400d830 tx=0x7f0e5400dcf0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 shutdown_connections 2023-10-30T13:44:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0e500664d0 0x7f0e50068990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e6415ac70 0x7f0e640c2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0e640c8eb0 0x7f0e640c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 --2- 172.21.15.31:0/2061477396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e6406aeb0 0x7f0e641eeb00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 >> 172.21.15.31:0/2061477396 conn(0x7f0e640cef00 msgr2=0x7f0e640cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 shutdown_connections 2023-10-30T13:44:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:22.363+0000 7f0e689a0640 1 -- 172.21.15.31:0/2061477396 wait complete. 2023-10-30T13:44:22.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:24.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89ff6ea640 1 Processor -- start 2023-10-30T13:44:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89ff6ea640 1 -- start start 2023-10-30T13:44:24.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89ff6ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f89f80c4a10 0x7f89f815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89ff6ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c42a0 0x7f89f80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89ff6ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f8159f20 con 0x7f89f8153c80 2023-10-30T13:44:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89ff6ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f815dec0 con 0x7f89f80c4a10 2023-10-30T13:44:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89ff6ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f81599a0 con 0x7f89f80c42a0 2023-10-30T13:44:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fcc85640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f89f80c4a10 0x7f89f815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fdc87640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c42a0 0x7f89f80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fcc85640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f89f80c4a10 0x7f89f815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:49794/0 (socket says 172.21.15.31:49794) 2023-10-30T13:44:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fcc85640 1 -- 172.21.15.31:0/2151343945 learned_addr learned my addr 172.21.15.31:0/2151343945 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fd486640 1 --1- 172.21.15.31:0/2151343945 >> v1:172.21.15.33:6789/0 conn(0x7f89f8153c80 0x7f89f815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:44070/0 (socket says 172.21.15.31:44070) 2023-10-30T13:44:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fdc87640 1 -- 172.21.15.31:0/2151343945 >> v2:172.21.15.33:3300/0 conn(0x7f89f80c4a10 msgr2=0x7f89f815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fdc87640 1 --2- 172.21.15.31:0/2151343945 >> v2:172.21.15.33:3300/0 conn(0x7f89f80c4a10 0x7f89f815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fdc87640 1 -- 172.21.15.31:0/2151343945 >> v1:172.21.15.33:6789/0 conn(0x7f89f8153c80 legacy=0x7f89f815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fdc87640 1 -- 172.21.15.31:0/2151343945 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f815d620 con 0x7f89f80c42a0 2023-10-30T13:44:24.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.539+0000 7f89fcc85640 1 --2- 172.21.15.31:0/2151343945 >> v2:172.21.15.33:3300/0 conn(0x7f89f80c4a10 0x7f89f815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:44:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89fdc87640 1 --2- 172.21.15.31:0/2151343945 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c42a0 0x7f89f80ca1e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009d90 tx=0x7f89f001f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e38522d6e87deb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89effff640 1 -- 172.21.15.31:0/2151343945 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00af600 con 0x7f89f80c42a0 2023-10-30T13:44:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89effff640 1 -- 172.21.15.31:0/2151343945 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f00ab050 con 0x7f89f80c42a0 2023-10-30T13:44:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89effff640 1 -- 172.21.15.31:0/2151343945 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00af980 con 0x7f89f80c42a0 2023-10-30T13:44:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89ff6ea640 1 -- 172.21.15.31:0/2151343945 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c42a0 msgr2=0x7f89f80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/2151343945 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c42a0 0x7f89f80ca1e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009d90 tx=0x7f89f001f130 comp rx=0 tx=0).stop 2023-10-30T13:44:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89ff6ea640 1 -- 172.21.15.31:0/2151343945 shutdown_connections 2023-10-30T13:44:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/2151343945 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c42a0 0x7f89f80ca1e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/2151343945 >> v2:172.21.15.33:3300/0 conn(0x7f89f80c4a10 0x7f89f815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.543+0000 7f89ff6ea640 1 -- 172.21.15.31:0/2151343945 >> 172.21.15.31:0/2151343945 conn(0x7f89f80715c0 msgr2=0x7f89f80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 -- 172.21.15.31:0/2151343945 shutdown_connections 2023-10-30T13:44:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 -- 172.21.15.31:0/2151343945 wait complete. 2023-10-30T13:44:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 Processor -- start 2023-10-30T13:44:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 -- start start 2023-10-30T13:44:24.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f80c42a0 0x7f89f81f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:24.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c4a10 0x7f89f806c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89f8153c80 0x7f89f8149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f81f23a0 con 0x7f89f8153c80 2023-10-30T13:44:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f814f1e0 con 0x7f89f80c42a0 2023-10-30T13:44:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89ff6ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80b28e0 con 0x7f89f80c4a10 2023-10-30T13:44:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89f8153c80 0x7f89f8149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89f8153c80 0x7f89f8149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58188/0 (socket says 172.21.15.31:58188) 2023-10-30T13:44:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 -- 172.21.15.31:0/291165149 learned_addr learned my addr 172.21.15.31:0/291165149 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fcc85640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c4a10 0x7f89f806c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fd486640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f80c42a0 0x7f89f81f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 -- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c4a10 msgr2=0x7f89f806c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c4a10 0x7f89f806c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 -- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f80c42a0 msgr2=0x7f89f81f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f80c42a0 0x7f89f81f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 -- 172.21.15.31:0/291165149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e80094d0 con 0x7f89f8153c80 2023-10-30T13:44:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.547+0000 7f89fdc87640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89f8153c80 0x7f89f8149c40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f89f001cd60 tx=0x7f89f001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89edffb640 1 -- 172.21.15.31:0/291165149 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0007000 con 0x7f89f8153c80 2023-10-30T13:44:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89edffb640 1 -- 172.21.15.31:0/291165149 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0009450 con 0x7f89f8153c80 2023-10-30T13:44:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89edffb640 1 -- 172.21.15.31:0/291165149 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00ba420 con 0x7f89f8153c80 2023-10-30T13:44:24.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f0009a70 con 0x7f89f8153c80 2023-10-30T13:44:24.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f80b2ea0 con 0x7f89f8153c80 2023-10-30T13:44:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f804e170 con 0x7f89f8153c80 2023-10-30T13:44:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89edffb640 1 -- 172.21.15.31:0/291165149 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89f00ba5a0 con 0x7f89f8153c80 2023-10-30T13:44:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89edffb640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89dc0664d0 0x7f89dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89edffb640 1 -- 172.21.15.31:0/291165149 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f89f0122ad0 con 0x7f89f8153c80 2023-10-30T13:44:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.551+0000 7f89fd486640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89dc0664d0 0x7f89dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.555+0000 7f89fd486640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89dc0664d0 0x7f89dc068990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f89f8157e60 tx=0x7f89f4007e90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.555+0000 7f89edffb640 1 -- 172.21.15.31:0/291165149 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f89f00efe00 con 0x7f89f8153c80 2023-10-30T13:44:24.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.743+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f814dd10 con 0x7f89f8153c80 2023-10-30T13:44:24.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89edffb640 1 -- 172.21.15.31:0/291165149 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6487 (secure 0 0 0) 0x7f89f00c8090 con 0x7f89f8153c80 2023-10-30T13:44:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:24.749 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:43:55.188918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5470},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5470":{"gid":5470,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":23,"addr":"172.21.15.33:6836/210349556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":210349556},{"type":"v1","addr":"172.21.15.33:6836","nonce":210349556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:24.749 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2023-10-30T13:44:24.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89dc0664d0 msgr2=0x7f89dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89dc0664d0 0x7f89dc068990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f89f8157e60 tx=0x7f89f4007e90 comp rx=0 tx=0).stop 2023-10-30T13:44:24.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89f8153c80 msgr2=0x7f89f8149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89f8153c80 0x7f89f8149c40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f89f001cd60 tx=0x7f89f001bed0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 shutdown_connections 2023-10-30T13:44:24.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89dc0664d0 0x7f89dc068990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89f8153c80 0x7f89f8149c40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89f80c4a10 0x7f89f806c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 --2- 172.21.15.31:0/291165149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f80c42a0 0x7f89f81f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 >> 172.21.15.31:0/291165149 conn(0x7f89f80715c0 msgr2=0x7f89f8158620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:24.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 shutdown_connections 2023-10-30T13:44:24.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:24.747+0000 7f89ff6ea640 1 -- 172.21.15.31:0/291165149 wait complete. 2023-10-30T13:44:24.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:44:26.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:26.415 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:26.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6e1491640 1 Processor -- start 2023-10-30T13:44:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6e1491640 1 -- start start 2023-10-30T13:44:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6e1491640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6dc0718d0 0x7fc6dc0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6e1491640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc153300 0x7fc6dc153750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6e1491640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc153c90 con 0x7fc6dc071cb0 2023-10-30T13:44:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6e1491640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc0cca10 con 0x7fc6dc0718d0 2023-10-30T13:44:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6e1491640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc15d0a0 con 0x7fc6dc153300 2023-10-30T13:44:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6daffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc6dc0718d0 0x7fc6dc0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6da7fc640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc6dc071cb0 0x7fc6dc15a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56264/0 (socket says 172.21.15.31:56264) 2023-10-30T13:44:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6da7fc640 1 -- 172.21.15.31:0/104151482 learned_addr learned my addr 172.21.15.31:0/104151482 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6db7fe640 1 --2- 172.21.15.31:0/104151482 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc153300 0x7fc6dc153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6daffd640 1 -- 172.21.15.31:0/104151482 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc153300 msgr2=0x7fc6dc153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6daffd640 1 --2- 172.21.15.31:0/104151482 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc153300 0x7fc6dc153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6daffd640 1 -- 172.21.15.31:0/104151482 >> v1:172.21.15.31:6789/0 conn(0x7fc6dc071cb0 legacy=0x7fc6dc15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6daffd640 1 -- 172.21.15.31:0/104151482 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc15d380 con 0x7fc6dc0718d0 2023-10-30T13:44:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.923+0000 7fc6d9ffb640 1 -- 172.21.15.31:0/104151482 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948936125 0 0) 0x7fc6dc153c90 con 0x7fc6dc071cb0 2023-10-30T13:44:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6daffd640 1 --2- 172.21.15.31:0/104151482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc0718d0 0x7fc6dc0c9a10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0097c0 tx=0x7fc6cc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3dc9ceb821537f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6d9ffb640 1 -- 172.21.15.31:0/104151482 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0a8970 con 0x7fc6dc0718d0 2023-10-30T13:44:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6d9ffb640 1 -- 172.21.15.31:0/104151482 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc0a8ad0 con 0x7fc6dc0718d0 2023-10-30T13:44:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6d9ffb640 1 -- 172.21.15.31:0/104151482 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0a8e60 con 0x7fc6dc0718d0 2023-10-30T13:44:26.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 -- 172.21.15.31:0/104151482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc0718d0 msgr2=0x7fc6dc0c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 --2- 172.21.15.31:0/104151482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc0718d0 0x7fc6dc0c9a10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0097c0 tx=0x7fc6cc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:44:26.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 -- 172.21.15.31:0/104151482 shutdown_connections 2023-10-30T13:44:26.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 --2- 172.21.15.31:0/104151482 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc153300 0x7fc6dc153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 --2- 172.21.15.31:0/104151482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc0718d0 0x7fc6dc0c9a10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 -- 172.21.15.31:0/104151482 >> 172.21.15.31:0/104151482 conn(0x7fc6dc06ab00 msgr2=0x7fc6dc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 -- 172.21.15.31:0/104151482 shutdown_connections 2023-10-30T13:44:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.927+0000 7fc6e1491640 1 -- 172.21.15.31:0/104151482 wait complete. 2023-10-30T13:44:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 Processor -- start 2023-10-30T13:44:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 -- start start 2023-10-30T13:44:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6dc0718d0 0x7fc6dc0caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc071cb0 0x7fc6dc15c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc153300 0x7fc6dc1ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1f2b00 con 0x7fc6dc153300 2023-10-30T13:44:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1f1ca0 con 0x7fc6dc0718d0 2023-10-30T13:44:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1f1ef0 con 0x7fc6dc071cb0 2023-10-30T13:44:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc153300 0x7fc6dc1ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc153300 0x7fc6dc1ee9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40056/0 (socket says 172.21.15.31:40056) 2023-10-30T13:44:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 -- 172.21.15.31:0/769002907 learned_addr learned my addr 172.21.15.31:0/769002907 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6daffd640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6dc0718d0 0x7fc6dc0caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6da7fc640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc071cb0 0x7fc6dc15c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 -- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc071cb0 msgr2=0x7fc6dc15c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc071cb0 0x7fc6dc15c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 -- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6dc0718d0 msgr2=0x7fc6dc0caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6dc0718d0 0x7fc6dc0caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 -- 172.21.15.31:0/769002907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc009420 con 0x7fc6dc153300 2023-10-30T13:44:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6db7fe640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc153300 0x7fc6dc1ee9a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4007e60 tx=0x7fc6d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6c37fe640 1 -- 172.21.15.31:0/769002907 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4014420 con 0x7fc6dc153300 2023-10-30T13:44:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6c37fe640 1 -- 172.21.15.31:0/769002907 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d40145a0 con 0x7fc6dc153300 2023-10-30T13:44:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6c37fe640 1 -- 172.21.15.31:0/769002907 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4014950 con 0x7fc6dc153300 2023-10-30T13:44:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.931+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc1f21d0 con 0x7fc6dc153300 2023-10-30T13:44:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.935+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc1f2790 con 0x7fc6dc153300 2023-10-30T13:44:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.935+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc1f2920 con 0x7fc6dc153300 2023-10-30T13:44:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.935+0000 7fc6c37fe640 1 -- 172.21.15.31:0/769002907 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6d4014ab0 con 0x7fc6dc153300 2023-10-30T13:44:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.935+0000 7fc6c37fe640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6c80664a0 0x7fc6c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.935+0000 7fc6c37fe640 1 -- 172.21.15.31:0/769002907 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fc6d40849a0 con 0x7fc6dc153300 2023-10-30T13:44:26.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.935+0000 7fc6daffd640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6c80664a0 0x7fc6c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.939+0000 7fc6daffd640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6c80664a0 0x7fc6c8068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0b2000 tx=0x7fc6cc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:26.939+0000 7fc6c37fe640 1 -- 172.21.15.31:0/769002907 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6d4051b60 con 0x7fc6dc153300 2023-10-30T13:44:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.131+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6dc1ee730 con 0x7fc6dc153300 2023-10-30T13:44:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.131+0000 7fc6c37fe640 1 -- 172.21.15.31:0/769002907 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5698 (secure 0 0 0) 0x7fc6d4055180 con 0x7fc6dc153300 2023-10-30T13:44:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:27.135 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:26.910304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:27.136 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 54 2023-10-30T13:44:27.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6c80664a0 msgr2=0x7fc6c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6c80664a0 0x7fc6c8068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0b2000 tx=0x7fc6cc008040 comp rx=0 tx=0).stop 2023-10-30T13:44:27.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc153300 msgr2=0x7fc6dc1ee9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc153300 0x7fc6dc1ee9a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4007e60 tx=0x7fc6d400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 shutdown_connections 2023-10-30T13:44:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc6c80664a0 0x7fc6c8068960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6dc153300 0x7fc6dc1ee9a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc6dc071cb0 0x7fc6dc15c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 --2- 172.21.15.31:0/769002907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6dc0718d0 0x7fc6dc0caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 >> 172.21.15.31:0/769002907 conn(0x7fc6dc06ab00 msgr2=0x7fc6dc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 shutdown_connections 2023-10-30T13:44:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.135+0000 7fc6e1491640 1 -- 172.21.15.31:0/769002907 wait complete. 2023-10-30T13:44:27.156 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:44:27.156 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:44:27.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:27.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.299+0000 7f7c51f63640 1 Processor -- start 2023-10-30T13:44:27.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.299+0000 7f7c51f63640 1 -- start start 2023-10-30T13:44:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.299+0000 7f7c51f63640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7c4c153b50 0x7f7c4c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.299+0000 7f7c51f63640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 0x7f7c4c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.299+0000 7f7c51f63640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4c15a770 con 0x7f7c4c0cb180 2023-10-30T13:44:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.299+0000 7f7c51f63640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4c159cf0 con 0x7f7c4c153b50 2023-10-30T13:44:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.299+0000 7f7c51f63640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c06aa30 con 0x7f7c4c0c62b0 2023-10-30T13:44:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7c4c0cb180 0x7f7c4c06ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56276/0 (socket says 172.21.15.31:56276) 2023-10-30T13:44:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 -- 172.21.15.31:0/766330950 learned_addr learned my addr 172.21.15.31:0/766330950 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c50f61640 1 --2- 172.21.15.31:0/766330950 >> v2:172.21.15.33:3300/0 conn(0x7f7c4c153b50 0x7f7c4c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4bfff640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 0x7f7c4c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909013234 0 0) 0x7f7c440014a0 con 0x7f7c4c0cb180 2023-10-30T13:44:27.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c3c003530 con 0x7f7c4c0cb180 2023-10-30T13:44:27.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1091762771 0 0) 0x7f7c3c003530 con 0x7f7c4c0cb180 2023-10-30T13:44:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 msgr2=0x7f7c4c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 0x7f7c4c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 >> v2:172.21.15.33:3300/0 conn(0x7f7c4c153b50 msgr2=0x7f7c4c1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 --2- 172.21.15.31:0/766330950 >> v2:172.21.15.33:3300/0 conn(0x7f7c4c153b50 0x7f7c4c1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0c9d40 con 0x7f7c4c0cb180 2023-10-30T13:44:27.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f7c44001b90 con 0x7f7c4c0cb180 2023-10-30T13:44:27.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c50f61640 1 --2- 172.21.15.31:0/766330950 >> v2:172.21.15.33:3300/0 conn(0x7f7c4c153b50 0x7f7c4c1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:44:27.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 >> v1:172.21.15.31:6789/0 conn(0x7f7c4c0cb180 legacy=0x7f7c4c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c003980 0x7f7c3c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 0x7f7c3c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c3c004d10 0x7f7c3c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c159cf0 con 0x7f7c3c004d10 2023-10-30T13:44:27.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c06aa30 con 0x7f7c3c003980 2023-10-30T13:44:27.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c00b990 con 0x7f7c4c0c62b0 2023-10-30T13:44:27.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c3c004d10 0x7f7c3c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c4bfff640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 0x7f7c3c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c50f61640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c003980 0x7f7c3c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 -- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 msgr2=0x7f7c3c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:27.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c0c62b0 0x7f7c3c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 -- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c003980 msgr2=0x7f7c3c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:27.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c003980 0x7f7c3c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.303+0000 7f7c51762640 1 -- 172.21.15.31:0/766330950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c003530 con 0x7f7c3c004d10 2023-10-30T13:44:27.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51762640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c3c004d10 0x7f7c3c005190 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400a840 tx=0x7f7c4401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e0cb52450666e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:27.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c440ad630 con 0x7f7c3c004d10 2023-10-30T13:44:27.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c440a8070 con 0x7f7c3c004d10 2023-10-30T13:44:27.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c4b7fe640 1 -- 172.21.15.31:0/766330950 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c440ad9f0 con 0x7f7c3c004d10 2023-10-30T13:44:27.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 -- 172.21.15.31:0/766330950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c3c004d10 msgr2=0x7f7c3c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 --2- 172.21.15.31:0/766330950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c3c004d10 0x7f7c3c005190 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400a840 tx=0x7f7c4401d340 comp rx=0 tx=0).stop 2023-10-30T13:44:27.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51762640 1 -- 172.21.15.31:0/766330950 reap_dead start 2023-10-30T13:44:27.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 -- 172.21.15.31:0/766330950 shutdown_connections 2023-10-30T13:44:27.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 -- 172.21.15.31:0/766330950 >> 172.21.15.31:0/766330950 conn(0x7f7c4c065a80 msgr2=0x7f7c4c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:27.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 -- 172.21.15.31:0/766330950 shutdown_connections 2023-10-30T13:44:27.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 -- 172.21.15.31:0/766330950 wait complete. 2023-10-30T13:44:27.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 Processor -- start 2023-10-30T13:44:27.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.307+0000 7f7c51f63640 1 -- start start 2023-10-30T13:44:27.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c004d10 0x7f7c4c154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c156760 0x7f7c4c0c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c156030 0x7f7c4c0ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0ce3d0 con 0x7f7c4c156760 2023-10-30T13:44:27.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0cdd80 con 0x7f7c3c004d10 2023-10-30T13:44:27.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c156ca0 con 0x7f7c4c156030 2023-10-30T13:44:27.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c156760 0x7f7c4c0c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c156760 0x7f7c4c0c40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40088/0 (socket says 172.21.15.31:40088) 2023-10-30T13:44:27.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 -- 172.21.15.31:0/2300207005 learned_addr learned my addr 172.21.15.31:0/2300207005 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c50f61640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c004d10 0x7f7c4c154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51762640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c156030 0x7f7c4c0ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 -- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c156030 msgr2=0x7f7c4c0ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c156030 0x7f7c4c0ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 -- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c004d10 msgr2=0x7f7c4c154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c004d10 0x7f7c4c154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 -- 172.21.15.31:0/2300207005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c156f80 con 0x7f7c4c156760 2023-10-30T13:44:27.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c50f61640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c004d10 0x7f7c4c154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:27.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c4bfff640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c156760 0x7f7c4c0c40e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7c38011960 tx=0x7f7c38011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:27.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c497fa640 1 -- 172.21.15.31:0/2300207005 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3800faf0 con 0x7f7c4c156760 2023-10-30T13:44:27.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c497fa640 1 -- 172.21.15.31:0/2300207005 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3800fc50 con 0x7f7c4c156760 2023-10-30T13:44:27.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c497fa640 1 -- 172.21.15.31:0/2300207005 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38014440 con 0x7f7c4c156760 2023-10-30T13:44:27.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4c0cdff0 con 0x7f7c4c156760 2023-10-30T13:44:27.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4c157260 con 0x7f7c4c156760 2023-10-30T13:44:27.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.311+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4c0ce510 con 0x7f7c4c156760 2023-10-30T13:44:27.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.315+0000 7f7c497fa640 1 -- 172.21.15.31:0/2300207005 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c380145a0 con 0x7f7c4c156760 2023-10-30T13:44:27.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.315+0000 7f7c497fa640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7c300664d0 0x7f7c30068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:27.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.315+0000 7f7c497fa640 1 -- 172.21.15.31:0/2300207005 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f7c38084aa0 con 0x7f7c4c156760 2023-10-30T13:44:27.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.315+0000 7f7c50f61640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7c300664d0 0x7f7c30068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:27.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.315+0000 7f7c50f61640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7c300664d0 0x7f7c30068990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009d70 tx=0x7f7c40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:27.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.319+0000 7f7c497fa640 1 -- 172.21.15.31:0/2300207005 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c4c0ce510 con 0x7f7c4c156760 2023-10-30T13:44:27.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.507+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c4c0ce510 con 0x7f7c4c156760 2023-10-30T13:44:27.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.507+0000 7f7c497fa640 1 -- 172.21.15.31:0/2300207005 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5698 (secure 0 0 0) 0x7f7c4c0ce510 con 0x7f7c4c156760 2023-10-30T13:44:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:27.513 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:26.910304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:27.514 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 54 2023-10-30T13:44:27.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7c300664d0 msgr2=0x7f7c30068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7c300664d0 0x7f7c30068990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009d70 tx=0x7f7c40008040 comp rx=0 tx=0).stop 2023-10-30T13:44:27.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c156760 msgr2=0x7f7c4c0c40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:27.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c156760 0x7f7c4c0c40e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7c38011960 tx=0x7f7c38011e20 comp rx=0 tx=0).stop 2023-10-30T13:44:27.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 shutdown_connections 2023-10-30T13:44:27.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7c300664d0 0x7f7c30068990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7c4c156030 0x7f7c4c0ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c156760 0x7f7c4c0c40e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 --2- 172.21.15.31:0/2300207005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c3c004d10 0x7f7c4c154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:27.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 >> 172.21.15.31:0/2300207005 conn(0x7f7c4c065a80 msgr2=0x7f7c4c0c7360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:27.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 shutdown_connections 2023-10-30T13:44:27.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:27.511+0000 7f7c51f63640 1 -- 172.21.15.31:0/2300207005 wait complete. 2023-10-30T13:44:27.533 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5448": { "addr": "172.21.15.31:6837/349618961", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 349618961, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 349618961, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5448, "incarnation": 54, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:44:26.910304+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5448 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:44:29.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:29.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f27678c9640 1 Processor -- start 2023-10-30T13:44:29.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f27678c9640 1 -- start start 2023-10-30T13:44:29.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f27678c9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f27600c4a10 0x7f276015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:29.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f27678c9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f27600c42a0 0x7f27600ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f27678c9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2760159f20 con 0x7f2760153c80 2023-10-30T13:44:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f27678c9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f276015dec0 con 0x7f27600c4a10 2023-10-30T13:44:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f27678c9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27601599a0 con 0x7f27600c42a0 2023-10-30T13:44:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2764e64640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f27600c4a10 0x7f276015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2764e64640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f27600c4a10 0x7f276015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40100/0 (socket says 172.21.15.31:40100) 2023-10-30T13:44:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2765665640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2760153c80 0x7f276015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56278/0 (socket says 172.21.15.31:56278) 2023-10-30T13:44:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2764e64640 1 -- 172.21.15.31:0/1321557101 learned_addr learned my addr 172.21.15.31:0/1321557101 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2765e66640 1 --2- 172.21.15.31:0/1321557101 >> v2:172.21.15.33:3300/0 conn(0x7f27600c42a0 0x7f27600ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2764e64640 1 -- 172.21.15.31:0/1321557101 >> v2:172.21.15.33:3300/0 conn(0x7f27600c42a0 msgr2=0x7f27600ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2764e64640 1 --2- 172.21.15.31:0/1321557101 >> v2:172.21.15.33:3300/0 conn(0x7f27600c42a0 0x7f27600ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2764e64640 1 -- 172.21.15.31:0/1321557101 >> v1:172.21.15.31:6789/0 conn(0x7f2760153c80 legacy=0x7f276015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2764e64640 1 -- 172.21.15.31:0/1321557101 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276015d620 con 0x7f27600c4a10 2023-10-30T13:44:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.687+0000 7f2757fff640 1 -- 172.21.15.31:0/1321557101 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081216478 0 0) 0x7f2760159f20 con 0x7f2760153c80 2023-10-30T13:44:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f2764e64640 1 --2- 172.21.15.31:0/1321557101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f276015cf90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f27600b1d40 tx=0x7f275001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5905a3f61e89eaab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f2757fff640 1 -- 172.21.15.31:0/1321557101 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27500a89f0 con 0x7f27600c4a10 2023-10-30T13:44:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f2757fff640 1 -- 172.21.15.31:0/1321557101 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27500a8b50 con 0x7f27600c4a10 2023-10-30T13:44:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f2757fff640 1 -- 172.21.15.31:0/1321557101 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27500b0440 con 0x7f27600c4a10 2023-10-30T13:44:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 -- 172.21.15.31:0/1321557101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 msgr2=0x7f276015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 --2- 172.21.15.31:0/1321557101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f276015cf90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f27600b1d40 tx=0x7f275001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 -- 172.21.15.31:0/1321557101 shutdown_connections 2023-10-30T13:44:29.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 --2- 172.21.15.31:0/1321557101 >> v2:172.21.15.33:3300/0 conn(0x7f27600c42a0 0x7f27600ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 --2- 172.21.15.31:0/1321557101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f276015cf90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 -- 172.21.15.31:0/1321557101 >> 172.21.15.31:0/1321557101 conn(0x7f27600715c0 msgr2=0x7f27600719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:29.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 -- 172.21.15.31:0/1321557101 shutdown_connections 2023-10-30T13:44:29.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.691+0000 7f27678c9640 1 -- 172.21.15.31:0/1321557101 wait complete. 2023-10-30T13:44:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 Processor -- start 2023-10-30T13:44:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 -- start start 2023-10-30T13:44:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27600c42a0 0x7f2760155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f27601f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2760153c80 0x7f27601f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276014ea10 con 0x7f27600c4a10 2023-10-30T13:44:29.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276014ded0 con 0x7f2760153c80 2023-10-30T13:44:29.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27601541e0 con 0x7f27600c42a0 2023-10-30T13:44:29.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f27601f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:29.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f27601f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40102/0 (socket says 172.21.15.31:40102) 2023-10-30T13:44:29.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 -- 172.21.15.31:0/1693463795 learned_addr learned my addr 172.21.15.31:0/1693463795 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2765665640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27600c42a0 0x7f2760155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2765e66640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2760153c80 0x7f27601f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 -- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27600c42a0 msgr2=0x7f2760155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27600c42a0 0x7f2760155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 -- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2760153c80 msgr2=0x7f27601f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2760153c80 0x7f27601f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 -- 172.21.15.31:0/1693463795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500094d0 con 0x7f27600c4a10 2023-10-30T13:44:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2764e64640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f27601f2420 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f27500b3040 tx=0x7f275001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2755ffb640 1 -- 172.21.15.31:0/1693463795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27500b0e00 con 0x7f27600c4a10 2023-10-30T13:44:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2755ffb640 1 -- 172.21.15.31:0/1693463795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27500bb470 con 0x7f27600c4a10 2023-10-30T13:44:29.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f2755ffb640 1 -- 172.21.15.31:0/1693463795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27500bb820 con 0x7f27600c4a10 2023-10-30T13:44:29.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276014e0e0 con 0x7f27600c4a10 2023-10-30T13:44:29.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.695+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276014e6a0 con 0x7f27600c4a10 2023-10-30T13:44:29.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.699+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276004e170 con 0x7f27600c4a10 2023-10-30T13:44:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.699+0000 7f2755ffb640 1 -- 172.21.15.31:0/1693463795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27500b7070 con 0x7f27600c4a10 2023-10-30T13:44:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.699+0000 7f2755ffb640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27440664d0 0x7f2744068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.699+0000 7f2755ffb640 1 -- 172.21.15.31:0/1693463795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f2750122340 con 0x7f27600c4a10 2023-10-30T13:44:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.699+0000 7f2765665640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27440664d0 0x7f2744068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.699+0000 7f2765665640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27440664d0 0x7f2744068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f275c000f70 tx=0x7f275c007df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.703+0000 7f2755ffb640 1 -- 172.21.15.31:0/1693463795 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27500ef570 con 0x7f27600c4a10 2023-10-30T13:44:29.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.895+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27601f2180 con 0x7f27600c4a10 2023-10-30T13:44:29.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.895+0000 7f2755ffb640 1 -- 172.21.15.31:0/1693463795 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5698 (secure 0 0 0) 0x7f27500f2b90 con 0x7f27600c4a10 2023-10-30T13:44:29.899 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:29.899 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:26.910304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:29.899 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 54 2023-10-30T13:44:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27440664d0 msgr2=0x7f2744068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27440664d0 0x7f2744068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f275c000f70 tx=0x7f275c007df0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 msgr2=0x7f27601f2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f27601f2420 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f27500b3040 tx=0x7f275001ba90 comp rx=0 tx=0).stop 2023-10-30T13:44:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 shutdown_connections 2023-10-30T13:44:29.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f27440664d0 0x7f2744068990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2760153c80 0x7f27601f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27600c4a10 0x7f27601f2420 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 --2- 172.21.15.31:0/1693463795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f27600c42a0 0x7f2760155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 >> 172.21.15.31:0/1693463795 conn(0x7f27600715c0 msgr2=0x7f27601587b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:29.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 shutdown_connections 2023-10-30T13:44:29.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:29.899+0000 7f27678c9640 1 -- 172.21.15.31:0/1693463795 wait complete. 2023-10-30T13:44:29.919 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:31.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:32.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.083+0000 7f5208491640 1 Processor -- start 2023-10-30T13:44:32.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.083+0000 7f5208491640 1 -- start start 2023-10-30T13:44:32.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5208491640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f52000c4a10 0x7f520015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:32.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5208491640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5200159f20 con 0x7f52000c42a0 2023-10-30T13:44:32.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5208491640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520015d620 con 0x7f5200153c80 2023-10-30T13:44:32.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5208491640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520015dec0 con 0x7f52000c4a10 2023-10-30T13:44:32.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5206a2e640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f52000c42a0 0x7f52000ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56292/0 (socket says 172.21.15.31:56292) 2023-10-30T13:44:32.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5206a2e640 1 -- 172.21.15.31:0/1384154112 learned_addr learned my addr 172.21.15.31:0/1384154112 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:32.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5205a2c640 1 --2- 172.21.15.31:0/1384154112 >> v2:172.21.15.33:3300/0 conn(0x7f52000c4a10 0x7f520015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:32.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f520522b640 1 -- 172.21.15.31:0/1384154112 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024254936 0 0) 0x7f5200159f20 con 0x7f52000c42a0 2023-10-30T13:44:32.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f520522b640 1 -- 172.21.15.31:0/1384154112 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51ec003530 con 0x7f52000c42a0 2023-10-30T13:44:32.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5205a2c640 1 -- 172.21.15.31:0/1384154112 >> v1:172.21.15.33:6789/0 conn(0x7f5200153c80 legacy=0x7f520015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:32.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5205a2c640 1 -- 172.21.15.31:0/1384154112 >> v1:172.21.15.31:6789/0 conn(0x7f52000c42a0 legacy=0x7f52000ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:32.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.087+0000 7f5205a2c640 1 -- 172.21.15.31:0/1384154112 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520015d900 con 0x7f52000c4a10 2023-10-30T13:44:32.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5205a2c640 1 --2- 172.21.15.31:0/1384154112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c4a10 0x7f520015cf90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f51f00097c0 tx=0x7f51f001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd34a568a1e06137 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:32.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f520522b640 1 -- 172.21.15.31:0/1384154112 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f00a8970 con 0x7f52000c4a10 2023-10-30T13:44:32.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f520522b640 1 -- 172.21.15.31:0/1384154112 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f00a8ad0 con 0x7f52000c4a10 2023-10-30T13:44:32.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f520522b640 1 -- 172.21.15.31:0/1384154112 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f00a8e60 con 0x7f52000c4a10 2023-10-30T13:44:32.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 -- 172.21.15.31:0/1384154112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c4a10 msgr2=0x7f520015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:32.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 --2- 172.21.15.31:0/1384154112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c4a10 0x7f520015cf90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f51f00097c0 tx=0x7f51f001ec70 comp rx=0 tx=0).stop 2023-10-30T13:44:32.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 -- 172.21.15.31:0/1384154112 shutdown_connections 2023-10-30T13:44:32.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 --2- 172.21.15.31:0/1384154112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c4a10 0x7f520015cf90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:32.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 -- 172.21.15.31:0/1384154112 >> 172.21.15.31:0/1384154112 conn(0x7f52000715c0 msgr2=0x7f52000719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:32.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 -- 172.21.15.31:0/1384154112 shutdown_connections 2023-10-30T13:44:32.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 -- 172.21.15.31:0/1384154112 wait complete. 2023-10-30T13:44:32.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 Processor -- start 2023-10-30T13:44:32.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.091+0000 7f5208491640 1 -- start start 2023-10-30T13:44:32.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c42a0 0x7f5200152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:32.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f52000c4a10 0x7f5200146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:32.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5200153c80 0x7f52001471e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:32.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200157e10 con 0x7f5200153c80 2023-10-30T13:44:32.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52001f2810 con 0x7f52000c42a0 2023-10-30T13:44:32.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52001f2af0 con 0x7f52000c4a10 2023-10-30T13:44:32.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5200153c80 0x7f52001471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:32.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5200153c80 0x7f52001471e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40106/0 (socket says 172.21.15.31:40106) 2023-10-30T13:44:32.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 -- 172.21.15.31:0/545738872 learned_addr learned my addr 172.21.15.31:0/545738872 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:32.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f520622d640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c42a0 0x7f5200152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:32.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5205a2c640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f52000c4a10 0x7f5200146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:32.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 -- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f52000c4a10 msgr2=0x7f5200146ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:32.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f52000c4a10 0x7f5200146ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:32.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 -- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c42a0 msgr2=0x7f5200152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:32.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c42a0 0x7f5200152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:32.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 -- 172.21.15.31:0/545738872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0009420 con 0x7f5200153c80 2023-10-30T13:44:32.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5206a2e640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5200153c80 0x7f52001471e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f51f800cb40 tx=0x7f51f800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:32.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f51f6ffd640 1 -- 172.21.15.31:0/545738872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800aca0 con 0x7f5200153c80 2023-10-30T13:44:32.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f51f6ffd640 1 -- 172.21.15.31:0/545738872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f800ae00 con 0x7f5200153c80 2023-10-30T13:44:32.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f51f6ffd640 1 -- 172.21.15.31:0/545738872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8012620 con 0x7f5200153c80 2023-10-30T13:44:32.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52001f2dd0 con 0x7f5200153c80 2023-10-30T13:44:32.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52001f3390 con 0x7f5200153c80 2023-10-30T13:44:32.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.095+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52001eefe0 con 0x7f5200153c80 2023-10-30T13:44:32.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.099+0000 7f51f6ffd640 1 -- 172.21.15.31:0/545738872 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51f8007070 con 0x7f5200153c80 2023-10-30T13:44:32.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.099+0000 7f51f6ffd640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51e40664d0 0x7f51e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:32.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.099+0000 7f51f6ffd640 1 -- 172.21.15.31:0/545738872 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f51f8084150 con 0x7f5200153c80 2023-10-30T13:44:32.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.099+0000 7f520622d640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51e40664d0 0x7f51e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:32.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.103+0000 7f520622d640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51e40664d0 0x7f51e4068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f51fc007fd0 tx=0x7f51fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:32.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.103+0000 7f51f6ffd640 1 -- 172.21.15.31:0/545738872 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f52001eefe0 con 0x7f5200153c80 2023-10-30T13:44:32.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52001eefe0 con 0x7f5200153c80 2023-10-30T13:44:32.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f51f6ffd640 1 -- 172.21.15.31:0/545738872 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5698 (secure 0 0 0) 0x7f52001eefe0 con 0x7f5200153c80 2023-10-30T13:44:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:32.306 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:26.910304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:32.306 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 54 2023-10-30T13:44:32.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51e40664d0 msgr2=0x7f51e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:32.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51e40664d0 0x7f51e4068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f51fc007fd0 tx=0x7f51fc009040 comp rx=0 tx=0).stop 2023-10-30T13:44:32.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5200153c80 msgr2=0x7f52001471e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:32.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5200153c80 0x7f52001471e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f51f800cb40 tx=0x7f51f800a060 comp rx=0 tx=0).stop 2023-10-30T13:44:32.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 shutdown_connections 2023-10-30T13:44:32.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51e40664d0 0x7f51e4068990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:32.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5200153c80 0x7f52001471e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:32.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f52000c4a10 0x7f5200146ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:32.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 --2- 172.21.15.31:0/545738872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52000c42a0 0x7f5200152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:32.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 >> 172.21.15.31:0/545738872 conn(0x7f52000715c0 msgr2=0x7f5200150c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:32.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 shutdown_connections 2023-10-30T13:44:32.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:32.303+0000 7f5208491640 1 -- 172.21.15.31:0/545738872 wait complete. 2023-10-30T13:44:32.327 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:34.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60d5bd3640 1 Processor -- start 2023-10-30T13:44:34.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60d5bd3640 1 -- start start 2023-10-30T13:44:34.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60d5bd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:34.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60d5bd3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f60d00c42a0 0x7f60d00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:34.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60d5bd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d0159f20 con 0x7f60d00c4a10 2023-10-30T13:44:34.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60d5bd3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d015dec0 con 0x7f60d00c42a0 2023-10-30T13:44:34.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60d5bd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d01599a0 con 0x7f60d0153c80 2023-10-30T13:44:34.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f60d00c42a0 0x7f60d00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:34.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f60d00c42a0 0x7f60d00ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40112/0 (socket says 172.21.15.31:40112) 2023-10-30T13:44:34.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cffff640 1 -- 172.21.15.31:0/3471155293 learned_addr learned my addr 172.21.15.31:0/3471155293 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:34.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cf7fe640 1 --2- 172.21.15.31:0/3471155293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:34.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cffff640 1 -- 172.21.15.31:0/3471155293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 msgr2=0x7f60d015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:34.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cffff640 1 --2- 172.21.15.31:0/3471155293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cffff640 1 -- 172.21.15.31:0/3471155293 >> v1:172.21.15.33:6789/0 conn(0x7f60d00c4a10 legacy=0x7f60d015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:34.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.474+0000 7f60cffff640 1 -- 172.21.15.31:0/3471155293 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d015d620 con 0x7f60d00c42a0 2023-10-30T13:44:34.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60cffff640 1 --2- 172.21.15.31:0/3471155293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d00ca1e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009e50 tx=0x7f60c801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8a112cd2b35ae2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:34.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60ce7fc640 1 -- 172.21.15.31:0/3471155293 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80af400 con 0x7f60d00c42a0 2023-10-30T13:44:34.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60ce7fc640 1 -- 172.21.15.31:0/3471155293 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c80ab050 con 0x7f60d00c42a0 2023-10-30T13:44:34.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60ce7fc640 1 -- 172.21.15.31:0/3471155293 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80af780 con 0x7f60d00c42a0 2023-10-30T13:44:34.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 -- 172.21.15.31:0/3471155293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 msgr2=0x7f60d00ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:34.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/3471155293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d00ca1e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009e50 tx=0x7f60c801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 -- 172.21.15.31:0/3471155293 shutdown_connections 2023-10-30T13:44:34.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/3471155293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d00ca1e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/3471155293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 -- 172.21.15.31:0/3471155293 >> 172.21.15.31:0/3471155293 conn(0x7f60d00715c0 msgr2=0x7f60d00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:34.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 -- 172.21.15.31:0/3471155293 shutdown_connections 2023-10-30T13:44:34.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.478+0000 7f60d5bd3640 1 -- 172.21.15.31:0/3471155293 wait complete. 2023-10-30T13:44:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 Processor -- start 2023-10-30T13:44:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 -- start start 2023-10-30T13:44:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d015cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d00c4a10 0x7f60d00ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d014ca50 con 0x7f60d00c42a0 2023-10-30T13:44:34.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d01f3260 con 0x7f60d00c4a10 2023-10-30T13:44:34.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d0148730 con 0x7f60d0153c80 2023-10-30T13:44:34.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d015cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:34.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d015cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40128/0 (socket says 172.21.15.31:40128) 2023-10-30T13:44:34.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 -- 172.21.15.31:0/477128213 learned_addr learned my addr 172.21.15.31:0/477128213 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cffff640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60ceffd640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d00c4a10 0x7f60d00ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 -- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 msgr2=0x7f60d014a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d014a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 -- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d00c4a10 msgr2=0x7f60d00ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:34.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d00c4a10 0x7f60d00ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 -- 172.21.15.31:0/477128213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8009b30 con 0x7f60d00c42a0 2023-10-30T13:44:34.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60cf7fe640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d015cb00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f60c000d650 tx=0x7f60c000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:34.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60b7fff640 1 -- 172.21.15.31:0/477128213 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c000b6c0 con 0x7f60d00c42a0 2023-10-30T13:44:34.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60b7fff640 1 -- 172.21.15.31:0/477128213 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c000b840 con 0x7f60d00c42a0 2023-10-30T13:44:34.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d014d610 con 0x7f60d00c42a0 2023-10-30T13:44:34.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60b7fff640 1 -- 172.21.15.31:0/477128213 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c000bbf0 con 0x7f60d00c42a0 2023-10-30T13:44:34.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d0148450 con 0x7f60d00c42a0 2023-10-30T13:44:34.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.482+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d0159450 con 0x7f60d00c42a0 2023-10-30T13:44:34.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.486+0000 7f60b7fff640 1 -- 172.21.15.31:0/477128213 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60c000bd50 con 0x7f60d00c42a0 2023-10-30T13:44:34.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.486+0000 7f60b7fff640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:34.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.486+0000 7f60b7fff640 1 -- 172.21.15.31:0/477128213 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f60c00844c0 con 0x7f60d00c42a0 2023-10-30T13:44:34.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.486+0000 7f60ceffd640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:34.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.486+0000 7f60ceffd640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f60d015d040 tx=0x7f60c4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:34.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.490+0000 7f60b7fff640 1 -- 172.21.15.31:0/477128213 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60c0051660 con 0x7f60d00c42a0 2023-10-30T13:44:34.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.678+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d014d610 con 0x7f60d00c42a0 2023-10-30T13:44:34.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.678+0000 7f60b7fff640 1 -- 172.21.15.31:0/477128213 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f60d014d610 con 0x7f60d00c42a0 2023-10-30T13:44:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:34.683 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:33.060124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:34.683 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 55 2023-10-30T13:44:34.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 msgr2=0x7f60bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:34.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f60d015d040 tx=0x7f60c4009040 comp rx=0 tx=0).stop 2023-10-30T13:44:34.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 msgr2=0x7f60d015cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:34.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d015cb00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f60c000d650 tx=0x7f60c000db10 comp rx=0 tx=0).stop 2023-10-30T13:44:34.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 shutdown_connections 2023-10-30T13:44:34.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d0153c80 0x7f60d014a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d00c4a10 0x7f60d00ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 --2- 172.21.15.31:0/477128213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d00c42a0 0x7f60d015cb00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:34.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 >> 172.21.15.31:0/477128213 conn(0x7f60d00715c0 msgr2=0x7f60d0157c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:34.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 shutdown_connections 2023-10-30T13:44:34.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:34.682+0000 7f60d5bd3640 1 -- 172.21.15.31:0/477128213 wait complete. 2023-10-30T13:44:34.703 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:36.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ec3c4640 1 Processor -- start 2023-10-30T13:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ec3c4640 1 -- start start 2023-10-30T13:44:36.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ec3c4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f61e4151840 0x7f61e40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ec3c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e40cb8b0 0x7f61e406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ec3c4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40c5030 con 0x7f61e4151c20 2023-10-30T13:44:36.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ec3c4640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e4157830 con 0x7f61e4151840 2023-10-30T13:44:36.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ec3c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4157b10 con 0x7f61e40cb8b0 2023-10-30T13:44:36.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea961640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e40cb8b0 0x7f61e406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea160640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f61e4151840 0x7f61e40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea961640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e40cb8b0 0x7f61e406b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:50198/0 (socket says 172.21.15.31:50198) 2023-10-30T13:44:36.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea961640 1 -- 172.21.15.31:0/3357574753 learned_addr learned my addr 172.21.15.31:0/3357574753 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:36.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea961640 1 -- 172.21.15.31:0/3357574753 >> v2:172.21.15.33:3300/0 conn(0x7f61e4151840 msgr2=0x7f61e40c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:36.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea961640 1 --2- 172.21.15.31:0/3357574753 >> v2:172.21.15.33:3300/0 conn(0x7f61e4151840 0x7f61e40c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea961640 1 -- 172.21.15.31:0/3357574753 >> v1:172.21.15.33:6789/0 conn(0x7f61e4151c20 legacy=0x7f61e4072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.858+0000 7f61ea961640 1 -- 172.21.15.31:0/3357574753 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4157df0 con 0x7f61e40cb8b0 2023-10-30T13:44:36.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ea961640 1 --2- 172.21.15.31:0/3357574753 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e40cb8b0 0x7f61e406b600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009e50 tx=0x7f61dc01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37807a01c99306cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61e915e640 1 -- 172.21.15.31:0/3357574753 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc0af400 con 0x7f61e40cb8b0 2023-10-30T13:44:36.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61e915e640 1 -- 172.21.15.31:0/3357574753 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc0ab050 con 0x7f61e40cb8b0 2023-10-30T13:44:36.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61e915e640 1 -- 172.21.15.31:0/3357574753 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc0af780 con 0x7f61e40cb8b0 2023-10-30T13:44:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3357574753 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e40cb8b0 msgr2=0x7f61e406b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3357574753 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e40cb8b0 0x7f61e406b600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009e50 tx=0x7f61dc01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3357574753 shutdown_connections 2023-10-30T13:44:36.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3357574753 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e40cb8b0 0x7f61e406b600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3357574753 >> v2:172.21.15.33:3300/0 conn(0x7f61e4151840 0x7f61e40c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3357574753 >> 172.21.15.31:0/3357574753 conn(0x7f61e40cac70 msgr2=0x7f61e40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:36.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3357574753 shutdown_connections 2023-10-30T13:44:36.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.862+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3357574753 wait complete. 2023-10-30T13:44:36.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 Processor -- start 2023-10-30T13:44:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 -- start start 2023-10-30T13:44:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61e40cb8b0 0x7f61e4157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e4151840 0x7f61e4145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e4151c20 0x7f61e4143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e414fe60 con 0x7f61e4151c20 2023-10-30T13:44:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4154770 con 0x7f61e40cb8b0 2023-10-30T13:44:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e414a9c0 con 0x7f61e4151840 2023-10-30T13:44:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e4151c20 0x7f61e4143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e4151c20 0x7f61e4143500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51860/0 (socket says 172.21.15.31:51860) 2023-10-30T13:44:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 -- 172.21.15.31:0/3699718248 learned_addr learned my addr 172.21.15.31:0/3699718248 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea160640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61e40cb8b0 0x7f61e4157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61e995f640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e4151840 0x7f61e4145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 -- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e4151840 msgr2=0x7f61e4145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e4151840 0x7f61e4145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 -- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61e40cb8b0 msgr2=0x7f61e4157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61e40cb8b0 0x7f61e4157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 -- 172.21.15.31:0/3699718248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc009b30 con 0x7f61e4151c20 2023-10-30T13:44:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ea961640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e4151c20 0x7f61e4143500 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f61dc01ce20 tx=0x7f61dc01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61daffd640 1 -- 172.21.15.31:0/3699718248 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc007000 con 0x7f61e4151c20 2023-10-30T13:44:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61daffd640 1 -- 172.21.15.31:0/3699718248 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc0afdb0 con 0x7f61e4151c20 2023-10-30T13:44:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e4157590 con 0x7f61e4151c20 2023-10-30T13:44:36.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61daffd640 1 -- 172.21.15.31:0/3699718248 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc0b9630 con 0x7f61e4151c20 2023-10-30T13:44:36.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.866+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e4150f10 con 0x7f61e4151c20 2023-10-30T13:44:36.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.870+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e404e170 con 0x7f61e4151c20 2023-10-30T13:44:36.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.870+0000 7f61daffd640 1 -- 172.21.15.31:0/3699718248 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61dc0b6070 con 0x7f61e4151c20 2023-10-30T13:44:36.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.870+0000 7f61daffd640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f61c80664d0 0x7f61c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.870+0000 7f61daffd640 1 -- 172.21.15.31:0/3699718248 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f61dc122250 con 0x7f61e4151c20 2023-10-30T13:44:36.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.870+0000 7f61ea160640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f61c80664d0 0x7f61c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.874+0000 7f61ea160640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f61c80664d0 0x7f61c8068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009b60 tx=0x7f61e0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:36.874+0000 7f61daffd640 1 -- 172.21.15.31:0/3699718248 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61dc0ef530 con 0x7f61e4151c20 2023-10-30T13:44:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.066+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e4066510 con 0x7f61e4151c20 2023-10-30T13:44:37.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.066+0000 7f61daffd640 1 -- 172.21.15.31:0/3699718248 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f61dc009510 con 0x7f61e4151c20 2023-10-30T13:44:37.072 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:37.072 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:33.060124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:37.072 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 55 2023-10-30T13:44:37.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f61c80664d0 msgr2=0x7f61c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:37.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f61c80664d0 0x7f61c8068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009b60 tx=0x7f61e0009190 comp rx=0 tx=0).stop 2023-10-30T13:44:37.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e4151c20 msgr2=0x7f61e4143500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:37.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e4151c20 0x7f61e4143500 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f61dc01ce20 tx=0x7f61dc01bf90 comp rx=0 tx=0).stop 2023-10-30T13:44:37.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 shutdown_connections 2023-10-30T13:44:37.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f61c80664d0 0x7f61c8068990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:37.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e4151c20 0x7f61e4143500 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:37.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f61e4151840 0x7f61e4145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:37.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 --2- 172.21.15.31:0/3699718248 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61e40cb8b0 0x7f61e4157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:37.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 >> 172.21.15.31:0/3699718248 conn(0x7f61e40cac70 msgr2=0x7f61e4072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:37.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 shutdown_connections 2023-10-30T13:44:37.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:37.070+0000 7f61ec3c4640 1 -- 172.21.15.31:0/3699718248 wait complete. 2023-10-30T13:44:37.092 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:39.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f48104d6640 1 Processor -- start 2023-10-30T13:44:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f48104d6640 1 -- start start 2023-10-30T13:44:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f48104d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4808153c80 0x7f480815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f48104d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c42a0 0x7f48080ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f48104d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4808159f20 con 0x7f48080c4a10 2023-10-30T13:44:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f48104d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480815dec0 con 0x7f4808153c80 2023-10-30T13:44:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f48104d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48081599a0 con 0x7f48080c42a0 2023-10-30T13:44:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480e272640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4808153c80 0x7f480815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480e272640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4808153c80 0x7f480815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51868/0 (socket says 172.21.15.31:51868) 2023-10-30T13:44:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480e272640 1 -- 172.21.15.31:0/2984389467 learned_addr learned my addr 172.21.15.31:0/2984389467 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480ea73640 1 --2- 172.21.15.31:0/2984389467 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c42a0 0x7f48080ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480e272640 1 -- 172.21.15.31:0/2984389467 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c42a0 msgr2=0x7f48080ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480e272640 1 --2- 172.21.15.31:0/2984389467 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c42a0 0x7f48080ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480e272640 1 -- 172.21.15.31:0/2984389467 >> v1:172.21.15.31:6789/0 conn(0x7f48080c4a10 legacy=0x7f480815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.246+0000 7f480e272640 1 -- 172.21.15.31:0/2984389467 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480815d620 con 0x7f4808153c80 2023-10-30T13:44:39.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f480e272640 1 --2- 172.21.15.31:0/2984389467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f480815cf90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4804009870 tx=0x7f480401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a740a3d46c92d89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:39.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f480d270640 1 -- 172.21.15.31:0/2984389467 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48040a8a30 con 0x7f4808153c80 2023-10-30T13:44:39.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f480d270640 1 -- 172.21.15.31:0/2984389467 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48040a8b90 con 0x7f4808153c80 2023-10-30T13:44:39.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f480d270640 1 -- 172.21.15.31:0/2984389467 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48040b0440 con 0x7f4808153c80 2023-10-30T13:44:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 -- 172.21.15.31:0/2984389467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 msgr2=0x7f480815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 --2- 172.21.15.31:0/2984389467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f480815cf90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4804009870 tx=0x7f480401ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 -- 172.21.15.31:0/2984389467 shutdown_connections 2023-10-30T13:44:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 --2- 172.21.15.31:0/2984389467 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c42a0 0x7f48080ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 --2- 172.21.15.31:0/2984389467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f480815cf90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 -- 172.21.15.31:0/2984389467 >> 172.21.15.31:0/2984389467 conn(0x7f48080715c0 msgr2=0x7f48080719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 -- 172.21.15.31:0/2984389467 shutdown_connections 2023-10-30T13:44:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.250+0000 7f48104d6640 1 -- 172.21.15.31:0/2984389467 wait complete. 2023-10-30T13:44:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 Processor -- start 2023-10-30T13:44:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 -- start start 2023-10-30T13:44:39.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48080c42a0 0x7f4808155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:39.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c4a10 0x7f480814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:39.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f48081511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:39.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48081f3320 con 0x7f4808153c80 2023-10-30T13:44:39.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48081556d0 con 0x7f48080c42a0 2023-10-30T13:44:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4808157ea0 con 0x7f48080c4a10 2023-10-30T13:44:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f48081511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480e272640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48080c42a0 0x7f4808155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480e272640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48080c42a0 0x7f4808155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:33326/0 (socket says 172.21.15.31:33326) 2023-10-30T13:44:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480da71640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c4a10 0x7f480814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480e272640 1 -- 172.21.15.31:0/1552965491 learned_addr learned my addr 172.21.15.31:0/1552965491 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f48081511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51874/0 (socket says 172.21.15.31:51874) 2023-10-30T13:44:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 -- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c4a10 msgr2=0x7f480814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c4a10 0x7f480814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 -- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48080c42a0 msgr2=0x7f4808155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:39.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48080c42a0 0x7f4808155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 -- 172.21.15.31:0/1552965491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48040094d0 con 0x7f4808153c80 2023-10-30T13:44:39.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f480ea73640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f48081511b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4800007d90 tx=0x7f480000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:39.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f47feffd640 1 -- 172.21.15.31:0/1552965491 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800018070 con 0x7f4808153c80 2023-10-30T13:44:39.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f47feffd640 1 -- 172.21.15.31:0/1552965491 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480000edd0 con 0x7f4808153c80 2023-10-30T13:44:39.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f47feffd640 1 -- 172.21.15.31:0/1552965491 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800003bc0 con 0x7f4808153c80 2023-10-30T13:44:39.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48081f2430 con 0x7f4808153c80 2023-10-30T13:44:39.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48081f29f0 con 0x7f4808153c80 2023-10-30T13:44:39.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.254+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480804e170 con 0x7f4808153c80 2023-10-30T13:44:39.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.258+0000 7f47feffd640 1 -- 172.21.15.31:0/1552965491 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4800014420 con 0x7f4808153c80 2023-10-30T13:44:39.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.258+0000 7f47feffd640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47ec0664a0 0x7f47ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:39.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.258+0000 7f47feffd640 1 -- 172.21.15.31:0/1552965491 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f48000841c0 con 0x7f4808153c80 2023-10-30T13:44:39.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.258+0000 7f480e272640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47ec0664a0 0x7f47ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:39.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.258+0000 7f480e272640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47ec0664a0 0x7f47ec068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f480401ed70 tx=0x7f4804009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:39.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.262+0000 7f47feffd640 1 -- 172.21.15.31:0/1552965491 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48000514b0 con 0x7f4808153c80 2023-10-30T13:44:39.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4808150f10 con 0x7f4808153c80 2023-10-30T13:44:39.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f47feffd640 1 -- 172.21.15.31:0/1552965491 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f4800054ad0 con 0x7f4808153c80 2023-10-30T13:44:39.461 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:39.461 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:38.265973+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 56 2023-10-30T13:44:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47ec0664a0 msgr2=0x7f47ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47ec0664a0 0x7f47ec068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f480401ed70 tx=0x7f4804009040 comp rx=0 tx=0).stop 2023-10-30T13:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 msgr2=0x7f48081511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f48081511b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4800007d90 tx=0x7f480000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 shutdown_connections 2023-10-30T13:44:39.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f47ec0664a0 0x7f47ec068960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4808153c80 0x7f48081511b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f48080c4a10 0x7f480814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 --2- 172.21.15.31:0/1552965491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48080c42a0 0x7f4808155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:39.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 >> 172.21.15.31:0/1552965491 conn(0x7f48080715c0 msgr2=0x7f4808157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:39.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 shutdown_connections 2023-10-30T13:44:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:39.458+0000 7f48104d6640 1 -- 172.21.15.31:0/1552965491 wait complete. 2023-10-30T13:44:39.482 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5448": { "addr": "172.21.15.31:6837/349618961", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 349618961, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 349618961, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5448, "incarnation": 54, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:44:38.265973+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5448 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:44:41.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:41.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78d1417640 1 Processor -- start 2023-10-30T13:44:41.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78d1417640 1 -- start start 2023-10-30T13:44:41.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78d1417640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f78cc0c4a10 0x7f78cc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78d1417640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f78cc0c42a0 0x7f78cc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78d1417640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78cc159f20 con 0x7f78cc153c80 2023-10-30T13:44:41.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78d1417640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc15dec0 con 0x7f78cc0c4a10 2023-10-30T13:44:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78d1417640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc1599a0 con 0x7f78cc0c42a0 2023-10-30T13:44:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78c3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f78cc0c4a10 0x7f78cc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78c3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f78cc0c4a10 0x7f78cc15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51888/0 (socket says 172.21.15.31:51888) 2023-10-30T13:44:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78c3fff640 1 -- 172.21.15.31:0/381842283 learned_addr learned my addr 172.21.15.31:0/381842283 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78cb7fe640 1 --2- 172.21.15.31:0/381842283 >> v2:172.21.15.33:3300/0 conn(0x7f78cc0c42a0 0x7f78cc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78c3fff640 1 -- 172.21.15.31:0/381842283 >> v2:172.21.15.33:3300/0 conn(0x7f78cc0c42a0 msgr2=0x7f78cc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:41.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78c3fff640 1 --2- 172.21.15.31:0/381842283 >> v2:172.21.15.33:3300/0 conn(0x7f78cc0c42a0 0x7f78cc0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78c3fff640 1 -- 172.21.15.31:0/381842283 >> v1:172.21.15.33:6789/0 conn(0x7f78cc153c80 legacy=0x7f78cc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78c3fff640 1 -- 172.21.15.31:0/381842283 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc15d620 con 0x7f78cc0c4a10 2023-10-30T13:44:41.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.634+0000 7f78cb7fe640 1 --2- 172.21.15.31:0/381842283 >> v2:172.21.15.33:3300/0 conn(0x7f78cc0c42a0 0x7f78cc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:41.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78c3fff640 1 --2- 172.21.15.31:0/381842283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc15cf90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0b1d40 tx=0x7f78b401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8bcc8af4159752 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:41.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78ca7fc640 1 -- 172.21.15.31:0/381842283 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40a89f0 con 0x7f78cc0c4a10 2023-10-30T13:44:41.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78ca7fc640 1 -- 172.21.15.31:0/381842283 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b40a8b50 con 0x7f78cc0c4a10 2023-10-30T13:44:41.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78ca7fc640 1 -- 172.21.15.31:0/381842283 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40b0440 con 0x7f78cc0c4a10 2023-10-30T13:44:41.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 -- 172.21.15.31:0/381842283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 msgr2=0x7f78cc15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 --2- 172.21.15.31:0/381842283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc15cf90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0b1d40 tx=0x7f78b401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 -- 172.21.15.31:0/381842283 shutdown_connections 2023-10-30T13:44:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 --2- 172.21.15.31:0/381842283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc15cf90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 --2- 172.21.15.31:0/381842283 >> v2:172.21.15.33:3300/0 conn(0x7f78cc0c42a0 0x7f78cc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 -- 172.21.15.31:0/381842283 >> 172.21.15.31:0/381842283 conn(0x7f78cc0715c0 msgr2=0x7f78cc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:41.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 -- 172.21.15.31:0/381842283 shutdown_connections 2023-10-30T13:44:41.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.638+0000 7f78d1417640 1 -- 172.21.15.31:0/381842283 wait complete. 2023-10-30T13:44:41.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 Processor -- start 2023-10-30T13:44:41.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 -- start start 2023-10-30T13:44:41.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f78cc0c42a0 0x7f78cc154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc0cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78cc153c80 0x7f78cc0b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc14fd00 con 0x7f78cc0c4a10 2023-10-30T13:44:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc14ffe0 con 0x7f78cc153c80 2023-10-30T13:44:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc1502c0 con 0x7f78cc0c42a0 2023-10-30T13:44:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc0cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc0cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51896/0 (socket says 172.21.15.31:51896) 2023-10-30T13:44:41.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 -- 172.21.15.31:0/3212368073 learned_addr learned my addr 172.21.15.31:0/3212368073 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:41.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78cb7fe640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78cc153c80 0x7f78cc0b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78caffd640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f78cc0c42a0 0x7f78cc154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 -- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f78cc0c42a0 msgr2=0x7f78cc154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f78cc0c42a0 0x7f78cc154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 -- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78cc153c80 msgr2=0x7f78cc0b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78cc153c80 0x7f78cc0b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 -- 172.21.15.31:0/3212368073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b40094d0 con 0x7f78cc0c4a10 2023-10-30T13:44:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c3fff640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc0cf620 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f78b40b3040 tx=0x7f78b401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c37fe640 1 -- 172.21.15.31:0/3212368073 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40b0e00 con 0x7f78cc0c4a10 2023-10-30T13:44:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c37fe640 1 -- 172.21.15.31:0/3212368073 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b40bb470 con 0x7f78cc0c4a10 2023-10-30T13:44:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc0b37e0 con 0x7f78cc0c4a10 2023-10-30T13:44:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78c37fe640 1 -- 172.21.15.31:0/3212368073 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40bb820 con 0x7f78cc0c4a10 2023-10-30T13:44:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.642+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc0b3da0 con 0x7f78cc0c4a10 2023-10-30T13:44:41.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.646+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc04e170 con 0x7f78cc0c4a10 2023-10-30T13:44:41.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.646+0000 7f78c37fe640 1 -- 172.21.15.31:0/3212368073 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78b40b7070 con 0x7f78cc0c4a10 2023-10-30T13:44:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.646+0000 7f78c37fe640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f78ac0664a0 0x7f78ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.646+0000 7f78c37fe640 1 -- 172.21.15.31:0/3212368073 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f78b4122530 con 0x7f78cc0c4a10 2023-10-30T13:44:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.646+0000 7f78caffd640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f78ac0664a0 0x7f78ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.646+0000 7f78caffd640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f78ac0664a0 0x7f78ac068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f78bc000f70 tx=0x7f78bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:41.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.650+0000 7f78c37fe640 1 -- 172.21.15.31:0/3212368073 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78b40ef640 con 0x7f78cc0c4a10 2023-10-30T13:44:41.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.838+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78cc0cad40 con 0x7f78cc0c4a10 2023-10-30T13:44:41.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.838+0000 7f78c37fe640 1 -- 172.21.15.31:0/3212368073 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f78b40b5090 con 0x7f78cc0c4a10 2023-10-30T13:44:41.843 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:41.843 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:38.265973+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 56 2023-10-30T13:44:41.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f78ac0664a0 msgr2=0x7f78ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f78ac0664a0 0x7f78ac068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f78bc000f70 tx=0x7f78bc009040 comp rx=0 tx=0).stop 2023-10-30T13:44:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 msgr2=0x7f78cc0cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc0cf620 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f78b40b3040 tx=0x7f78b401ba90 comp rx=0 tx=0).stop 2023-10-30T13:44:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 shutdown_connections 2023-10-30T13:44:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f78ac0664a0 0x7f78ac068960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78cc153c80 0x7f78cc0b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78cc0c4a10 0x7f78cc0cf620 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 --2- 172.21.15.31:0/3212368073 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f78cc0c42a0 0x7f78cc154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 >> 172.21.15.31:0/3212368073 conn(0x7f78cc0715c0 msgr2=0x7f78cc159140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 shutdown_connections 2023-10-30T13:44:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:41.842+0000 7f78d1417640 1 -- 172.21.15.31:0/3212368073 wait complete. 2023-10-30T13:44:41.863 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:43.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f355973e640 1 Processor -- start 2023-10-30T13:44:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f355973e640 1 -- start start 2023-10-30T13:44:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f355973e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3554071cb0 0x7f35540c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f355973e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 0x7f3554153790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f355973e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3554153cd0 con 0x7f35540718d0 2023-10-30T13:44:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f355973e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35540cca10 con 0x7f3554071cb0 2023-10-30T13:44:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f355973e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355415d0a0 con 0x7f3554153300 2023-10-30T13:44:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3552ffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f35540718d0 0x7f355415a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52314/0 (socket says 172.21.15.31:52314) 2023-10-30T13:44:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3552ffd640 1 -- 172.21.15.31:0/3412858496 learned_addr learned my addr 172.21.15.31:0/3412858496 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35527fc640 1 --2- 172.21.15.31:0/3412858496 >> v2:172.21.15.33:3300/0 conn(0x7f3554071cb0 0x7f35540c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35537fe640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 0x7f3554153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215871414 0 0) 0x7f3544001120 con 0x7f35540718d0 2023-10-30T13:44:44.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c003530 con 0x7f35540718d0 2023-10-30T13:44:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 862862634 0 0) 0x7f353c003530 con 0x7f35540718d0 2023-10-30T13:44:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 msgr2=0x7f3554153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 0x7f3554153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 >> v2:172.21.15.33:3300/0 conn(0x7f3554071cb0 msgr2=0x7f35540c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 --2- 172.21.15.31:0/3412858496 >> v2:172.21.15.33:3300/0 conn(0x7f3554071cb0 0x7f35540c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355415d380 con 0x7f35540718d0 2023-10-30T13:44:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f3544001810 con 0x7f35540718d0 2023-10-30T13:44:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 >> v1:172.21.15.31:6789/0 conn(0x7f35540718d0 legacy=0x7f355415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 0x7f353c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c003f00 0x7f353c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f353c004d10 0x7f353c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35540cca10 con 0x7f353c004d10 2023-10-30T13:44:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355415d0a0 con 0x7f353c003f00 2023-10-30T13:44:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c00b9c0 con 0x7f3554153300 2023-10-30T13:44:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35537fe640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f353c004d10 0x7f353c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f3552ffd640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 0x7f353c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35527fc640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c003f00 0x7f353c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35537fe640 1 -- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 msgr2=0x7f353c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35537fe640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3554153300 0x7f353c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35537fe640 1 -- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c003f00 msgr2=0x7f353c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35537fe640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c003f00 0x7f353c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.018+0000 7f35537fe640 1 -- 172.21.15.31:0/3412858496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c003530 con 0x7f353c004d10 2023-10-30T13:44:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f35537fe640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f353c004d10 0x7f353c005190 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f354c009fc0 tx=0x7f354c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7705d11861afbcea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0a5c70 con 0x7f353c004d10 2023-10-30T13:44:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c00f040 con 0x7f353c004d10 2023-10-30T13:44:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f3551ffb640 1 -- 172.21.15.31:0/3412858496 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0ad440 con 0x7f353c004d10 2023-10-30T13:44:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f355973e640 1 -- 172.21.15.31:0/3412858496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f353c004d10 msgr2=0x7f353c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f355973e640 1 --2- 172.21.15.31:0/3412858496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f353c004d10 0x7f353c005190 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f354c009fc0 tx=0x7f354c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:44:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f35537fe640 1 -- 172.21.15.31:0/3412858496 reap_dead start 2023-10-30T13:44:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f355973e640 1 -- 172.21.15.31:0/3412858496 shutdown_connections 2023-10-30T13:44:44.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f355973e640 1 -- 172.21.15.31:0/3412858496 >> 172.21.15.31:0/3412858496 conn(0x7f355406ab00 msgr2=0x7f3554065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:44.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f355973e640 1 -- 172.21.15.31:0/3412858496 shutdown_connections 2023-10-30T13:44:44.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.022+0000 7f355973e640 1 -- 172.21.15.31:0/3412858496 wait complete. 2023-10-30T13:44:44.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 Processor -- start 2023-10-30T13:44:44.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 -- start start 2023-10-30T13:44:44.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c004d10 0x7f355414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f355415a970 0x7f35540b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35541f1bf0 0x7f35540cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355415c1d0 con 0x7f35541f1bf0 2023-10-30T13:44:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355414edb0 con 0x7f353c004d10 2023-10-30T13:44:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355414f780 con 0x7f355415a970 2023-10-30T13:44:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35541f1bf0 0x7f35540cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35541f1bf0 0x7f35540cf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51918/0 (socket says 172.21.15.31:51918) 2023-10-30T13:44:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 -- 172.21.15.31:0/3073680346 learned_addr learned my addr 172.21.15.31:0/3073680346 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35527fc640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f355415a970 0x7f35540b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f3552ffd640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c004d10 0x7f355414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 -- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f355415a970 msgr2=0x7f35540b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f355415a970 0x7f35540b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 -- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c004d10 msgr2=0x7f355414f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c004d10 0x7f355414f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 -- 172.21.15.31:0/3073680346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35541f24a0 con 0x7f35541f1bf0 2023-10-30T13:44:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f3552ffd640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c004d10 0x7f355414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f35537fe640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35541f1bf0 0x7f35540cf240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b0040 tx=0x7f354c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f353b7fe640 1 -- 172.21.15.31:0/3073680346 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0039f0 con 0x7f35541f1bf0 2023-10-30T13:44:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f353b7fe640 1 -- 172.21.15.31:0/3073680346 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c003b70 con 0x7f35541f1bf0 2023-10-30T13:44:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f353b7fe640 1 -- 172.21.15.31:0/3073680346 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0ae890 con 0x7f35541f1bf0 2023-10-30T13:44:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35541f2780 con 0x7f35541f1bf0 2023-10-30T13:44:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.026+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355414e760 con 0x7f35541f1bf0 2023-10-30T13:44:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.030+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355404e170 con 0x7f35541f1bf0 2023-10-30T13:44:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.030+0000 7f353b7fe640 1 -- 172.21.15.31:0/3073680346 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f354c0b4040 con 0x7f35541f1bf0 2023-10-30T13:44:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.030+0000 7f353b7fe640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35400664a0 0x7f3540068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.030+0000 7f353b7fe640 1 -- 172.21.15.31:0/3073680346 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f354c11dc50 con 0x7f35541f1bf0 2023-10-30T13:44:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.030+0000 7f3552ffd640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35400664a0 0x7f3540068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.030+0000 7f3552ffd640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35400664a0 0x7f3540068960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3544000c10 tx=0x7f3544009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:44.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.034+0000 7f353b7fe640 1 -- 172.21.15.31:0/3073680346 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f354c0eaf30 con 0x7f35541f1bf0 2023-10-30T13:44:44.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.226+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35540c9f80 con 0x7f35541f1bf0 2023-10-30T13:44:44.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.226+0000 7f353b7fe640 1 -- 172.21.15.31:0/3073680346 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f354c0b2020 con 0x7f35541f1bf0 2023-10-30T13:44:44.230 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:44.231 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:43.334469+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:44.231 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 57 2023-10-30T13:44:44.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35400664a0 msgr2=0x7f3540068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:44.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35400664a0 0x7f3540068960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3544000c10 tx=0x7f3544009040 comp rx=0 tx=0).stop 2023-10-30T13:44:44.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35541f1bf0 msgr2=0x7f35540cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:44.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35541f1bf0 0x7f35540cf240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b0040 tx=0x7f354c009750 comp rx=0 tx=0).stop 2023-10-30T13:44:44.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 shutdown_connections 2023-10-30T13:44:44.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35400664a0 0x7f3540068960 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35541f1bf0 0x7f35540cf240 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f355415a970 0x7f35540b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 --2- 172.21.15.31:0/3073680346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c004d10 0x7f355414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:44.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 >> 172.21.15.31:0/3073680346 conn(0x7f355406ab00 msgr2=0x7f35540c47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:44.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 shutdown_connections 2023-10-30T13:44:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:44.230+0000 7f355973e640 1 -- 172.21.15.31:0/3073680346 wait complete. 2023-10-30T13:44:44.252 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:46.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:46.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.402+0000 7f5adb5f8640 1 Processor -- start 2023-10-30T13:44:46.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5adb5f8640 1 -- start start 2023-10-30T13:44:46.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5adb5f8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad41537f0 0x7f5ad40701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:46.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5adb5f8640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5ad415a2e0 0x7f5ad40ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:46.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5adb5f8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40c2aa0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5adb5f8640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad40c3ab0 con 0x7f5ad415a2e0 2023-10-30T13:44:46.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5adb5f8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad40706e0 con 0x7f5ad41537f0 2023-10-30T13:44:46.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad8b93640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5ad415a2e0 0x7f5ad40ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:46.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad8b93640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5ad415a2e0 0x7f5ad40ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57166/0 (socket says 172.21.15.31:57166) 2023-10-30T13:44:46.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad8b93640 1 -- 172.21.15.31:0/3167320247 learned_addr learned my addr 172.21.15.31:0/3167320247 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:46.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad9394640 1 --2- 172.21.15.31:0/3167320247 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad41537f0 0x7f5ad40701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:46.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad8b93640 1 -- 172.21.15.31:0/3167320247 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad41537f0 msgr2=0x7f5ad40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:46.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad8b93640 1 --2- 172.21.15.31:0/3167320247 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad41537f0 0x7f5ad40701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad8b93640 1 -- 172.21.15.31:0/3167320247 >> v1:172.21.15.33:6789/0 conn(0x7f5ad40cd4a0 legacy=0x7f5ad406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.406+0000 7f5ad8b93640 1 -- 172.21.15.31:0/3167320247 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad40708e0 con 0x7f5ad415a2e0 2023-10-30T13:44:46.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5ad8b93640 1 --2- 172.21.15.31:0/3167320247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad415a2e0 0x7f5ad40ccf60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009870 tx=0x7f5ac401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61368f3985117f15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:46.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5acbfff640 1 -- 172.21.15.31:0/3167320247 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40a8a30 con 0x7f5ad415a2e0 2023-10-30T13:44:46.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5acbfff640 1 -- 172.21.15.31:0/3167320247 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac40a8b90 con 0x7f5ad415a2e0 2023-10-30T13:44:46.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5acbfff640 1 -- 172.21.15.31:0/3167320247 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40b0440 con 0x7f5ad415a2e0 2023-10-30T13:44:46.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 -- 172.21.15.31:0/3167320247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad415a2e0 msgr2=0x7f5ad40ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/3167320247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad415a2e0 0x7f5ad40ccf60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009870 tx=0x7f5ac401ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:46.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 -- 172.21.15.31:0/3167320247 shutdown_connections 2023-10-30T13:44:46.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/3167320247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad415a2e0 0x7f5ad40ccf60 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/3167320247 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad41537f0 0x7f5ad40701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 -- 172.21.15.31:0/3167320247 >> 172.21.15.31:0/3167320247 conn(0x7f5ad4071fb0 msgr2=0x7f5ad40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:46.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 -- 172.21.15.31:0/3167320247 shutdown_connections 2023-10-30T13:44:46.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 -- 172.21.15.31:0/3167320247 wait complete. 2023-10-30T13:44:46.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 Processor -- start 2023-10-30T13:44:46.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.410+0000 7f5adb5f8640 1 -- start start 2023-10-30T13:44:46.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad40cd4a0 0x7f5ad41efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:46.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ad41537f0 0x7f5ad40cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:46.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad415a2e0 0x7f5ad414c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:46.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4155360 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad40c7130 con 0x7f5ad41537f0 2023-10-30T13:44:46.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4159900 con 0x7f5ad415a2e0 2023-10-30T13:44:46.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad40cd4a0 0x7f5ad41efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:46.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad40cd4a0 0x7f5ad41efdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57170/0 (socket says 172.21.15.31:57170) 2023-10-30T13:44:46.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 -- 172.21.15.31:0/468315889 learned_addr learned my addr 172.21.15.31:0/468315889 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:46.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad8b93640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ad41537f0 0x7f5ad40cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:46.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9b95640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad415a2e0 0x7f5ad414c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:46.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 -- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad415a2e0 msgr2=0x7f5ad414c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad415a2e0 0x7f5ad414c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 -- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ad41537f0 msgr2=0x7f5ad40cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ad41537f0 0x7f5ad40cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 -- 172.21.15.31:0/468315889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac40094d0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ad9394640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad40cd4a0 0x7f5ad41efdf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000e9e0 tx=0x7f5ad000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:46.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ac9ffb640 1 -- 172.21.15.31:0/468315889 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad00096b0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ac9ffb640 1 -- 172.21.15.31:0/468315889 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad000c970 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad4154da0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5ac9ffb640 1 -- 172.21.15.31:0/468315889 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad000cd00 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad4159340 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.414+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad41591a0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.418+0000 7f5ac9ffb640 1 -- 172.21.15.31:0/468315889 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ad0013470 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.418+0000 7f5ac9ffb640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ab80664a0 0x7f5ab8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:46.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.418+0000 7f5ac9ffb640 1 -- 172.21.15.31:0/468315889 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f5ad00843a0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.418+0000 7f5ad8b93640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ab80664a0 0x7f5ab8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:46.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.418+0000 7f5ad8b93640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ab80664a0 0x7f5ab8068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40b2000 tx=0x7f5ac40aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:46.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.422+0000 7f5ac9ffb640 1 -- 172.21.15.31:0/468315889 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ad41591a0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.614+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ad41574a0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.614+0000 7f5ac9ffb640 1 -- 172.21.15.31:0/468315889 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f5ad41574a0 con 0x7f5ad40cd4a0 2023-10-30T13:44:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:46.619 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:43.334469+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:46.620 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 57 2023-10-30T13:44:46.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ab80664a0 msgr2=0x7f5ab8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ab80664a0 0x7f5ab8068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40b2000 tx=0x7f5ac40aa070 comp rx=0 tx=0).stop 2023-10-30T13:44:46.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad40cd4a0 msgr2=0x7f5ad41efdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad40cd4a0 0x7f5ad41efdf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000e9e0 tx=0x7f5ad000eea0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 shutdown_connections 2023-10-30T13:44:46.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5ab80664a0 0x7f5ab8068960 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5ad415a2e0 0x7f5ad414c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ad41537f0 0x7f5ad40cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 --2- 172.21.15.31:0/468315889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad40cd4a0 0x7f5ad41efdf0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 >> 172.21.15.31:0/468315889 conn(0x7f5ad4071fb0 msgr2=0x7f5ad4072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:46.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 shutdown_connections 2023-10-30T13:44:46.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:46.618+0000 7f5adb5f8640 1 -- 172.21.15.31:0/468315889 wait complete. 2023-10-30T13:44:46.640 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:48.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db80c9640 1 Processor -- start 2023-10-30T13:44:48.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db80c9640 1 -- start start 2023-10-30T13:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db80c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db0153c80 0x7f1db015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db80c9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1db00c42a0 0x7f1db00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db80c9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db0159f20 con 0x7f1db00c4a10 2023-10-30T13:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db80c9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db015dec0 con 0x7f1db00c42a0 2023-10-30T13:44:48.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db80c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db01599a0 con 0x7f1db0153c80 2023-10-30T13:44:48.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db6666640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1db00c42a0 0x7f1db00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db5664640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f1db00c4a10 0x7f1db015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46938/0 (socket says 172.21.15.31:46938) 2023-10-30T13:44:48.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db5664640 1 -- 172.21.15.31:0/39836706 learned_addr learned my addr 172.21.15.31:0/39836706 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:48.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db5e65640 1 --2- 172.21.15.31:0/39836706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db0153c80 0x7f1db015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db6666640 1 -- 172.21.15.31:0/39836706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db0153c80 msgr2=0x7f1db015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db6666640 1 --2- 172.21.15.31:0/39836706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db0153c80 0x7f1db015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db6666640 1 -- 172.21.15.31:0/39836706 >> v1:172.21.15.31:6789/0 conn(0x7f1db00c4a10 legacy=0x7f1db015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.782+0000 7f1db6666640 1 -- 172.21.15.31:0/39836706 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db015d620 con 0x7f1db00c42a0 2023-10-30T13:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db6666640 1 --2- 172.21.15.31:0/39836706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db00c42a0 0x7f1db00ca1e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1da8009d90 tx=0x7f1da801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57481c19217d7b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db4e63640 1 -- 172.21.15.31:0/39836706 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80af600 con 0x7f1db00c42a0 2023-10-30T13:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db4e63640 1 -- 172.21.15.31:0/39836706 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da80ab050 con 0x7f1db00c42a0 2023-10-30T13:44:48.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db4e63640 1 -- 172.21.15.31:0/39836706 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80af980 con 0x7f1db00c42a0 2023-10-30T13:44:48.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 -- 172.21.15.31:0/39836706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db00c42a0 msgr2=0x7f1db00ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 --2- 172.21.15.31:0/39836706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db00c42a0 0x7f1db00ca1e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1da8009d90 tx=0x7f1da801f130 comp rx=0 tx=0).stop 2023-10-30T13:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 -- 172.21.15.31:0/39836706 shutdown_connections 2023-10-30T13:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 --2- 172.21.15.31:0/39836706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db00c42a0 0x7f1db00ca1e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 --2- 172.21.15.31:0/39836706 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db0153c80 0x7f1db015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 -- 172.21.15.31:0/39836706 >> 172.21.15.31:0/39836706 conn(0x7f1db00715c0 msgr2=0x7f1db00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:48.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 -- 172.21.15.31:0/39836706 shutdown_connections 2023-10-30T13:44:48.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.786+0000 7f1db80c9640 1 -- 172.21.15.31:0/39836706 wait complete. 2023-10-30T13:44:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 Processor -- start 2023-10-30T13:44:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 -- start start 2023-10-30T13:44:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db00c42a0 0x7f1db015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1db00c4a10 0x7f1db00cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db0153c80 0x7f1db00cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db014f310 con 0x7f1db0153c80 2023-10-30T13:44:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db014fb50 con 0x7f1db00c4a10 2023-10-30T13:44:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db014e9e0 con 0x7f1db00c42a0 2023-10-30T13:44:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db0153c80 0x7f1db00cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db0153c80 0x7f1db00cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57188/0 (socket says 172.21.15.31:57188) 2023-10-30T13:44:48.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 -- 172.21.15.31:0/1302700867 learned_addr learned my addr 172.21.15.31:0/1302700867 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:48.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db5e65640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db00c42a0 0x7f1db015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db5664640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1db00c4a10 0x7f1db00cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 -- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db00c42a0 msgr2=0x7f1db015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db00c42a0 0x7f1db015b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 -- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1db00c4a10 msgr2=0x7f1db00cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1db00c4a10 0x7f1db00cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 -- 172.21.15.31:0/1302700867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da8009a70 con 0x7f1db0153c80 2023-10-30T13:44:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db5e65640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db00c42a0 0x7f1db015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db6666640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db0153c80 0x7f1db00cd610 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1da80b2040 tx=0x7f1da801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1da67fc640 1 -- 172.21.15.31:0/1302700867 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80b9610 con 0x7f1db0153c80 2023-10-30T13:44:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1da67fc640 1 -- 172.21.15.31:0/1302700867 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da80b9770 con 0x7f1db0153c80 2023-10-30T13:44:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db014ecc0 con 0x7f1db0153c80 2023-10-30T13:44:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1da67fc640 1 -- 172.21.15.31:0/1302700867 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80b9b20 con 0x7f1db0153c80 2023-10-30T13:44:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.790+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db014fdf0 con 0x7f1db0153c80 2023-10-30T13:44:48.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.794+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db014f4c0 con 0x7f1db0153c80 2023-10-30T13:44:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.794+0000 7f1da67fc640 1 -- 172.21.15.31:0/1302700867 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1da80b6050 con 0x7f1db0153c80 2023-10-30T13:44:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.794+0000 7f1da67fc640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1d940664a0 0x7f1d94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.794+0000 7f1da67fc640 1 -- 172.21.15.31:0/1302700867 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f1da81228f0 con 0x7f1db0153c80 2023-10-30T13:44:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.794+0000 7f1db5e65640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1d940664a0 0x7f1d94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.794+0000 7f1db5e65640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1d940664a0 0x7f1d94068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00b9f0 tx=0x7f1dac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:48.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.798+0000 7f1da67fc640 1 -- 172.21.15.31:0/1302700867 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1db014f4c0 con 0x7f1db0153c80 2023-10-30T13:44:48.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1db014f4c0 con 0x7f1db0153c80 2023-10-30T13:44:48.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1da67fc640 1 -- 172.21.15.31:0/1302700867 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f1db014f4c0 con 0x7f1db0153c80 2023-10-30T13:44:48.997 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:48.997 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:48.997 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 58 2023-10-30T13:44:48.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1d940664a0 msgr2=0x7f1d94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1d940664a0 0x7f1d94068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00b9f0 tx=0x7f1dac008040 comp rx=0 tx=0).stop 2023-10-30T13:44:48.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db0153c80 msgr2=0x7f1db00cd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:49.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db0153c80 0x7f1db00cd610 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1da80b2040 tx=0x7f1da801bed0 comp rx=0 tx=0).stop 2023-10-30T13:44:49.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 shutdown_connections 2023-10-30T13:44:49.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f1d940664a0 0x7f1d94068960 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:49.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db0153c80 0x7f1db00cd610 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:49.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1db00c4a10 0x7f1db00cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 --2- 172.21.15.31:0/1302700867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1db00c42a0 0x7f1db015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 >> 172.21.15.31:0/1302700867 conn(0x7f1db00715c0 msgr2=0x7f1db0158590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 shutdown_connections 2023-10-30T13:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:48.994+0000 7f1db80c9640 1 -- 172.21.15.31:0/1302700867 wait complete. 2023-10-30T13:44:49.018 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5470 2023-10-30T13:44:49.018 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.d 2023-10-30T13:44:52.019 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:44:52.019 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:44:52.020 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:44:52.022 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:44:52.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:52.103 INFO:tasks.ceph.mds.d.smithi033.stdout:starting mds.d at 2023-10-30T13:44:52.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe452f9d640 1 Processor -- start 2023-10-30T13:44:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe452f9d640 1 -- start start 2023-10-30T13:44:52.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe452f9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe44c0cb7d0 0x7fe44c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe452f9d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44c0c2310 con 0x7fe44c150830 2023-10-30T13:44:52.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe452f9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44c156710 con 0x7fe44c0c3e40 2023-10-30T13:44:52.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe452f9d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44c06bc10 con 0x7fe44c0cb7d0 2023-10-30T13:44:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe44bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe44c0cb7d0 0x7fe44c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe450d39640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe44c150830 0x7fe44c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46940/0 (socket says 172.21.15.31:46940) 2023-10-30T13:44:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.174+0000 7fe450d39640 1 -- 172.21.15.31:0/582555923 learned_addr learned my addr 172.21.15.31:0/582555923 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:44:52.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44bfff640 1 -- 172.21.15.31:0/582555923 >> v1:172.21.15.33:6789/0 conn(0x7fe44c0c3e40 legacy=0x7fe44c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44bfff640 1 -- 172.21.15.31:0/582555923 >> v1:172.21.15.31:6789/0 conn(0x7fe44c150830 legacy=0x7fe44c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44bfff640 1 -- 172.21.15.31:0/582555923 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c0c4360 con 0x7fe44c0cb7d0 2023-10-30T13:44:52.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44b7fe640 1 -- 172.21.15.31:0/582555923 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491116565 0 0) 0x7fe44c0c2310 con 0x7fe44c150830 2023-10-30T13:44:52.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44bfff640 1 --2- 172.21.15.31:0/582555923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c0cb7d0 0x7fe44c0c3900 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe4380097c0 tx=0x7fe43801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d5c4d731869eed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:52.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44b7fe640 1 -- 172.21.15.31:0/582555923 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380a8970 con 0x7fe44c0cb7d0 2023-10-30T13:44:52.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44b7fe640 1 -- 172.21.15.31:0/582555923 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4380a8ad0 con 0x7fe44c0cb7d0 2023-10-30T13:44:52.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe44b7fe640 1 -- 172.21.15.31:0/582555923 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380a8e60 con 0x7fe44c0cb7d0 2023-10-30T13:44:52.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe452f9d640 1 -- 172.21.15.31:0/582555923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c0cb7d0 msgr2=0x7fe44c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe452f9d640 1 --2- 172.21.15.31:0/582555923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c0cb7d0 0x7fe44c0c3900 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe4380097c0 tx=0x7fe43801ec70 comp rx=0 tx=0).stop 2023-10-30T13:44:52.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe452f9d640 1 -- 172.21.15.31:0/582555923 shutdown_connections 2023-10-30T13:44:52.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe452f9d640 1 --2- 172.21.15.31:0/582555923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c0cb7d0 0x7fe44c0c3900 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.178+0000 7fe452f9d640 1 -- 172.21.15.31:0/582555923 >> 172.21.15.31:0/582555923 conn(0x7fe44c0cac70 msgr2=0x7fe44c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:52.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 -- 172.21.15.31:0/582555923 shutdown_connections 2023-10-30T13:44:52.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 -- 172.21.15.31:0/582555923 wait complete. 2023-10-30T13:44:52.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 Processor -- start 2023-10-30T13:44:52.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 -- start start 2023-10-30T13:44:52.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe44c0c3e40 0x7fe44c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe44c0cb7d0 0x7fe44c1e3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c150830 0x7fe44c1e3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c1e1410 con 0x7fe44c150830 2023-10-30T13:44:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c1e26f0 con 0x7fe44c0cb7d0 2023-10-30T13:44:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe452f9d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c14cf10 con 0x7fe44c0c3e40 2023-10-30T13:44:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c150830 0x7fe44c1e3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c150830 0x7fe44c1e3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57206/0 (socket says 172.21.15.31:57206) 2023-10-30T13:44:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 -- 172.21.15.31:0/3400285725 learned_addr learned my addr 172.21.15.31:0/3400285725 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe450d39640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe44c0c3e40 0x7fe44c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe44bfff640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe44c0cb7d0 0x7fe44c1e3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 -- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe44c0c3e40 msgr2=0x7fe44c159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe44c0c3e40 0x7fe44c159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 -- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe44c0cb7d0 msgr2=0x7fe44c1e3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe44c0cb7d0 0x7fe44c1e3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 -- 172.21.15.31:0/3400285725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe438009420 con 0x7fe44c150830 2023-10-30T13:44:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe450d39640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe44c0c3e40 0x7fe44c159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe45153a640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c150830 0x7fe44c1e3b90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c7a0 tx=0x7fe44400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.182+0000 7fe4497fa640 1 -- 172.21.15.31:0/3400285725 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444003b80 con 0x7fe44c150830 2023-10-30T13:44:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe4497fa640 1 -- 172.21.15.31:0/3400285725 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe444003ce0 con 0x7fe44c150830 2023-10-30T13:44:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44c1e0150 con 0x7fe44c150830 2023-10-30T13:44:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe4497fa640 1 -- 172.21.15.31:0/3400285725 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44400aa30 con 0x7fe44c150830 2023-10-30T13:44:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44c1e1130 con 0x7fe44c150830 2023-10-30T13:44:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44c04e170 con 0x7fe44c150830 2023-10-30T13:44:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe4497fa640 1 -- 172.21.15.31:0/3400285725 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe44400ab90 con 0x7fe44c150830 2023-10-30T13:44:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe4497fa640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4300664d0 0x7fe430068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe4497fa640 1 -- 172.21.15.31:0/3400285725 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fe444083f00 con 0x7fe44c150830 2023-10-30T13:44:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.186+0000 7fe450d39640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4300664d0 0x7fe430068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.190+0000 7fe450d39640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4300664d0 0x7fe430068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe440000f70 tx=0x7fe440009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:52.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.190+0000 7fe4497fa640 1 -- 172.21.15.31:0/3400285725 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4440510a0 con 0x7fe44c150830 2023-10-30T13:44:52.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe44c070ac0 con 0x7fe44c150830 2023-10-30T13:44:52.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe4497fa640 1 -- 172.21.15.31:0/3400285725 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5699 (secure 0 0 0) 0x7fe4440546c0 con 0x7fe44c150830 2023-10-30T13:44:52.389 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:52.389 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:52.390 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 58 2023-10-30T13:44:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4300664d0 msgr2=0x7fe430068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4300664d0 0x7fe430068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe440000f70 tx=0x7fe440009040 comp rx=0 tx=0).stop 2023-10-30T13:44:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c150830 msgr2=0x7fe44c1e3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c150830 0x7fe44c1e3b90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c7a0 tx=0x7fe44400cc60 comp rx=0 tx=0).stop 2023-10-30T13:44:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 shutdown_connections 2023-10-30T13:44:52.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4300664d0 0x7fe430068990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c150830 0x7fe44c1e3b90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe44c0cb7d0 0x7fe44c1e3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 --2- 172.21.15.31:0/3400285725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe44c0c3e40 0x7fe44c159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 >> 172.21.15.31:0/3400285725 conn(0x7fe44c0cac70 msgr2=0x7fe44c154470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:52.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 shutdown_connections 2023-10-30T13:44:52.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:52.386+0000 7fe452f9d640 1 -- 172.21.15.31:0/3400285725 wait complete. 2023-10-30T13:44:52.409 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:44:53.942 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/25' 2023-10-30T13:44:53.943 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/30' 2023-10-30T13:44:53.943 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/33' 2023-10-30T13:44:53.943 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/23' 2023-10-30T13:44:53.943 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/35' 2023-10-30T13:44:53.944 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/26' 2023-10-30T13:44:53.944 INFO:tasks.workunit.client.0.smithi031.stdout:removed './246/37' 2023-10-30T13:44:53.944 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './246' 2023-10-30T13:44:53.944 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/21' 2023-10-30T13:44:53.944 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/27' 2023-10-30T13:44:53.945 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/31' 2023-10-30T13:44:53.945 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/36' 2023-10-30T13:44:53.945 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/20' 2023-10-30T13:44:53.945 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/39' 2023-10-30T13:44:53.945 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/34' 2023-10-30T13:44:53.945 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/38' 2023-10-30T13:44:53.946 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/28' 2023-10-30T13:44:53.946 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/24' 2023-10-30T13:44:53.946 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/29' 2023-10-30T13:44:53.946 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/32' 2023-10-30T13:44:53.946 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/22' 2023-10-30T13:44:53.947 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/25' 2023-10-30T13:44:53.947 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/30' 2023-10-30T13:44:53.947 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/33' 2023-10-30T13:44:53.947 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/23' 2023-10-30T13:44:53.947 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/35' 2023-10-30T13:44:53.947 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/26' 2023-10-30T13:44:53.948 INFO:tasks.workunit.client.0.smithi031.stdout:removed './247/37' 2023-10-30T13:44:53.948 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './247' 2023-10-30T13:44:53.948 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/21' 2023-10-30T13:44:53.948 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/27' 2023-10-30T13:44:53.948 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/31' 2023-10-30T13:44:53.949 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/36' 2023-10-30T13:44:53.949 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/20' 2023-10-30T13:44:53.949 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/39' 2023-10-30T13:44:53.949 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/34' 2023-10-30T13:44:53.949 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/38' 2023-10-30T13:44:53.949 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/28' 2023-10-30T13:44:53.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/24' 2023-10-30T13:44:53.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/29' 2023-10-30T13:44:53.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/32' 2023-10-30T13:44:53.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/22' 2023-10-30T13:44:53.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/25' 2023-10-30T13:44:53.950 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/30' 2023-10-30T13:44:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/33' 2023-10-30T13:44:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/23' 2023-10-30T13:44:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/35' 2023-10-30T13:44:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/26' 2023-10-30T13:44:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed './248/37' 2023-10-30T13:44:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './248' 2023-10-30T13:44:53.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/21' 2023-10-30T13:44:53.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/27' 2023-10-30T13:44:53.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/31' 2023-10-30T13:44:53.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/36' 2023-10-30T13:44:53.952 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/20' 2023-10-30T13:44:53.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/39' 2023-10-30T13:44:53.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/34' 2023-10-30T13:44:53.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/38' 2023-10-30T13:44:53.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/28' 2023-10-30T13:44:53.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/24' 2023-10-30T13:44:53.953 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/29' 2023-10-30T13:44:53.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/32' 2023-10-30T13:44:53.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/22' 2023-10-30T13:44:53.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/25' 2023-10-30T13:44:53.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/30' 2023-10-30T13:44:53.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/33' 2023-10-30T13:44:53.954 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/23' 2023-10-30T13:44:53.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/35' 2023-10-30T13:44:53.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/26' 2023-10-30T13:44:53.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './249/37' 2023-10-30T13:44:53.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './249' 2023-10-30T13:44:53.955 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/21' 2023-10-30T13:44:53.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/27' 2023-10-30T13:44:53.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/31' 2023-10-30T13:44:53.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/36' 2023-10-30T13:44:53.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/20' 2023-10-30T13:44:53.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/39' 2023-10-30T13:44:53.956 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/34' 2023-10-30T13:44:53.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/38' 2023-10-30T13:44:53.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/28' 2023-10-30T13:44:53.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/24' 2023-10-30T13:44:53.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/29' 2023-10-30T13:44:53.957 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/32' 2023-10-30T13:44:53.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/22' 2023-10-30T13:44:53.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/25' 2023-10-30T13:44:53.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/30' 2023-10-30T13:44:53.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/33' 2023-10-30T13:44:53.958 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/23' 2023-10-30T13:44:53.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/35' 2023-10-30T13:44:53.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/26' 2023-10-30T13:44:53.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './250/37' 2023-10-30T13:44:53.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './250' 2023-10-30T13:44:53.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/21' 2023-10-30T13:44:53.959 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/27' 2023-10-30T13:44:53.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/31' 2023-10-30T13:44:53.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/36' 2023-10-30T13:44:53.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/20' 2023-10-30T13:44:53.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/39' 2023-10-30T13:44:53.960 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/34' 2023-10-30T13:44:53.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/38' 2023-10-30T13:44:53.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/28' 2023-10-30T13:44:53.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/24' 2023-10-30T13:44:53.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/29' 2023-10-30T13:44:53.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/32' 2023-10-30T13:44:53.961 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/22' 2023-10-30T13:44:53.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/25' 2023-10-30T13:44:53.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/30' 2023-10-30T13:44:53.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/33' 2023-10-30T13:44:53.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/23' 2023-10-30T13:44:53.962 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/35' 2023-10-30T13:44:53.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/26' 2023-10-30T13:44:53.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './251/37' 2023-10-30T13:44:53.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './251' 2023-10-30T13:44:53.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/21' 2023-10-30T13:44:53.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/27' 2023-10-30T13:44:53.963 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/31' 2023-10-30T13:44:53.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/36' 2023-10-30T13:44:53.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/20' 2023-10-30T13:44:53.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/39' 2023-10-30T13:44:53.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/34' 2023-10-30T13:44:53.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/38' 2023-10-30T13:44:53.964 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/28' 2023-10-30T13:44:53.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/24' 2023-10-30T13:44:53.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/29' 2023-10-30T13:44:53.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/32' 2023-10-30T13:44:53.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/22' 2023-10-30T13:44:53.965 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/25' 2023-10-30T13:44:53.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/30' 2023-10-30T13:44:53.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/33' 2023-10-30T13:44:53.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/23' 2023-10-30T13:44:53.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/35' 2023-10-30T13:44:53.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/26' 2023-10-30T13:44:53.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './252/37' 2023-10-30T13:44:53.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './252' 2023-10-30T13:44:53.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/21' 2023-10-30T13:44:53.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/27' 2023-10-30T13:44:53.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/31' 2023-10-30T13:44:53.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/36' 2023-10-30T13:44:53.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/20' 2023-10-30T13:44:53.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/39' 2023-10-30T13:44:53.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/34' 2023-10-30T13:44:53.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/38' 2023-10-30T13:44:53.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/28' 2023-10-30T13:44:53.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/24' 2023-10-30T13:44:53.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/29' 2023-10-30T13:44:53.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/32' 2023-10-30T13:44:53.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/22' 2023-10-30T13:44:53.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/25' 2023-10-30T13:44:53.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/30' 2023-10-30T13:44:53.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/33' 2023-10-30T13:44:53.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/23' 2023-10-30T13:44:53.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/35' 2023-10-30T13:44:53.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/26' 2023-10-30T13:44:53.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './253/37' 2023-10-30T13:44:53.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './253' 2023-10-30T13:44:53.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/21' 2023-10-30T13:44:53.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/27' 2023-10-30T13:44:53.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/31' 2023-10-30T13:44:53.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/36' 2023-10-30T13:44:53.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/20' 2023-10-30T13:44:53.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/39' 2023-10-30T13:44:53.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/34' 2023-10-30T13:44:53.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/38' 2023-10-30T13:44:53.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/28' 2023-10-30T13:44:53.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/24' 2023-10-30T13:44:53.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/29' 2023-10-30T13:44:53.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/32' 2023-10-30T13:44:53.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/22' 2023-10-30T13:44:53.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/25' 2023-10-30T13:44:53.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/30' 2023-10-30T13:44:53.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/33' 2023-10-30T13:44:53.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/23' 2023-10-30T13:44:53.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/35' 2023-10-30T13:44:53.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/26' 2023-10-30T13:44:53.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './254/37' 2023-10-30T13:44:53.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './254' 2023-10-30T13:44:53.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/21' 2023-10-30T13:44:53.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/27' 2023-10-30T13:44:53.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/31' 2023-10-30T13:44:53.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/36' 2023-10-30T13:44:53.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/20' 2023-10-30T13:44:53.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/39' 2023-10-30T13:44:53.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/34' 2023-10-30T13:44:53.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/38' 2023-10-30T13:44:53.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/28' 2023-10-30T13:44:53.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/24' 2023-10-30T13:44:53.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/29' 2023-10-30T13:44:53.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/32' 2023-10-30T13:44:53.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/22' 2023-10-30T13:44:53.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/25' 2023-10-30T13:44:53.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/30' 2023-10-30T13:44:53.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/33' 2023-10-30T13:44:53.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/23' 2023-10-30T13:44:53.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/35' 2023-10-30T13:44:53.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/26' 2023-10-30T13:44:53.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './255/37' 2023-10-30T13:44:53.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './255' 2023-10-30T13:44:53.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/21' 2023-10-30T13:44:53.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/27' 2023-10-30T13:44:53.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/31' 2023-10-30T13:44:53.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/36' 2023-10-30T13:44:53.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/20' 2023-10-30T13:44:53.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/39' 2023-10-30T13:44:53.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/34' 2023-10-30T13:44:53.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/38' 2023-10-30T13:44:53.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/28' 2023-10-30T13:44:53.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/24' 2023-10-30T13:44:53.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/29' 2023-10-30T13:44:53.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/32' 2023-10-30T13:44:53.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/22' 2023-10-30T13:44:53.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/25' 2023-10-30T13:44:53.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/30' 2023-10-30T13:44:54.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:54.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7849e640 1 Processor -- start 2023-10-30T13:44:54.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7849e640 1 -- start start 2023-10-30T13:44:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7849e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0b70150480 0x7f0b7006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7849e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b700c12c0 con 0x7f0b700c7710 2023-10-30T13:44:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7849e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b700c1e40 con 0x7f0b700c0ce0 2023-10-30T13:44:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7849e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b700bff00 con 0x7f0b70150480 2023-10-30T13:44:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7623a640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0b70150480 0x7f0b7006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:54.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7623a640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0b70150480 0x7f0b7006fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57210/0 (socket says 172.21.15.31:57210) 2023-10-30T13:44:54.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7623a640 1 -- 172.21.15.31:0/2757511087 learned_addr learned my addr 172.21.15.31:0/2757511087 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:54.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b75238640 1 -- 172.21.15.31:0/2757511087 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677271913 0 0) 0x7f0b700c12c0 con 0x7f0b700c7710 2023-10-30T13:44:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7623a640 1 -- 172.21.15.31:0/2757511087 >> v1:172.21.15.33:6789/0 conn(0x7f0b700c0ce0 legacy=0x7f0b70160ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7623a640 1 -- 172.21.15.31:0/2757511087 >> v1:172.21.15.31:6789/0 conn(0x7f0b700c7710 legacy=0x7f0b700c05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.562+0000 7f0b7623a640 1 -- 172.21.15.31:0/2757511087 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700c1510 con 0x7f0b70150480 2023-10-30T13:44:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7623a640 1 --2- 172.21.15.31:0/2757511087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b70150480 0x7f0b7006fdf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00bb10 tx=0x7f0b6c021400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa188e033bc75188 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b75238640 1 -- 172.21.15.31:0/2757511087 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0b1600 con 0x7f0b70150480 2023-10-30T13:44:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b75238640 1 -- 172.21.15.31:0/2757511087 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c0ad050 con 0x7f0b70150480 2023-10-30T13:44:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b75238640 1 -- 172.21.15.31:0/2757511087 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0b1980 con 0x7f0b70150480 2023-10-30T13:44:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7849e640 1 -- 172.21.15.31:0/2757511087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b70150480 msgr2=0x7f0b7006fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:54.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7849e640 1 --2- 172.21.15.31:0/2757511087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b70150480 0x7f0b7006fdf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00bb10 tx=0x7f0b6c021400 comp rx=0 tx=0).stop 2023-10-30T13:44:54.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7849e640 1 -- 172.21.15.31:0/2757511087 shutdown_connections 2023-10-30T13:44:54.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7849e640 1 --2- 172.21.15.31:0/2757511087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b70150480 0x7f0b7006fdf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7849e640 1 -- 172.21.15.31:0/2757511087 >> 172.21.15.31:0/2757511087 conn(0x7f0b70069e90 msgr2=0x7f0b7006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:54.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7849e640 1 -- 172.21.15.31:0/2757511087 shutdown_connections 2023-10-30T13:44:54.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.566+0000 7f0b7849e640 1 -- 172.21.15.31:0/2757511087 wait complete. 2023-10-30T13:44:54.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 Processor -- start 2023-10-30T13:44:54.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 -- start start 2023-10-30T13:44:54.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b700c0ce0 0x7f0b7014f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:54.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0b700c7710 0x7f0b701442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:54.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b70150480 0x7f0b701447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:54.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b70143970 con 0x7f0b700c0ce0 2023-10-30T13:44:54.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7014d360 con 0x7f0b70150480 2023-10-30T13:44:54.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b70142160 con 0x7f0b700c7710 2023-10-30T13:44:54.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b700c0ce0 0x7f0b7014f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:54.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b700c0ce0 0x7f0b7014f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57216/0 (socket says 172.21.15.31:57216) 2023-10-30T13:44:54.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 -- 172.21.15.31:0/32065096 learned_addr learned my addr 172.21.15.31:0/32065096 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:44:54.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b75a39640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0b700c7710 0x7f0b701442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b76a3b640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b70150480 0x7f0b701447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 -- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0b700c7710 msgr2=0x7f0b701442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0b700c7710 0x7f0b701442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 -- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b70150480 msgr2=0x7f0b701447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b70150480 0x7f0b701447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 -- 172.21.15.31:0/32065096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c00b770 con 0x7f0b700c0ce0 2023-10-30T13:44:54.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b76a3b640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b70150480 0x7f0b701447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:54.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7623a640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b700c0ce0 0x7f0b7014f5e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00b2c0 tx=0x7f0b6c01e1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:54.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b66ffd640 1 -- 172.21.15.31:0/32065096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0ba600 con 0x7f0b700c0ce0 2023-10-30T13:44:54.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b66ffd640 1 -- 172.21.15.31:0/32065096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c0ba780 con 0x7f0b700c0ce0 2023-10-30T13:44:54.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b70142440 con 0x7f0b700c0ce0 2023-10-30T13:44:54.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b66ffd640 1 -- 172.21.15.31:0/32065096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0bab30 con 0x7f0b700c0ce0 2023-10-30T13:44:54.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7006bd70 con 0x7f0b700c0ce0 2023-10-30T13:44:54.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.570+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b70142620 con 0x7f0b700c0ce0 2023-10-30T13:44:54.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.574+0000 7f0b66ffd640 1 -- 172.21.15.31:0/32065096 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b6c0b8040 con 0x7f0b700c0ce0 2023-10-30T13:44:54.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.574+0000 7f0b66ffd640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b540664a0 0x7f0b54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:54.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.574+0000 7f0b66ffd640 1 -- 172.21.15.31:0/32065096 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f0b6c124550 con 0x7f0b700c0ce0 2023-10-30T13:44:54.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.574+0000 7f0b75a39640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b540664a0 0x7f0b54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:54.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.574+0000 7f0b75a39640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b540664a0 0x7f0b54068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0b70150db0 tx=0x7f0b60009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:54.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.578+0000 7f0b66ffd640 1 -- 172.21.15.31:0/32065096 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b6c0f1780 con 0x7f0b700c0ce0 2023-10-30T13:44:54.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.766+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b700c17c0 con 0x7f0b700c0ce0 2023-10-30T13:44:54.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b66ffd640 1 -- 172.21.15.31:0/32065096 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f0b6c0b6020 con 0x7f0b700c0ce0 2023-10-30T13:44:54.772 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:44:54.772 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:54.773 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:44:54.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b540664a0 msgr2=0x7f0b54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:54.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b540664a0 0x7f0b54068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0b70150db0 tx=0x7f0b60009040 comp rx=0 tx=0).stop 2023-10-30T13:44:54.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b700c0ce0 msgr2=0x7f0b7014f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:54.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b700c0ce0 0x7f0b7014f5e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00b2c0 tx=0x7f0b6c01e1a0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 shutdown_connections 2023-10-30T13:44:54.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0b540664a0 0x7f0b54068960 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b70150480 0x7f0b701447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0b700c7710 0x7f0b701442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 --2- 172.21.15.31:0/32065096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b700c0ce0 0x7f0b7014f5e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:54.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 >> 172.21.15.31:0/32065096 conn(0x7f0b70069e90 msgr2=0x7f0b700c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:54.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 shutdown_connections 2023-10-30T13:44:54.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:44:54.770+0000 7f0b7849e640 1 -- 172.21.15.31:0/32065096 wait complete. 2023-10-30T13:44:54.793 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:44:54.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 119 secs before thrashing 2023-10-30T13:44:54.914 INFO:tasks.ceph.mds.a.smithi031.stderr:2023-10-30T13:44:54.910+0000 7f44e788e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:44:56.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/33' 2023-10-30T13:44:56.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/23' 2023-10-30T13:44:56.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/35' 2023-10-30T13:44:56.069 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/26' 2023-10-30T13:44:56.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './256/37' 2023-10-30T13:44:56.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './256' 2023-10-30T13:44:56.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/21' 2023-10-30T13:44:56.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/27' 2023-10-30T13:44:56.070 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/31' 2023-10-30T13:44:56.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/36' 2023-10-30T13:44:56.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/20' 2023-10-30T13:44:56.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/39' 2023-10-30T13:44:56.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/34' 2023-10-30T13:44:56.071 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/38' 2023-10-30T13:44:56.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/28' 2023-10-30T13:44:56.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/24' 2023-10-30T13:44:56.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/29' 2023-10-30T13:44:56.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/32' 2023-10-30T13:44:56.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/22' 2023-10-30T13:44:56.072 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/25' 2023-10-30T13:44:56.073 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/30' 2023-10-30T13:44:56.073 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/33' 2023-10-30T13:44:56.073 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/23' 2023-10-30T13:44:56.073 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/35' 2023-10-30T13:44:56.073 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/26' 2023-10-30T13:44:56.074 INFO:tasks.workunit.client.0.smithi031.stdout:removed './257/37' 2023-10-30T13:44:56.074 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './257' 2023-10-30T13:44:56.074 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/21' 2023-10-30T13:44:56.074 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/27' 2023-10-30T13:44:56.074 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/31' 2023-10-30T13:44:56.075 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/36' 2023-10-30T13:44:56.075 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/20' 2023-10-30T13:44:56.075 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/39' 2023-10-30T13:44:56.075 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/34' 2023-10-30T13:44:56.075 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/38' 2023-10-30T13:44:56.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/28' 2023-10-30T13:44:56.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/24' 2023-10-30T13:44:56.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/29' 2023-10-30T13:44:56.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/32' 2023-10-30T13:44:56.076 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/22' 2023-10-30T13:44:56.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/25' 2023-10-30T13:44:56.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/30' 2023-10-30T13:44:56.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/33' 2023-10-30T13:44:56.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/23' 2023-10-30T13:44:56.077 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/35' 2023-10-30T13:44:56.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/26' 2023-10-30T13:44:56.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './258/37' 2023-10-30T13:44:56.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './258' 2023-10-30T13:44:56.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/21' 2023-10-30T13:44:56.078 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/27' 2023-10-30T13:44:56.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/31' 2023-10-30T13:44:56.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/36' 2023-10-30T13:44:56.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/20' 2023-10-30T13:44:56.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/39' 2023-10-30T13:44:56.079 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/34' 2023-10-30T13:44:56.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/38' 2023-10-30T13:44:56.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/28' 2023-10-30T13:44:56.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/24' 2023-10-30T13:44:56.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/29' 2023-10-30T13:44:56.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/32' 2023-10-30T13:44:56.080 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/22' 2023-10-30T13:44:56.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/25' 2023-10-30T13:44:56.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/30' 2023-10-30T13:44:56.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/33' 2023-10-30T13:44:56.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/23' 2023-10-30T13:44:56.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/35' 2023-10-30T13:44:56.081 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/26' 2023-10-30T13:44:56.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './259/37' 2023-10-30T13:44:56.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './259' 2023-10-30T13:44:56.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/21' 2023-10-30T13:44:56.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/27' 2023-10-30T13:44:56.082 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/31' 2023-10-30T13:44:56.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/36' 2023-10-30T13:44:56.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/20' 2023-10-30T13:44:56.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/39' 2023-10-30T13:44:56.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/34' 2023-10-30T13:44:56.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/38' 2023-10-30T13:44:56.083 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/28' 2023-10-30T13:44:56.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/24' 2023-10-30T13:44:56.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/29' 2023-10-30T13:44:56.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/32' 2023-10-30T13:44:56.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/22' 2023-10-30T13:44:56.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/25' 2023-10-30T13:44:56.084 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/30' 2023-10-30T13:44:56.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/33' 2023-10-30T13:44:56.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/23' 2023-10-30T13:44:56.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/35' 2023-10-30T13:44:56.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/26' 2023-10-30T13:44:56.085 INFO:tasks.workunit.client.0.smithi031.stdout:removed './260/37' 2023-10-30T13:44:56.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './260' 2023-10-30T13:44:56.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/21' 2023-10-30T13:44:56.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/27' 2023-10-30T13:44:56.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/31' 2023-10-30T13:44:56.086 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/36' 2023-10-30T13:44:56.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/20' 2023-10-30T13:44:56.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/39' 2023-10-30T13:44:56.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/34' 2023-10-30T13:44:56.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/38' 2023-10-30T13:44:56.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/28' 2023-10-30T13:44:56.087 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/24' 2023-10-30T13:44:56.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/29' 2023-10-30T13:44:56.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/32' 2023-10-30T13:44:56.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/22' 2023-10-30T13:44:56.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/25' 2023-10-30T13:44:56.088 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/30' 2023-10-30T13:44:56.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/33' 2023-10-30T13:44:56.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/23' 2023-10-30T13:44:56.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/35' 2023-10-30T13:44:56.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/26' 2023-10-30T13:44:56.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed './261/37' 2023-10-30T13:44:56.089 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './261' 2023-10-30T13:44:56.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/21' 2023-10-30T13:44:56.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/27' 2023-10-30T13:44:56.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/31' 2023-10-30T13:44:56.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/36' 2023-10-30T13:44:56.090 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/20' 2023-10-30T13:44:56.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/39' 2023-10-30T13:44:56.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/34' 2023-10-30T13:44:56.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/38' 2023-10-30T13:44:56.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/28' 2023-10-30T13:44:56.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/24' 2023-10-30T13:44:56.091 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/29' 2023-10-30T13:44:56.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/32' 2023-10-30T13:44:56.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/22' 2023-10-30T13:44:56.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/25' 2023-10-30T13:44:56.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/30' 2023-10-30T13:44:56.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/33' 2023-10-30T13:44:56.092 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/23' 2023-10-30T13:44:56.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/35' 2023-10-30T13:44:56.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/26' 2023-10-30T13:44:56.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './262/37' 2023-10-30T13:44:56.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './262' 2023-10-30T13:44:56.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/21' 2023-10-30T13:44:56.093 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/27' 2023-10-30T13:44:56.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/31' 2023-10-30T13:44:56.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/36' 2023-10-30T13:44:56.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/20' 2023-10-30T13:44:56.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/39' 2023-10-30T13:44:56.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/34' 2023-10-30T13:44:56.094 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/38' 2023-10-30T13:44:56.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/28' 2023-10-30T13:44:56.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/24' 2023-10-30T13:44:56.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/29' 2023-10-30T13:44:56.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/32' 2023-10-30T13:44:56.095 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/22' 2023-10-30T13:44:56.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/25' 2023-10-30T13:44:56.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/30' 2023-10-30T13:44:56.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/33' 2023-10-30T13:44:56.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/23' 2023-10-30T13:44:56.096 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/35' 2023-10-30T13:44:56.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/26' 2023-10-30T13:44:56.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './263/37' 2023-10-30T13:44:56.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './263' 2023-10-30T13:44:56.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/21' 2023-10-30T13:44:56.097 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/27' 2023-10-30T13:44:56.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/31' 2023-10-30T13:44:56.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/36' 2023-10-30T13:44:56.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/20' 2023-10-30T13:44:56.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/39' 2023-10-30T13:44:56.098 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/34' 2023-10-30T13:44:56.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/38' 2023-10-30T13:44:56.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/28' 2023-10-30T13:44:56.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/24' 2023-10-30T13:44:56.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/29' 2023-10-30T13:44:56.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/32' 2023-10-30T13:44:56.099 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/22' 2023-10-30T13:44:56.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/25' 2023-10-30T13:44:56.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/30' 2023-10-30T13:44:56.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/33' 2023-10-30T13:44:56.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/23' 2023-10-30T13:44:56.100 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/35' 2023-10-30T13:44:56.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/26' 2023-10-30T13:44:56.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './264/37' 2023-10-30T13:44:56.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './264' 2023-10-30T13:44:56.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/21' 2023-10-30T13:44:56.101 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/27' 2023-10-30T13:44:56.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/31' 2023-10-30T13:44:56.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/36' 2023-10-30T13:44:56.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/20' 2023-10-30T13:44:56.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/39' 2023-10-30T13:44:56.102 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/34' 2023-10-30T13:44:56.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/38' 2023-10-30T13:44:56.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/28' 2023-10-30T13:44:56.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/24' 2023-10-30T13:44:56.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/29' 2023-10-30T13:44:56.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/32' 2023-10-30T13:44:56.103 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/22' 2023-10-30T13:44:56.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/25' 2023-10-30T13:44:56.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/30' 2023-10-30T13:44:56.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/33' 2023-10-30T13:44:56.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/23' 2023-10-30T13:44:56.104 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/35' 2023-10-30T13:44:56.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/26' 2023-10-30T13:44:56.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './265/37' 2023-10-30T13:44:56.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './265' 2023-10-30T13:44:56.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/21' 2023-10-30T13:44:56.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/27' 2023-10-30T13:44:56.105 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/31' 2023-10-30T13:44:56.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/36' 2023-10-30T13:44:56.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/20' 2023-10-30T13:44:56.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/39' 2023-10-30T13:44:56.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/34' 2023-10-30T13:44:56.106 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/38' 2023-10-30T13:44:56.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/28' 2023-10-30T13:44:56.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/24' 2023-10-30T13:44:56.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/29' 2023-10-30T13:44:56.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/32' 2023-10-30T13:44:56.107 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/22' 2023-10-30T13:44:56.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/25' 2023-10-30T13:44:56.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/30' 2023-10-30T13:44:56.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/33' 2023-10-30T13:44:56.108 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/23' 2023-10-30T13:44:56.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:56.468 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:57.276 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/35' 2023-10-30T13:44:57.276 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/26' 2023-10-30T13:44:57.276 INFO:tasks.workunit.client.0.smithi031.stdout:removed './266/37' 2023-10-30T13:44:57.277 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './266' 2023-10-30T13:44:57.277 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/21' 2023-10-30T13:44:57.277 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/27' 2023-10-30T13:44:57.277 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/31' 2023-10-30T13:44:57.277 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/36' 2023-10-30T13:44:57.278 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/20' 2023-10-30T13:44:57.278 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/39' 2023-10-30T13:44:57.278 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/34' 2023-10-30T13:44:57.278 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/38' 2023-10-30T13:44:57.278 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/28' 2023-10-30T13:44:57.279 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/24' 2023-10-30T13:44:57.279 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/29' 2023-10-30T13:44:57.279 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/32' 2023-10-30T13:44:57.279 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/22' 2023-10-30T13:44:57.279 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/25' 2023-10-30T13:44:57.280 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/30' 2023-10-30T13:44:57.280 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/33' 2023-10-30T13:44:57.280 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/23' 2023-10-30T13:44:57.280 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/35' 2023-10-30T13:44:57.280 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/26' 2023-10-30T13:44:57.280 INFO:tasks.workunit.client.0.smithi031.stdout:removed './267/37' 2023-10-30T13:44:57.281 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './267' 2023-10-30T13:44:57.281 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/21' 2023-10-30T13:44:57.281 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/27' 2023-10-30T13:44:57.281 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/31' 2023-10-30T13:44:57.281 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/36' 2023-10-30T13:44:57.281 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/20' 2023-10-30T13:44:57.282 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/39' 2023-10-30T13:44:57.282 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/34' 2023-10-30T13:44:57.282 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/38' 2023-10-30T13:44:57.282 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/28' 2023-10-30T13:44:57.282 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/24' 2023-10-30T13:44:57.282 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/29' 2023-10-30T13:44:57.283 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/32' 2023-10-30T13:44:57.283 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/22' 2023-10-30T13:44:57.283 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/25' 2023-10-30T13:44:57.283 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/30' 2023-10-30T13:44:57.283 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/33' 2023-10-30T13:44:57.283 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/23' 2023-10-30T13:44:57.284 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/35' 2023-10-30T13:44:57.284 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/26' 2023-10-30T13:44:57.284 INFO:tasks.workunit.client.0.smithi031.stdout:removed './268/37' 2023-10-30T13:44:57.284 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './268' 2023-10-30T13:44:57.284 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/21' 2023-10-30T13:44:57.284 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/27' 2023-10-30T13:44:57.285 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/31' 2023-10-30T13:44:57.285 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/36' 2023-10-30T13:44:57.285 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/20' 2023-10-30T13:44:57.285 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/39' 2023-10-30T13:44:57.285 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/34' 2023-10-30T13:44:57.286 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/38' 2023-10-30T13:44:57.286 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/28' 2023-10-30T13:44:57.286 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/24' 2023-10-30T13:44:57.286 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/29' 2023-10-30T13:44:57.286 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/32' 2023-10-30T13:44:57.286 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/22' 2023-10-30T13:44:57.287 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/25' 2023-10-30T13:44:57.287 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/30' 2023-10-30T13:44:57.287 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/33' 2023-10-30T13:44:57.287 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/23' 2023-10-30T13:44:57.287 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/35' 2023-10-30T13:44:57.288 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/26' 2023-10-30T13:44:57.288 INFO:tasks.workunit.client.0.smithi031.stdout:removed './269/37' 2023-10-30T13:44:57.288 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './269' 2023-10-30T13:44:57.288 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/21' 2023-10-30T13:44:57.288 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/27' 2023-10-30T13:44:57.288 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/31' 2023-10-30T13:44:57.289 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/36' 2023-10-30T13:44:57.289 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/20' 2023-10-30T13:44:57.289 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/39' 2023-10-30T13:44:57.289 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/34' 2023-10-30T13:44:57.289 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/38' 2023-10-30T13:44:57.289 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/28' 2023-10-30T13:44:57.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/24' 2023-10-30T13:44:57.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/29' 2023-10-30T13:44:57.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/32' 2023-10-30T13:44:57.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/22' 2023-10-30T13:44:57.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/25' 2023-10-30T13:44:57.290 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/30' 2023-10-30T13:44:57.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/33' 2023-10-30T13:44:57.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/23' 2023-10-30T13:44:57.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/35' 2023-10-30T13:44:57.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/26' 2023-10-30T13:44:57.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './270/37' 2023-10-30T13:44:57.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './270' 2023-10-30T13:44:57.291 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/21' 2023-10-30T13:44:57.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/27' 2023-10-30T13:44:57.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/31' 2023-10-30T13:44:57.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/36' 2023-10-30T13:44:57.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/20' 2023-10-30T13:44:57.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/39' 2023-10-30T13:44:57.292 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/34' 2023-10-30T13:44:57.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/38' 2023-10-30T13:44:57.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/28' 2023-10-30T13:44:57.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/24' 2023-10-30T13:44:57.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/29' 2023-10-30T13:44:57.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/32' 2023-10-30T13:44:57.293 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/22' 2023-10-30T13:44:57.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/25' 2023-10-30T13:44:57.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/30' 2023-10-30T13:44:57.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/33' 2023-10-30T13:44:57.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/23' 2023-10-30T13:44:57.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/35' 2023-10-30T13:44:57.294 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/26' 2023-10-30T13:44:57.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './271/37' 2023-10-30T13:44:57.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './271' 2023-10-30T13:44:57.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/21' 2023-10-30T13:44:57.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/27' 2023-10-30T13:44:57.295 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/31' 2023-10-30T13:44:57.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/36' 2023-10-30T13:44:57.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/20' 2023-10-30T13:44:57.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/39' 2023-10-30T13:44:57.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/34' 2023-10-30T13:44:57.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/38' 2023-10-30T13:44:57.296 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/28' 2023-10-30T13:44:57.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/24' 2023-10-30T13:44:57.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/29' 2023-10-30T13:44:57.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/32' 2023-10-30T13:44:57.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/22' 2023-10-30T13:44:57.297 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/25' 2023-10-30T13:44:57.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/30' 2023-10-30T13:44:57.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/33' 2023-10-30T13:44:57.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/23' 2023-10-30T13:44:57.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/35' 2023-10-30T13:44:57.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/26' 2023-10-30T13:44:57.298 INFO:tasks.workunit.client.0.smithi031.stdout:removed './272/37' 2023-10-30T13:44:57.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './272' 2023-10-30T13:44:57.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/21' 2023-10-30T13:44:57.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/27' 2023-10-30T13:44:57.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/31' 2023-10-30T13:44:57.299 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/36' 2023-10-30T13:44:57.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/20' 2023-10-30T13:44:57.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/39' 2023-10-30T13:44:57.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/34' 2023-10-30T13:44:57.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/38' 2023-10-30T13:44:57.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/28' 2023-10-30T13:44:57.300 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/24' 2023-10-30T13:44:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/29' 2023-10-30T13:44:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/32' 2023-10-30T13:44:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/22' 2023-10-30T13:44:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/25' 2023-10-30T13:44:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/30' 2023-10-30T13:44:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/33' 2023-10-30T13:44:57.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/23' 2023-10-30T13:44:57.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/35' 2023-10-30T13:44:57.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/26' 2023-10-30T13:44:57.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed './273/37' 2023-10-30T13:44:57.302 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './273' 2023-10-30T13:44:57.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/21' 2023-10-30T13:44:57.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/27' 2023-10-30T13:44:57.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/31' 2023-10-30T13:44:57.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/36' 2023-10-30T13:44:57.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/20' 2023-10-30T13:44:57.303 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/39' 2023-10-30T13:44:57.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/34' 2023-10-30T13:44:57.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/38' 2023-10-30T13:44:57.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/28' 2023-10-30T13:44:57.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/24' 2023-10-30T13:44:57.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/29' 2023-10-30T13:44:57.304 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/32' 2023-10-30T13:44:57.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/22' 2023-10-30T13:44:57.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/25' 2023-10-30T13:44:57.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/30' 2023-10-30T13:44:57.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/33' 2023-10-30T13:44:57.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/23' 2023-10-30T13:44:57.305 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/35' 2023-10-30T13:44:57.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/26' 2023-10-30T13:44:57.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './274/37' 2023-10-30T13:44:57.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './274' 2023-10-30T13:44:57.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/21' 2023-10-30T13:44:57.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/27' 2023-10-30T13:44:57.306 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/31' 2023-10-30T13:44:57.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/36' 2023-10-30T13:44:57.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/20' 2023-10-30T13:44:57.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/39' 2023-10-30T13:44:57.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/34' 2023-10-30T13:44:57.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/38' 2023-10-30T13:44:57.307 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/28' 2023-10-30T13:44:57.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/24' 2023-10-30T13:44:57.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/29' 2023-10-30T13:44:57.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/32' 2023-10-30T13:44:57.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/22' 2023-10-30T13:44:57.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/25' 2023-10-30T13:44:57.308 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/30' 2023-10-30T13:44:57.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/33' 2023-10-30T13:44:57.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/23' 2023-10-30T13:44:57.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/35' 2023-10-30T13:44:57.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/26' 2023-10-30T13:44:57.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed './275/37' 2023-10-30T13:44:57.309 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './275' 2023-10-30T13:44:57.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/21' 2023-10-30T13:44:57.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/27' 2023-10-30T13:44:57.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/31' 2023-10-30T13:44:57.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/36' 2023-10-30T13:44:57.310 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/20' 2023-10-30T13:44:57.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/39' 2023-10-30T13:44:57.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/34' 2023-10-30T13:44:57.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/38' 2023-10-30T13:44:57.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/28' 2023-10-30T13:44:57.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/24' 2023-10-30T13:44:57.311 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/29' 2023-10-30T13:44:57.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/32' 2023-10-30T13:44:57.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/22' 2023-10-30T13:44:57.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/25' 2023-10-30T13:44:57.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/30' 2023-10-30T13:44:57.312 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/33' 2023-10-30T13:44:57.313 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/23' 2023-10-30T13:44:57.313 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/35' 2023-10-30T13:44:57.313 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/26' 2023-10-30T13:44:59.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed './276/37' 2023-10-30T13:44:59.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './276' 2023-10-30T13:44:59.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/21' 2023-10-30T13:44:59.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/27' 2023-10-30T13:44:59.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/31' 2023-10-30T13:44:59.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/36' 2023-10-30T13:44:59.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/20' 2023-10-30T13:44:59.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/39' 2023-10-30T13:44:59.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/34' 2023-10-30T13:44:59.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/38' 2023-10-30T13:44:59.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/28' 2023-10-30T13:44:59.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/24' 2023-10-30T13:44:59.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/29' 2023-10-30T13:44:59.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/32' 2023-10-30T13:44:59.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/22' 2023-10-30T13:44:59.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/25' 2023-10-30T13:44:59.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/30' 2023-10-30T13:44:59.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/33' 2023-10-30T13:44:59.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/23' 2023-10-30T13:44:59.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/35' 2023-10-30T13:44:59.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/26' 2023-10-30T13:44:59.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './277/37' 2023-10-30T13:44:59.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './277' 2023-10-30T13:44:59.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/21' 2023-10-30T13:44:59.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/27' 2023-10-30T13:44:59.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/31' 2023-10-30T13:44:59.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/36' 2023-10-30T13:44:59.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/20' 2023-10-30T13:44:59.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/39' 2023-10-30T13:44:59.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/34' 2023-10-30T13:44:59.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/38' 2023-10-30T13:44:59.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/28' 2023-10-30T13:44:59.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/24' 2023-10-30T13:44:59.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/29' 2023-10-30T13:44:59.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/32' 2023-10-30T13:44:59.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/22' 2023-10-30T13:44:59.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/25' 2023-10-30T13:44:59.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/30' 2023-10-30T13:44:59.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/33' 2023-10-30T13:44:59.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/23' 2023-10-30T13:44:59.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/35' 2023-10-30T13:44:59.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/26' 2023-10-30T13:44:59.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './278/37' 2023-10-30T13:44:59.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './278' 2023-10-30T13:44:59.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/21' 2023-10-30T13:44:59.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/27' 2023-10-30T13:44:59.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/31' 2023-10-30T13:44:59.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/36' 2023-10-30T13:44:59.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/20' 2023-10-30T13:44:59.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/39' 2023-10-30T13:44:59.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/34' 2023-10-30T13:44:59.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/38' 2023-10-30T13:44:59.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/28' 2023-10-30T13:44:59.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/24' 2023-10-30T13:44:59.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/29' 2023-10-30T13:44:59.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/32' 2023-10-30T13:44:59.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/22' 2023-10-30T13:44:59.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/25' 2023-10-30T13:44:59.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/30' 2023-10-30T13:44:59.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/33' 2023-10-30T13:44:59.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/23' 2023-10-30T13:44:59.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/35' 2023-10-30T13:44:59.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/26' 2023-10-30T13:44:59.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './279/37' 2023-10-30T13:44:59.402 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './279' 2023-10-30T13:44:59.402 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/21' 2023-10-30T13:44:59.402 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/27' 2023-10-30T13:44:59.402 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/31' 2023-10-30T13:44:59.402 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/36' 2023-10-30T13:44:59.402 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/20' 2023-10-30T13:44:59.403 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/39' 2023-10-30T13:44:59.403 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/34' 2023-10-30T13:44:59.403 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/38' 2023-10-30T13:44:59.403 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/28' 2023-10-30T13:44:59.403 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/24' 2023-10-30T13:44:59.403 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/29' 2023-10-30T13:44:59.404 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/32' 2023-10-30T13:44:59.404 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/22' 2023-10-30T13:44:59.404 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/25' 2023-10-30T13:44:59.404 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/30' 2023-10-30T13:44:59.404 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/33' 2023-10-30T13:44:59.405 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/23' 2023-10-30T13:44:59.405 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/35' 2023-10-30T13:44:59.405 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/26' 2023-10-30T13:44:59.405 INFO:tasks.workunit.client.0.smithi031.stdout:removed './280/37' 2023-10-30T13:44:59.405 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './280' 2023-10-30T13:44:59.406 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/21' 2023-10-30T13:44:59.406 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/27' 2023-10-30T13:44:59.406 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/31' 2023-10-30T13:44:59.406 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/36' 2023-10-30T13:44:59.406 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/20' 2023-10-30T13:44:59.406 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/39' 2023-10-30T13:44:59.407 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/34' 2023-10-30T13:44:59.407 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/38' 2023-10-30T13:44:59.407 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/28' 2023-10-30T13:44:59.407 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/24' 2023-10-30T13:44:59.407 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/29' 2023-10-30T13:44:59.408 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/32' 2023-10-30T13:44:59.408 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/22' 2023-10-30T13:44:59.408 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/25' 2023-10-30T13:44:59.408 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/30' 2023-10-30T13:44:59.408 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/33' 2023-10-30T13:44:59.408 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/23' 2023-10-30T13:44:59.409 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/35' 2023-10-30T13:44:59.409 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/26' 2023-10-30T13:44:59.409 INFO:tasks.workunit.client.0.smithi031.stdout:removed './281/37' 2023-10-30T13:44:59.409 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './281' 2023-10-30T13:44:59.409 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/21' 2023-10-30T13:44:59.410 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/27' 2023-10-30T13:44:59.410 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/31' 2023-10-30T13:44:59.410 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/36' 2023-10-30T13:44:59.410 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/20' 2023-10-30T13:44:59.410 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/39' 2023-10-30T13:44:59.411 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/34' 2023-10-30T13:44:59.411 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/38' 2023-10-30T13:44:59.411 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/28' 2023-10-30T13:44:59.411 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/24' 2023-10-30T13:44:59.411 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/29' 2023-10-30T13:44:59.411 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/32' 2023-10-30T13:44:59.412 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/22' 2023-10-30T13:44:59.412 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/25' 2023-10-30T13:44:59.412 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/30' 2023-10-30T13:44:59.412 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/33' 2023-10-30T13:44:59.412 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/23' 2023-10-30T13:44:59.412 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/35' 2023-10-30T13:44:59.413 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/26' 2023-10-30T13:44:59.413 INFO:tasks.workunit.client.0.smithi031.stdout:removed './282/37' 2023-10-30T13:44:59.413 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './282' 2023-10-30T13:44:59.413 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/21' 2023-10-30T13:44:59.413 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/27' 2023-10-30T13:44:59.413 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/31' 2023-10-30T13:44:59.414 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/36' 2023-10-30T13:44:59.414 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/20' 2023-10-30T13:44:59.414 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/39' 2023-10-30T13:44:59.414 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/34' 2023-10-30T13:44:59.414 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/38' 2023-10-30T13:44:59.415 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/28' 2023-10-30T13:44:59.415 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/24' 2023-10-30T13:44:59.415 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/29' 2023-10-30T13:44:59.415 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/32' 2023-10-30T13:44:59.415 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/22' 2023-10-30T13:44:59.415 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/25' 2023-10-30T13:44:59.416 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/30' 2023-10-30T13:44:59.416 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/33' 2023-10-30T13:44:59.416 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/23' 2023-10-30T13:44:59.416 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/35' 2023-10-30T13:44:59.416 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/26' 2023-10-30T13:44:59.416 INFO:tasks.workunit.client.0.smithi031.stdout:removed './283/37' 2023-10-30T13:44:59.417 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './283' 2023-10-30T13:44:59.417 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/21' 2023-10-30T13:44:59.417 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/27' 2023-10-30T13:44:59.417 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/31' 2023-10-30T13:44:59.417 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/36' 2023-10-30T13:44:59.417 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/20' 2023-10-30T13:44:59.418 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/39' 2023-10-30T13:44:59.418 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/34' 2023-10-30T13:44:59.418 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/38' 2023-10-30T13:44:59.418 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/28' 2023-10-30T13:44:59.418 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/24' 2023-10-30T13:44:59.418 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/29' 2023-10-30T13:44:59.419 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/32' 2023-10-30T13:44:59.419 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/22' 2023-10-30T13:44:59.419 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/25' 2023-10-30T13:44:59.419 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/30' 2023-10-30T13:44:59.419 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/33' 2023-10-30T13:44:59.420 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/23' 2023-10-30T13:44:59.420 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/35' 2023-10-30T13:44:59.420 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/26' 2023-10-30T13:44:59.420 INFO:tasks.workunit.client.0.smithi031.stdout:removed './284/37' 2023-10-30T13:44:59.420 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './284' 2023-10-30T13:44:59.420 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/21' 2023-10-30T13:44:59.420 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/27' 2023-10-30T13:44:59.421 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/31' 2023-10-30T13:44:59.421 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/36' 2023-10-30T13:44:59.421 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/20' 2023-10-30T13:44:59.421 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/39' 2023-10-30T13:44:59.421 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/34' 2023-10-30T13:44:59.421 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/38' 2023-10-30T13:44:59.422 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/28' 2023-10-30T13:44:59.422 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/24' 2023-10-30T13:44:59.422 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/29' 2023-10-30T13:44:59.422 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/32' 2023-10-30T13:44:59.422 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/22' 2023-10-30T13:44:59.422 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/25' 2023-10-30T13:44:59.423 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/30' 2023-10-30T13:44:59.423 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/33' 2023-10-30T13:44:59.423 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/23' 2023-10-30T13:44:59.423 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/35' 2023-10-30T13:44:59.423 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/26' 2023-10-30T13:44:59.423 INFO:tasks.workunit.client.0.smithi031.stdout:removed './285/37' 2023-10-30T13:44:59.424 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './285' 2023-10-30T13:44:59.424 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/21' 2023-10-30T13:44:59.424 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/27' 2023-10-30T13:44:59.424 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/31' 2023-10-30T13:44:59.424 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/36' 2023-10-30T13:44:59.424 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/20' 2023-10-30T13:44:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/39' 2023-10-30T13:44:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/34' 2023-10-30T13:44:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/38' 2023-10-30T13:44:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/28' 2023-10-30T13:44:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/24' 2023-10-30T13:44:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/29' 2023-10-30T13:44:59.426 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/32' 2023-10-30T13:44:59.426 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/22' 2023-10-30T13:44:59.426 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/25' 2023-10-30T13:44:59.426 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/30' 2023-10-30T13:44:59.426 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/33' 2023-10-30T13:44:59.426 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/23' 2023-10-30T13:44:59.427 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/35' 2023-10-30T13:44:59.427 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/26' 2023-10-30T13:44:59.427 INFO:tasks.workunit.client.0.smithi031.stdout:removed './286/37' 2023-10-30T13:44:59.427 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './286' 2023-10-30T13:45:01.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/21' 2023-10-30T13:45:01.208 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/27' 2023-10-30T13:45:01.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/31' 2023-10-30T13:45:01.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/36' 2023-10-30T13:45:01.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/20' 2023-10-30T13:45:01.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/39' 2023-10-30T13:45:01.209 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/34' 2023-10-30T13:45:01.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/38' 2023-10-30T13:45:01.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/28' 2023-10-30T13:45:01.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/24' 2023-10-30T13:45:01.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/29' 2023-10-30T13:45:01.210 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/32' 2023-10-30T13:45:01.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/22' 2023-10-30T13:45:01.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/25' 2023-10-30T13:45:01.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/30' 2023-10-30T13:45:01.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/33' 2023-10-30T13:45:01.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/23' 2023-10-30T13:45:01.211 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/35' 2023-10-30T13:45:01.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/26' 2023-10-30T13:45:01.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './287/37' 2023-10-30T13:45:01.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './287' 2023-10-30T13:45:01.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/21' 2023-10-30T13:45:01.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/27' 2023-10-30T13:45:01.212 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/31' 2023-10-30T13:45:01.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/36' 2023-10-30T13:45:01.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/20' 2023-10-30T13:45:01.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/39' 2023-10-30T13:45:01.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/34' 2023-10-30T13:45:01.213 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/38' 2023-10-30T13:45:01.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/28' 2023-10-30T13:45:01.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/24' 2023-10-30T13:45:01.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/29' 2023-10-30T13:45:01.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/32' 2023-10-30T13:45:01.214 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/22' 2023-10-30T13:45:01.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/25' 2023-10-30T13:45:01.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/30' 2023-10-30T13:45:01.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/33' 2023-10-30T13:45:01.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/23' 2023-10-30T13:45:01.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/35' 2023-10-30T13:45:01.215 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/26' 2023-10-30T13:45:01.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './288/37' 2023-10-30T13:45:01.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './288' 2023-10-30T13:45:01.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/21' 2023-10-30T13:45:01.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/27' 2023-10-30T13:45:01.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/31' 2023-10-30T13:45:01.216 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/36' 2023-10-30T13:45:01.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/20' 2023-10-30T13:45:01.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/39' 2023-10-30T13:45:01.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/34' 2023-10-30T13:45:01.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/38' 2023-10-30T13:45:01.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/28' 2023-10-30T13:45:01.217 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/24' 2023-10-30T13:45:01.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/29' 2023-10-30T13:45:01.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/32' 2023-10-30T13:45:01.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/22' 2023-10-30T13:45:01.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/25' 2023-10-30T13:45:01.218 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/30' 2023-10-30T13:45:01.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/33' 2023-10-30T13:45:01.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/23' 2023-10-30T13:45:01.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/35' 2023-10-30T13:45:01.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/26' 2023-10-30T13:45:01.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed './289/37' 2023-10-30T13:45:01.219 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './289' 2023-10-30T13:45:01.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/21' 2023-10-30T13:45:01.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/27' 2023-10-30T13:45:01.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/31' 2023-10-30T13:45:01.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/36' 2023-10-30T13:45:01.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/20' 2023-10-30T13:45:01.220 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/39' 2023-10-30T13:45:01.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/34' 2023-10-30T13:45:01.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/38' 2023-10-30T13:45:01.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/28' 2023-10-30T13:45:01.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/24' 2023-10-30T13:45:01.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/29' 2023-10-30T13:45:01.221 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/32' 2023-10-30T13:45:01.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/22' 2023-10-30T13:45:01.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/25' 2023-10-30T13:45:01.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/30' 2023-10-30T13:45:01.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/33' 2023-10-30T13:45:01.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/23' 2023-10-30T13:45:01.222 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/35' 2023-10-30T13:45:01.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/26' 2023-10-30T13:45:01.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './290/37' 2023-10-30T13:45:01.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './290' 2023-10-30T13:45:01.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/21' 2023-10-30T13:45:01.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/27' 2023-10-30T13:45:01.223 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/31' 2023-10-30T13:45:01.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/36' 2023-10-30T13:45:01.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/20' 2023-10-30T13:45:01.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/39' 2023-10-30T13:45:01.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/34' 2023-10-30T13:45:01.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/38' 2023-10-30T13:45:01.224 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/28' 2023-10-30T13:45:01.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/24' 2023-10-30T13:45:01.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/29' 2023-10-30T13:45:01.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/32' 2023-10-30T13:45:01.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/22' 2023-10-30T13:45:01.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/25' 2023-10-30T13:45:01.225 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/30' 2023-10-30T13:45:01.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/33' 2023-10-30T13:45:01.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/23' 2023-10-30T13:45:01.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/35' 2023-10-30T13:45:01.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/26' 2023-10-30T13:45:01.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed './291/37' 2023-10-30T13:45:01.226 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './291' 2023-10-30T13:45:01.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/21' 2023-10-30T13:45:01.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/27' 2023-10-30T13:45:01.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/31' 2023-10-30T13:45:01.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/36' 2023-10-30T13:45:01.227 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/20' 2023-10-30T13:45:01.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/39' 2023-10-30T13:45:01.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/34' 2023-10-30T13:45:01.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/38' 2023-10-30T13:45:01.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/28' 2023-10-30T13:45:01.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/24' 2023-10-30T13:45:01.228 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/29' 2023-10-30T13:45:01.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/32' 2023-10-30T13:45:01.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/22' 2023-10-30T13:45:01.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/25' 2023-10-30T13:45:01.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/30' 2023-10-30T13:45:01.229 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/33' 2023-10-30T13:45:01.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/23' 2023-10-30T13:45:01.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/35' 2023-10-30T13:45:01.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/26' 2023-10-30T13:45:01.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './292/37' 2023-10-30T13:45:01.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './292' 2023-10-30T13:45:01.230 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/21' 2023-10-30T13:45:01.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/27' 2023-10-30T13:45:01.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/31' 2023-10-30T13:45:01.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/36' 2023-10-30T13:45:01.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/20' 2023-10-30T13:45:01.231 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/39' 2023-10-30T13:45:01.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/34' 2023-10-30T13:45:01.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/38' 2023-10-30T13:45:01.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/28' 2023-10-30T13:45:01.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/24' 2023-10-30T13:45:01.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/29' 2023-10-30T13:45:01.232 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/32' 2023-10-30T13:45:01.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/22' 2023-10-30T13:45:01.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/25' 2023-10-30T13:45:01.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/30' 2023-10-30T13:45:01.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/33' 2023-10-30T13:45:01.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/23' 2023-10-30T13:45:01.233 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/35' 2023-10-30T13:45:01.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/26' 2023-10-30T13:45:01.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './293/37' 2023-10-30T13:45:01.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './293' 2023-10-30T13:45:01.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/21' 2023-10-30T13:45:01.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/27' 2023-10-30T13:45:01.234 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/31' 2023-10-30T13:45:01.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/36' 2023-10-30T13:45:01.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/20' 2023-10-30T13:45:01.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/39' 2023-10-30T13:45:01.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/34' 2023-10-30T13:45:01.235 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/38' 2023-10-30T13:45:01.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/28' 2023-10-30T13:45:01.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/24' 2023-10-30T13:45:01.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/29' 2023-10-30T13:45:01.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/32' 2023-10-30T13:45:01.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/22' 2023-10-30T13:45:01.236 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/25' 2023-10-30T13:45:01.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/30' 2023-10-30T13:45:01.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/33' 2023-10-30T13:45:01.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/23' 2023-10-30T13:45:01.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/35' 2023-10-30T13:45:01.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/26' 2023-10-30T13:45:01.237 INFO:tasks.workunit.client.0.smithi031.stdout:removed './294/37' 2023-10-30T13:45:01.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './294' 2023-10-30T13:45:01.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/21' 2023-10-30T13:45:01.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/27' 2023-10-30T13:45:01.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/31' 2023-10-30T13:45:01.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/36' 2023-10-30T13:45:01.238 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/20' 2023-10-30T13:45:01.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/39' 2023-10-30T13:45:01.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/34' 2023-10-30T13:45:01.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/38' 2023-10-30T13:45:01.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/28' 2023-10-30T13:45:01.239 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/24' 2023-10-30T13:45:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/29' 2023-10-30T13:45:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/32' 2023-10-30T13:45:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/22' 2023-10-30T13:45:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/25' 2023-10-30T13:45:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/30' 2023-10-30T13:45:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/33' 2023-10-30T13:45:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/23' 2023-10-30T13:45:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/35' 2023-10-30T13:45:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/26' 2023-10-30T13:45:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './295/37' 2023-10-30T13:45:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './295' 2023-10-30T13:45:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/21' 2023-10-30T13:45:01.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/27' 2023-10-30T13:45:01.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/31' 2023-10-30T13:45:01.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/36' 2023-10-30T13:45:01.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/20' 2023-10-30T13:45:01.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/39' 2023-10-30T13:45:01.242 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/34' 2023-10-30T13:45:01.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/38' 2023-10-30T13:45:01.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/28' 2023-10-30T13:45:01.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/24' 2023-10-30T13:45:01.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/29' 2023-10-30T13:45:01.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/32' 2023-10-30T13:45:01.243 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/22' 2023-10-30T13:45:01.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/25' 2023-10-30T13:45:01.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/30' 2023-10-30T13:45:01.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/33' 2023-10-30T13:45:01.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/23' 2023-10-30T13:45:01.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/35' 2023-10-30T13:45:01.244 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/26' 2023-10-30T13:45:01.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './296/37' 2023-10-30T13:45:01.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './296' 2023-10-30T13:45:01.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/21' 2023-10-30T13:45:01.245 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/27' 2023-10-30T13:45:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/31' 2023-10-30T13:45:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/36' 2023-10-30T13:45:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/20' 2023-10-30T13:45:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/39' 2023-10-30T13:45:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/34' 2023-10-30T13:45:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/38' 2023-10-30T13:45:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/28' 2023-10-30T13:45:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/24' 2023-10-30T13:45:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/29' 2023-10-30T13:45:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/32' 2023-10-30T13:45:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/22' 2023-10-30T13:45:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/25' 2023-10-30T13:45:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/30' 2023-10-30T13:45:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/33' 2023-10-30T13:45:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/23' 2023-10-30T13:45:01.827 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/35' 2023-10-30T13:45:01.827 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/26' 2023-10-30T13:45:01.827 INFO:tasks.workunit.client.0.smithi031.stdout:removed './297/37' 2023-10-30T13:45:01.827 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './297' 2023-10-30T13:45:01.827 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/21' 2023-10-30T13:45:01.827 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/27' 2023-10-30T13:45:01.828 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/31' 2023-10-30T13:45:01.828 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/36' 2023-10-30T13:45:01.828 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/20' 2023-10-30T13:45:01.828 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/39' 2023-10-30T13:45:01.828 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/34' 2023-10-30T13:45:01.829 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/38' 2023-10-30T13:45:01.829 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/28' 2023-10-30T13:45:01.829 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/24' 2023-10-30T13:45:01.829 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/29' 2023-10-30T13:45:01.829 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/32' 2023-10-30T13:45:01.830 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/22' 2023-10-30T13:45:01.830 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/25' 2023-10-30T13:45:01.830 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/30' 2023-10-30T13:45:01.830 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/33' 2023-10-30T13:45:01.830 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/23' 2023-10-30T13:45:01.831 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/35' 2023-10-30T13:45:01.831 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/26' 2023-10-30T13:45:01.831 INFO:tasks.workunit.client.0.smithi031.stdout:removed './298/37' 2023-10-30T13:45:01.831 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './298' 2023-10-30T13:45:01.831 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/21' 2023-10-30T13:45:01.831 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/27' 2023-10-30T13:45:01.832 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/31' 2023-10-30T13:45:01.832 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/36' 2023-10-30T13:45:01.832 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/20' 2023-10-30T13:45:01.832 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/39' 2023-10-30T13:45:01.832 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/34' 2023-10-30T13:45:01.832 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/38' 2023-10-30T13:45:01.833 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/28' 2023-10-30T13:45:01.833 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/24' 2023-10-30T13:45:01.833 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/29' 2023-10-30T13:45:01.833 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/32' 2023-10-30T13:45:01.833 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/22' 2023-10-30T13:45:01.834 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/25' 2023-10-30T13:45:01.834 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/30' 2023-10-30T13:45:01.834 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/33' 2023-10-30T13:45:01.834 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/23' 2023-10-30T13:45:01.834 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/35' 2023-10-30T13:45:01.835 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/26' 2023-10-30T13:45:01.835 INFO:tasks.workunit.client.0.smithi031.stdout:removed './299/37' 2023-10-30T13:45:01.835 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './299' 2023-10-30T13:45:01.835 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/19' 2023-10-30T13:45:01.835 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/15' 2023-10-30T13:45:01.836 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/20' 2023-10-30T13:45:01.836 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/12' 2023-10-30T13:45:01.836 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/10' 2023-10-30T13:45:01.836 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/11' 2023-10-30T13:45:01.836 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/16' 2023-10-30T13:45:01.837 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/14' 2023-10-30T13:45:01.837 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/18' 2023-10-30T13:45:01.837 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/13' 2023-10-30T13:45:01.837 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/166/17' 2023-10-30T13:45:01.837 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/166' 2023-10-30T13:45:01.838 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/19' 2023-10-30T13:45:01.838 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/15' 2023-10-30T13:45:01.838 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/20' 2023-10-30T13:45:01.838 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/12' 2023-10-30T13:45:01.838 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/10' 2023-10-30T13:45:01.839 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/11' 2023-10-30T13:45:01.839 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/16' 2023-10-30T13:45:01.839 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/14' 2023-10-30T13:45:01.839 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/18' 2023-10-30T13:45:01.839 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/13' 2023-10-30T13:45:01.839 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/147/17' 2023-10-30T13:45:01.840 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/147' 2023-10-30T13:45:01.840 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/19' 2023-10-30T13:45:01.840 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/15' 2023-10-30T13:45:01.840 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/20' 2023-10-30T13:45:01.840 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/12' 2023-10-30T13:45:01.840 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/10' 2023-10-30T13:45:01.841 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/11' 2023-10-30T13:45:01.841 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/16' 2023-10-30T13:45:01.841 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/14' 2023-10-30T13:45:01.841 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/18' 2023-10-30T13:45:01.841 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/13' 2023-10-30T13:45:01.841 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/151/17' 2023-10-30T13:45:01.842 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/151' 2023-10-30T13:45:01.842 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/19' 2023-10-30T13:45:01.842 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/15' 2023-10-30T13:45:01.842 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/20' 2023-10-30T13:45:01.842 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/12' 2023-10-30T13:45:01.843 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/10' 2023-10-30T13:45:01.843 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/11' 2023-10-30T13:45:01.843 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/16' 2023-10-30T13:45:01.843 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/14' 2023-10-30T13:45:01.843 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/18' 2023-10-30T13:45:01.844 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/13' 2023-10-30T13:45:01.844 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/170/17' 2023-10-30T13:45:01.844 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/170' 2023-10-30T13:45:01.844 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/19' 2023-10-30T13:45:01.844 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/15' 2023-10-30T13:45:01.844 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/20' 2023-10-30T13:45:01.845 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/12' 2023-10-30T13:45:01.845 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/10' 2023-10-30T13:45:01.845 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/11' 2023-10-30T13:45:01.845 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/16' 2023-10-30T13:45:01.845 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/14' 2023-10-30T13:45:01.845 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/18' 2023-10-30T13:45:01.846 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/13' 2023-10-30T13:45:01.846 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/167/17' 2023-10-30T13:45:01.846 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/167' 2023-10-30T13:45:01.846 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/19' 2023-10-30T13:45:01.846 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/15' 2023-10-30T13:45:01.847 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/20' 2023-10-30T13:45:01.847 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/12' 2023-10-30T13:45:01.847 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/10' 2023-10-30T13:45:01.847 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/11' 2023-10-30T13:45:01.847 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/16' 2023-10-30T13:45:01.847 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/14' 2023-10-30T13:45:01.848 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/18' 2023-10-30T13:45:01.848 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/13' 2023-10-30T13:45:01.848 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/140/17' 2023-10-30T13:45:01.848 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/140' 2023-10-30T13:45:01.848 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/19' 2023-10-30T13:45:01.849 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/15' 2023-10-30T13:45:01.849 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/20' 2023-10-30T13:45:01.849 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/12' 2023-10-30T13:45:01.849 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/10' 2023-10-30T13:45:05.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/11' 2023-10-30T13:45:05.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/16' 2023-10-30T13:45:05.183 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/14' 2023-10-30T13:45:05.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/18' 2023-10-30T13:45:05.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/13' 2023-10-30T13:45:05.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/142/17' 2023-10-30T13:45:05.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/142' 2023-10-30T13:45:05.184 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/19' 2023-10-30T13:45:05.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/15' 2023-10-30T13:45:05.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/20' 2023-10-30T13:45:05.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/12' 2023-10-30T13:45:05.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/10' 2023-10-30T13:45:05.185 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/11' 2023-10-30T13:45:05.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/16' 2023-10-30T13:45:05.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/14' 2023-10-30T13:45:05.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/18' 2023-10-30T13:45:05.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/13' 2023-10-30T13:45:05.186 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/150/17' 2023-10-30T13:45:05.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/150' 2023-10-30T13:45:05.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/19' 2023-10-30T13:45:05.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/15' 2023-10-30T13:45:05.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/20' 2023-10-30T13:45:05.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/12' 2023-10-30T13:45:05.187 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/10' 2023-10-30T13:45:05.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/11' 2023-10-30T13:45:05.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/16' 2023-10-30T13:45:05.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/14' 2023-10-30T13:45:05.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/18' 2023-10-30T13:45:05.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/13' 2023-10-30T13:45:05.188 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/105/17' 2023-10-30T13:45:05.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/105' 2023-10-30T13:45:05.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/19' 2023-10-30T13:45:05.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/15' 2023-10-30T13:45:05.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/20' 2023-10-30T13:45:05.189 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/12' 2023-10-30T13:45:05.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/10' 2023-10-30T13:45:05.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/11' 2023-10-30T13:45:05.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/16' 2023-10-30T13:45:05.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/14' 2023-10-30T13:45:05.190 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/18' 2023-10-30T13:45:05.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/13' 2023-10-30T13:45:05.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/149/17' 2023-10-30T13:45:05.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/149' 2023-10-30T13:45:05.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/19' 2023-10-30T13:45:05.191 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/15' 2023-10-30T13:45:05.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/20' 2023-10-30T13:45:05.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/12' 2023-10-30T13:45:05.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/10' 2023-10-30T13:45:05.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/11' 2023-10-30T13:45:05.192 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/16' 2023-10-30T13:45:05.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/14' 2023-10-30T13:45:05.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/18' 2023-10-30T13:45:05.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/13' 2023-10-30T13:45:05.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/109/17' 2023-10-30T13:45:05.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/109' 2023-10-30T13:45:05.193 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/19' 2023-10-30T13:45:05.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/15' 2023-10-30T13:45:05.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/20' 2023-10-30T13:45:05.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/12' 2023-10-30T13:45:05.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/10' 2023-10-30T13:45:05.194 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/11' 2023-10-30T13:45:05.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/16' 2023-10-30T13:45:05.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/14' 2023-10-30T13:45:05.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/18' 2023-10-30T13:45:05.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/13' 2023-10-30T13:45:05.195 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/158/17' 2023-10-30T13:45:05.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/158' 2023-10-30T13:45:05.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/19' 2023-10-30T13:45:05.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/15' 2023-10-30T13:45:05.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/20' 2023-10-30T13:45:05.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/12' 2023-10-30T13:45:05.196 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/10' 2023-10-30T13:45:05.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/11' 2023-10-30T13:45:05.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/16' 2023-10-30T13:45:05.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/14' 2023-10-30T13:45:05.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/18' 2023-10-30T13:45:05.197 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/13' 2023-10-30T13:45:05.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/106/17' 2023-10-30T13:45:05.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/106' 2023-10-30T13:45:05.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/19' 2023-10-30T13:45:05.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/15' 2023-10-30T13:45:05.198 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/20' 2023-10-30T13:45:05.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/12' 2023-10-30T13:45:05.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/10' 2023-10-30T13:45:05.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/11' 2023-10-30T13:45:05.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/16' 2023-10-30T13:45:05.199 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/14' 2023-10-30T13:45:05.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/18' 2023-10-30T13:45:05.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/13' 2023-10-30T13:45:05.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/189/17' 2023-10-30T13:45:05.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/189' 2023-10-30T13:45:05.200 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/19' 2023-10-30T13:45:05.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/15' 2023-10-30T13:45:05.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/20' 2023-10-30T13:45:05.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/12' 2023-10-30T13:45:05.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/10' 2023-10-30T13:45:05.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/11' 2023-10-30T13:45:05.201 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/16' 2023-10-30T13:45:05.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/14' 2023-10-30T13:45:05.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/18' 2023-10-30T13:45:05.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/13' 2023-10-30T13:45:05.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/133/17' 2023-10-30T13:45:05.202 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/133' 2023-10-30T13:45:05.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/19' 2023-10-30T13:45:05.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/15' 2023-10-30T13:45:05.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/20' 2023-10-30T13:45:05.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/12' 2023-10-30T13:45:05.203 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/10' 2023-10-30T13:45:05.204 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/11' 2023-10-30T13:45:06.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/16' 2023-10-30T13:45:06.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/14' 2023-10-30T13:45:06.511 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/18' 2023-10-30T13:45:06.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/13' 2023-10-30T13:45:06.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/173/17' 2023-10-30T13:45:06.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/173' 2023-10-30T13:45:06.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/19' 2023-10-30T13:45:06.512 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/15' 2023-10-30T13:45:06.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/20' 2023-10-30T13:45:06.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/12' 2023-10-30T13:45:06.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/10' 2023-10-30T13:45:06.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/11' 2023-10-30T13:45:06.513 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/16' 2023-10-30T13:45:06.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/14' 2023-10-30T13:45:06.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/18' 2023-10-30T13:45:06.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/13' 2023-10-30T13:45:06.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/174/17' 2023-10-30T13:45:06.514 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/174' 2023-10-30T13:45:06.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/19' 2023-10-30T13:45:06.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/15' 2023-10-30T13:45:06.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/20' 2023-10-30T13:45:06.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/12' 2023-10-30T13:45:06.515 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/10' 2023-10-30T13:45:06.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/11' 2023-10-30T13:45:06.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/16' 2023-10-30T13:45:06.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/14' 2023-10-30T13:45:06.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/18' 2023-10-30T13:45:06.516 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/13' 2023-10-30T13:45:06.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/155/17' 2023-10-30T13:45:06.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/155' 2023-10-30T13:45:06.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/19' 2023-10-30T13:45:06.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/15' 2023-10-30T13:45:06.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/20' 2023-10-30T13:45:06.517 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/12' 2023-10-30T13:45:06.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/10' 2023-10-30T13:45:06.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/11' 2023-10-30T13:45:06.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/16' 2023-10-30T13:45:06.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/14' 2023-10-30T13:45:06.518 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/18' 2023-10-30T13:45:06.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/13' 2023-10-30T13:45:06.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/114/17' 2023-10-30T13:45:06.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/114' 2023-10-30T13:45:06.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/19' 2023-10-30T13:45:06.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/15' 2023-10-30T13:45:06.519 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/20' 2023-10-30T13:45:06.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/12' 2023-10-30T13:45:06.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/10' 2023-10-30T13:45:06.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/11' 2023-10-30T13:45:06.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/16' 2023-10-30T13:45:06.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/14' 2023-10-30T13:45:06.520 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/18' 2023-10-30T13:45:06.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/13' 2023-10-30T13:45:06.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/159/17' 2023-10-30T13:45:06.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/159' 2023-10-30T13:45:06.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/19' 2023-10-30T13:45:06.521 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/15' 2023-10-30T13:45:06.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/20' 2023-10-30T13:45:06.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/12' 2023-10-30T13:45:06.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/10' 2023-10-30T13:45:06.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/11' 2023-10-30T13:45:06.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/16' 2023-10-30T13:45:06.522 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/14' 2023-10-30T13:45:06.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/18' 2023-10-30T13:45:06.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/13' 2023-10-30T13:45:06.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/187/17' 2023-10-30T13:45:06.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/187' 2023-10-30T13:45:06.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/19' 2023-10-30T13:45:06.523 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/15' 2023-10-30T13:45:06.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/20' 2023-10-30T13:45:06.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/12' 2023-10-30T13:45:06.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/10' 2023-10-30T13:45:06.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/11' 2023-10-30T13:45:06.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/16' 2023-10-30T13:45:06.524 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/14' 2023-10-30T13:45:06.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/18' 2023-10-30T13:45:06.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/13' 2023-10-30T13:45:06.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/185/17' 2023-10-30T13:45:06.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/185' 2023-10-30T13:45:06.525 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/19' 2023-10-30T13:45:06.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/15' 2023-10-30T13:45:06.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/20' 2023-10-30T13:45:06.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/12' 2023-10-30T13:45:06.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/10' 2023-10-30T13:45:06.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/11' 2023-10-30T13:45:06.526 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/16' 2023-10-30T13:45:06.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/14' 2023-10-30T13:45:06.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/18' 2023-10-30T13:45:06.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/13' 2023-10-30T13:45:06.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/120/17' 2023-10-30T13:45:06.527 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/120' 2023-10-30T13:45:06.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/19' 2023-10-30T13:45:06.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/15' 2023-10-30T13:45:06.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/20' 2023-10-30T13:45:06.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/12' 2023-10-30T13:45:06.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/10' 2023-10-30T13:45:06.528 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/11' 2023-10-30T13:45:06.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/16' 2023-10-30T13:45:06.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/14' 2023-10-30T13:45:06.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/18' 2023-10-30T13:45:06.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/13' 2023-10-30T13:45:06.529 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/110/17' 2023-10-30T13:45:06.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/110' 2023-10-30T13:45:06.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/19' 2023-10-30T13:45:06.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/15' 2023-10-30T13:45:06.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/20' 2023-10-30T13:45:06.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/12' 2023-10-30T13:45:06.530 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/10' 2023-10-30T13:45:06.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/11' 2023-10-30T13:45:06.531 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/16' 2023-10-30T13:45:07.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/14' 2023-10-30T13:45:07.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/18' 2023-10-30T13:45:07.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/13' 2023-10-30T13:45:07.679 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/137/17' 2023-10-30T13:45:07.680 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/137' 2023-10-30T13:45:07.680 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/19' 2023-10-30T13:45:07.680 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/15' 2023-10-30T13:45:07.680 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/20' 2023-10-30T13:45:07.680 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/12' 2023-10-30T13:45:07.680 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/10' 2023-10-30T13:45:07.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/11' 2023-10-30T13:45:07.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/16' 2023-10-30T13:45:07.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/14' 2023-10-30T13:45:07.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/18' 2023-10-30T13:45:07.681 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/13' 2023-10-30T13:45:07.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/198/17' 2023-10-30T13:45:07.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/198' 2023-10-30T13:45:07.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/19' 2023-10-30T13:45:07.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/15' 2023-10-30T13:45:07.682 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/20' 2023-10-30T13:45:07.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/12' 2023-10-30T13:45:07.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/10' 2023-10-30T13:45:07.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/11' 2023-10-30T13:45:07.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/16' 2023-10-30T13:45:07.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/14' 2023-10-30T13:45:07.683 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/18' 2023-10-30T13:45:07.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/13' 2023-10-30T13:45:07.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/136/17' 2023-10-30T13:45:07.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/136' 2023-10-30T13:45:07.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/19' 2023-10-30T13:45:07.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/15' 2023-10-30T13:45:07.684 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/20' 2023-10-30T13:45:07.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/12' 2023-10-30T13:45:07.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/10' 2023-10-30T13:45:07.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/11' 2023-10-30T13:45:07.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/16' 2023-10-30T13:45:07.685 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/14' 2023-10-30T13:45:07.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/18' 2023-10-30T13:45:07.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/13' 2023-10-30T13:45:07.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/125/17' 2023-10-30T13:45:07.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/125' 2023-10-30T13:45:07.686 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/19' 2023-10-30T13:45:07.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/15' 2023-10-30T13:45:07.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/20' 2023-10-30T13:45:07.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/12' 2023-10-30T13:45:07.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/10' 2023-10-30T13:45:07.687 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/11' 2023-10-30T13:45:07.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/16' 2023-10-30T13:45:07.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/14' 2023-10-30T13:45:07.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/18' 2023-10-30T13:45:07.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/13' 2023-10-30T13:45:07.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/194/17' 2023-10-30T13:45:07.688 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/194' 2023-10-30T13:45:07.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/19' 2023-10-30T13:45:07.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/15' 2023-10-30T13:45:07.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/20' 2023-10-30T13:45:07.689 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/12' 2023-10-30T13:45:07.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/10' 2023-10-30T13:45:07.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/11' 2023-10-30T13:45:07.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/16' 2023-10-30T13:45:07.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/14' 2023-10-30T13:45:07.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/18' 2023-10-30T13:45:07.690 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/13' 2023-10-30T13:45:07.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/119/17' 2023-10-30T13:45:07.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/119' 2023-10-30T13:45:07.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/19' 2023-10-30T13:45:07.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/15' 2023-10-30T13:45:07.691 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/20' 2023-10-30T13:45:07.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/12' 2023-10-30T13:45:07.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/10' 2023-10-30T13:45:07.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/11' 2023-10-30T13:45:07.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/16' 2023-10-30T13:45:07.692 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/14' 2023-10-30T13:45:07.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/18' 2023-10-30T13:45:07.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/13' 2023-10-30T13:45:07.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/157/17' 2023-10-30T13:45:07.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/157' 2023-10-30T13:45:07.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/19' 2023-10-30T13:45:07.693 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/15' 2023-10-30T13:45:07.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/20' 2023-10-30T13:45:07.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/12' 2023-10-30T13:45:07.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/10' 2023-10-30T13:45:07.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/11' 2023-10-30T13:45:07.694 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/16' 2023-10-30T13:45:07.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/14' 2023-10-30T13:45:07.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/18' 2023-10-30T13:45:07.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/13' 2023-10-30T13:45:07.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/172/17' 2023-10-30T13:45:07.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/172' 2023-10-30T13:45:07.695 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/19' 2023-10-30T13:45:07.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/15' 2023-10-30T13:45:07.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/20' 2023-10-30T13:45:07.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/12' 2023-10-30T13:45:07.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/10' 2023-10-30T13:45:07.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/11' 2023-10-30T13:45:07.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/16' 2023-10-30T13:45:07.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/14' 2023-10-30T13:45:07.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/18' 2023-10-30T13:45:07.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/13' 2023-10-30T13:45:07.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/186/17' 2023-10-30T13:45:07.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/186' 2023-10-30T13:45:07.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/19' 2023-10-30T13:45:07.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/15' 2023-10-30T13:45:07.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/20' 2023-10-30T13:45:07.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/12' 2023-10-30T13:45:07.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/10' 2023-10-30T13:45:07.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/11' 2023-10-30T13:45:07.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/16' 2023-10-30T13:45:07.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/14' 2023-10-30T13:45:07.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/18' 2023-10-30T13:45:07.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/13' 2023-10-30T13:45:07.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/190/17' 2023-10-30T13:45:07.966 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/190' 2023-10-30T13:45:07.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/19' 2023-10-30T13:45:07.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/15' 2023-10-30T13:45:07.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/20' 2023-10-30T13:45:07.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/12' 2023-10-30T13:45:07.967 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/10' 2023-10-30T13:45:07.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/11' 2023-10-30T13:45:07.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/16' 2023-10-30T13:45:07.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/14' 2023-10-30T13:45:07.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/18' 2023-10-30T13:45:07.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/13' 2023-10-30T13:45:07.968 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/104/17' 2023-10-30T13:45:07.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/104' 2023-10-30T13:45:07.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/19' 2023-10-30T13:45:07.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/15' 2023-10-30T13:45:07.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/20' 2023-10-30T13:45:07.969 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/12' 2023-10-30T13:45:07.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/10' 2023-10-30T13:45:07.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/11' 2023-10-30T13:45:07.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/16' 2023-10-30T13:45:07.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/14' 2023-10-30T13:45:07.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/18' 2023-10-30T13:45:07.970 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/13' 2023-10-30T13:45:07.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/164/17' 2023-10-30T13:45:07.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/164' 2023-10-30T13:45:07.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/19' 2023-10-30T13:45:07.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/15' 2023-10-30T13:45:07.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/20' 2023-10-30T13:45:07.971 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/12' 2023-10-30T13:45:07.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/10' 2023-10-30T13:45:07.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/11' 2023-10-30T13:45:07.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/16' 2023-10-30T13:45:07.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/14' 2023-10-30T13:45:07.972 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/18' 2023-10-30T13:45:07.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/13' 2023-10-30T13:45:07.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/112/17' 2023-10-30T13:45:07.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/112' 2023-10-30T13:45:07.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/19' 2023-10-30T13:45:07.973 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/15' 2023-10-30T13:45:07.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/20' 2023-10-30T13:45:07.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/12' 2023-10-30T13:45:07.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/10' 2023-10-30T13:45:07.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/11' 2023-10-30T13:45:07.974 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/16' 2023-10-30T13:45:07.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/14' 2023-10-30T13:45:07.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/18' 2023-10-30T13:45:07.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/13' 2023-10-30T13:45:07.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/100/17' 2023-10-30T13:45:07.975 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/100' 2023-10-30T13:45:07.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/19' 2023-10-30T13:45:07.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/15' 2023-10-30T13:45:07.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/20' 2023-10-30T13:45:07.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/12' 2023-10-30T13:45:07.976 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/10' 2023-10-30T13:45:07.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/11' 2023-10-30T13:45:07.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/16' 2023-10-30T13:45:07.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/14' 2023-10-30T13:45:07.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/18' 2023-10-30T13:45:07.977 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/13' 2023-10-30T13:45:07.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/134/17' 2023-10-30T13:45:07.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/134' 2023-10-30T13:45:07.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/19' 2023-10-30T13:45:07.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/15' 2023-10-30T13:45:07.978 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/20' 2023-10-30T13:45:07.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/12' 2023-10-30T13:45:07.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/10' 2023-10-30T13:45:07.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/11' 2023-10-30T13:45:07.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/16' 2023-10-30T13:45:07.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/14' 2023-10-30T13:45:07.979 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/18' 2023-10-30T13:45:07.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/13' 2023-10-30T13:45:07.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/115/17' 2023-10-30T13:45:07.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/115' 2023-10-30T13:45:07.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/19' 2023-10-30T13:45:07.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/15' 2023-10-30T13:45:07.980 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/20' 2023-10-30T13:45:07.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/12' 2023-10-30T13:45:07.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/10' 2023-10-30T13:45:07.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/11' 2023-10-30T13:45:07.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/16' 2023-10-30T13:45:07.981 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/14' 2023-10-30T13:45:07.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/18' 2023-10-30T13:45:07.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/13' 2023-10-30T13:45:07.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/116/17' 2023-10-30T13:45:07.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/116' 2023-10-30T13:45:07.982 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/19' 2023-10-30T13:45:07.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/15' 2023-10-30T13:45:07.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/20' 2023-10-30T13:45:07.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/12' 2023-10-30T13:45:07.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/10' 2023-10-30T13:45:07.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/11' 2023-10-30T13:45:07.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/16' 2023-10-30T13:45:07.983 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/14' 2023-10-30T13:45:07.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/18' 2023-10-30T13:45:07.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/13' 2023-10-30T13:45:07.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/195/17' 2023-10-30T13:45:07.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/195' 2023-10-30T13:45:07.984 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/19' 2023-10-30T13:45:07.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/15' 2023-10-30T13:45:07.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/20' 2023-10-30T13:45:07.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/12' 2023-10-30T13:45:07.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/10' 2023-10-30T13:45:07.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/11' 2023-10-30T13:45:07.985 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/16' 2023-10-30T13:45:07.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/14' 2023-10-30T13:45:07.986 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/18' 2023-10-30T13:45:08.856 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/13' 2023-10-30T13:45:08.856 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/138/17' 2023-10-30T13:45:08.856 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/138' 2023-10-30T13:45:08.856 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/19' 2023-10-30T13:45:08.856 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/15' 2023-10-30T13:45:08.857 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/20' 2023-10-30T13:45:08.857 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/12' 2023-10-30T13:45:08.857 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/10' 2023-10-30T13:45:08.857 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/11' 2023-10-30T13:45:08.857 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/16' 2023-10-30T13:45:08.858 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/14' 2023-10-30T13:45:08.858 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/18' 2023-10-30T13:45:08.858 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/13' 2023-10-30T13:45:08.858 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/135/17' 2023-10-30T13:45:08.858 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/135' 2023-10-30T13:45:08.859 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/19' 2023-10-30T13:45:08.859 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/15' 2023-10-30T13:45:08.859 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/20' 2023-10-30T13:45:08.859 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/12' 2023-10-30T13:45:08.859 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/10' 2023-10-30T13:45:08.860 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/11' 2023-10-30T13:45:08.860 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/16' 2023-10-30T13:45:08.860 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/14' 2023-10-30T13:45:08.860 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/18' 2023-10-30T13:45:08.860 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/13' 2023-10-30T13:45:08.860 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/117/17' 2023-10-30T13:45:08.861 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/117' 2023-10-30T13:45:08.861 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/19' 2023-10-30T13:45:08.861 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/15' 2023-10-30T13:45:08.861 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/20' 2023-10-30T13:45:08.861 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/12' 2023-10-30T13:45:08.861 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/10' 2023-10-30T13:45:08.862 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/11' 2023-10-30T13:45:08.862 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/16' 2023-10-30T13:45:08.862 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/14' 2023-10-30T13:45:08.862 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/18' 2023-10-30T13:45:08.862 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/13' 2023-10-30T13:45:08.863 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/143/17' 2023-10-30T13:45:08.863 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/143' 2023-10-30T13:45:08.863 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/19' 2023-10-30T13:45:08.863 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/15' 2023-10-30T13:45:08.863 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/20' 2023-10-30T13:45:08.863 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/12' 2023-10-30T13:45:08.864 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/10' 2023-10-30T13:45:08.864 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/11' 2023-10-30T13:45:08.864 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/16' 2023-10-30T13:45:08.864 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/14' 2023-10-30T13:45:08.864 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/18' 2023-10-30T13:45:08.864 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/13' 2023-10-30T13:45:08.865 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/180/17' 2023-10-30T13:45:08.865 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/180' 2023-10-30T13:45:08.865 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/19' 2023-10-30T13:45:08.865 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/15' 2023-10-30T13:45:08.865 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/20' 2023-10-30T13:45:08.865 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/12' 2023-10-30T13:45:08.866 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/10' 2023-10-30T13:45:08.866 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/11' 2023-10-30T13:45:08.866 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/16' 2023-10-30T13:45:08.866 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/14' 2023-10-30T13:45:08.866 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/18' 2023-10-30T13:45:08.866 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/13' 2023-10-30T13:45:08.867 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/126/17' 2023-10-30T13:45:08.867 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/126' 2023-10-30T13:45:08.867 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/19' 2023-10-30T13:45:08.867 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/15' 2023-10-30T13:45:08.867 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/20' 2023-10-30T13:45:08.868 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/12' 2023-10-30T13:45:08.868 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/10' 2023-10-30T13:45:08.868 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/11' 2023-10-30T13:45:08.868 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/16' 2023-10-30T13:45:08.868 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/14' 2023-10-30T13:45:08.869 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/18' 2023-10-30T13:45:08.869 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/13' 2023-10-30T13:45:08.869 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/161/17' 2023-10-30T13:45:08.869 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/161' 2023-10-30T13:45:08.869 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/19' 2023-10-30T13:45:08.869 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/15' 2023-10-30T13:45:08.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/20' 2023-10-30T13:45:08.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/12' 2023-10-30T13:45:08.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/10' 2023-10-30T13:45:08.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/11' 2023-10-30T13:45:08.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/16' 2023-10-30T13:45:08.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/14' 2023-10-30T13:45:08.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/18' 2023-10-30T13:45:08.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/13' 2023-10-30T13:45:08.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/176/17' 2023-10-30T13:45:08.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/176' 2023-10-30T13:45:08.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/19' 2023-10-30T13:45:08.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/15' 2023-10-30T13:45:08.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/20' 2023-10-30T13:45:08.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/12' 2023-10-30T13:45:08.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/10' 2023-10-30T13:45:08.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/11' 2023-10-30T13:45:08.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/16' 2023-10-30T13:45:08.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/14' 2023-10-30T13:45:08.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/18' 2023-10-30T13:45:08.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/13' 2023-10-30T13:45:08.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/111/17' 2023-10-30T13:45:08.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/111' 2023-10-30T13:45:08.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/19' 2023-10-30T13:45:08.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/15' 2023-10-30T13:45:08.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/20' 2023-10-30T13:45:08.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/12' 2023-10-30T13:45:08.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/10' 2023-10-30T13:45:08.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/11' 2023-10-30T13:45:08.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/16' 2023-10-30T13:45:08.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/14' 2023-10-30T13:45:08.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/18' 2023-10-30T13:45:08.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/13' 2023-10-30T13:45:10.696 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/191/17' 2023-10-30T13:45:10.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/191' 2023-10-30T13:45:10.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/19' 2023-10-30T13:45:10.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/15' 2023-10-30T13:45:10.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/20' 2023-10-30T13:45:10.697 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/12' 2023-10-30T13:45:10.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/10' 2023-10-30T13:45:10.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/11' 2023-10-30T13:45:10.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/16' 2023-10-30T13:45:10.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/14' 2023-10-30T13:45:10.698 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/18' 2023-10-30T13:45:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/13' 2023-10-30T13:45:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/184/17' 2023-10-30T13:45:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/184' 2023-10-30T13:45:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/19' 2023-10-30T13:45:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/15' 2023-10-30T13:45:10.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/20' 2023-10-30T13:45:10.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/12' 2023-10-30T13:45:10.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/10' 2023-10-30T13:45:10.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/11' 2023-10-30T13:45:10.700 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/16' 2023-10-30T13:45:10.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/14' 2023-10-30T13:45:10.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/18' 2023-10-30T13:45:10.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/13' 2023-10-30T13:45:10.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/192/17' 2023-10-30T13:45:10.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/192' 2023-10-30T13:45:10.701 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/19' 2023-10-30T13:45:10.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/15' 2023-10-30T13:45:10.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/20' 2023-10-30T13:45:10.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/12' 2023-10-30T13:45:10.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/10' 2023-10-30T13:45:10.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/11' 2023-10-30T13:45:10.702 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/16' 2023-10-30T13:45:10.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/14' 2023-10-30T13:45:10.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/18' 2023-10-30T13:45:10.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/13' 2023-10-30T13:45:10.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/103/17' 2023-10-30T13:45:10.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/103' 2023-10-30T13:45:10.703 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/19' 2023-10-30T13:45:10.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/15' 2023-10-30T13:45:10.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/20' 2023-10-30T13:45:10.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/12' 2023-10-30T13:45:10.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/10' 2023-10-30T13:45:10.704 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/11' 2023-10-30T13:45:10.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/16' 2023-10-30T13:45:10.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/14' 2023-10-30T13:45:10.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/18' 2023-10-30T13:45:10.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/13' 2023-10-30T13:45:10.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/128/17' 2023-10-30T13:45:10.705 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/128' 2023-10-30T13:45:10.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/19' 2023-10-30T13:45:10.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/15' 2023-10-30T13:45:10.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/20' 2023-10-30T13:45:10.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/12' 2023-10-30T13:45:10.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/10' 2023-10-30T13:45:10.706 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/11' 2023-10-30T13:45:10.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/16' 2023-10-30T13:45:10.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/14' 2023-10-30T13:45:10.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/18' 2023-10-30T13:45:10.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/13' 2023-10-30T13:45:10.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/144/17' 2023-10-30T13:45:10.707 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/144' 2023-10-30T13:45:10.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/19' 2023-10-30T13:45:10.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/15' 2023-10-30T13:45:10.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/20' 2023-10-30T13:45:10.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/12' 2023-10-30T13:45:10.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/10' 2023-10-30T13:45:10.708 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/11' 2023-10-30T13:45:10.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/16' 2023-10-30T13:45:10.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/14' 2023-10-30T13:45:10.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/18' 2023-10-30T13:45:10.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/13' 2023-10-30T13:45:10.709 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/146/17' 2023-10-30T13:45:10.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/146' 2023-10-30T13:45:10.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/19' 2023-10-30T13:45:10.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/15' 2023-10-30T13:45:10.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/20' 2023-10-30T13:45:10.710 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/12' 2023-10-30T13:45:10.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/10' 2023-10-30T13:45:10.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/11' 2023-10-30T13:45:10.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/16' 2023-10-30T13:45:10.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/14' 2023-10-30T13:45:10.711 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/18' 2023-10-30T13:45:10.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/13' 2023-10-30T13:45:10.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/141/17' 2023-10-30T13:45:10.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/141' 2023-10-30T13:45:10.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/19' 2023-10-30T13:45:10.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/15' 2023-10-30T13:45:10.712 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/20' 2023-10-30T13:45:10.713 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/12' 2023-10-30T13:45:10.713 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/10' 2023-10-30T13:45:10.713 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/11' 2023-10-30T13:45:10.713 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/16' 2023-10-30T13:45:10.713 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/14' 2023-10-30T13:45:10.714 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/18' 2023-10-30T13:45:10.714 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/13' 2023-10-30T13:45:10.714 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/182/17' 2023-10-30T13:45:10.714 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/182' 2023-10-30T13:45:10.714 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/19' 2023-10-30T13:45:10.715 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/15' 2023-10-30T13:45:10.715 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/20' 2023-10-30T13:45:10.715 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/12' 2023-10-30T13:45:10.715 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/10' 2023-10-30T13:45:10.715 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/11' 2023-10-30T13:45:10.716 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/16' 2023-10-30T13:45:10.716 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/14' 2023-10-30T13:45:10.716 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/18' 2023-10-30T13:45:10.716 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/13' 2023-10-30T13:45:10.716 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/107/17' 2023-10-30T13:45:11.751 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/107' 2023-10-30T13:45:11.751 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/19' 2023-10-30T13:45:11.752 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/15' 2023-10-30T13:45:11.752 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/20' 2023-10-30T13:45:11.752 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/12' 2023-10-30T13:45:11.752 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/10' 2023-10-30T13:45:11.752 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/11' 2023-10-30T13:45:11.753 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/16' 2023-10-30T13:45:11.753 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/14' 2023-10-30T13:45:11.753 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/18' 2023-10-30T13:45:11.753 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/13' 2023-10-30T13:45:11.754 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/178/17' 2023-10-30T13:45:11.754 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/178' 2023-10-30T13:45:11.754 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/19' 2023-10-30T13:45:11.754 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/15' 2023-10-30T13:45:11.754 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/20' 2023-10-30T13:45:11.755 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/12' 2023-10-30T13:45:11.755 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/10' 2023-10-30T13:45:11.755 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/11' 2023-10-30T13:45:11.755 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/16' 2023-10-30T13:45:11.756 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/14' 2023-10-30T13:45:11.756 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/18' 2023-10-30T13:45:11.756 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/13' 2023-10-30T13:45:11.756 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/156/17' 2023-10-30T13:45:11.756 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/156' 2023-10-30T13:45:11.757 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/19' 2023-10-30T13:45:11.757 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/15' 2023-10-30T13:45:11.757 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/20' 2023-10-30T13:45:11.757 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/12' 2023-10-30T13:45:11.757 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/10' 2023-10-30T13:45:11.758 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/11' 2023-10-30T13:45:11.758 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/16' 2023-10-30T13:45:11.758 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/14' 2023-10-30T13:45:11.758 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/18' 2023-10-30T13:45:11.758 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/13' 2023-10-30T13:45:11.759 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/131/17' 2023-10-30T13:45:11.759 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/131' 2023-10-30T13:45:11.759 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/19' 2023-10-30T13:45:11.759 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/15' 2023-10-30T13:45:11.759 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/20' 2023-10-30T13:45:11.759 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/12' 2023-10-30T13:45:11.760 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/10' 2023-10-30T13:45:11.760 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/11' 2023-10-30T13:45:11.760 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/16' 2023-10-30T13:45:11.760 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/14' 2023-10-30T13:45:11.760 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/18' 2023-10-30T13:45:11.761 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/13' 2023-10-30T13:45:11.761 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/123/17' 2023-10-30T13:45:11.761 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/123' 2023-10-30T13:45:11.761 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/19' 2023-10-30T13:45:11.761 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/15' 2023-10-30T13:45:11.762 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/20' 2023-10-30T13:45:11.762 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/12' 2023-10-30T13:45:11.762 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/10' 2023-10-30T13:45:11.762 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/11' 2023-10-30T13:45:11.762 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/16' 2023-10-30T13:45:11.763 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/14' 2023-10-30T13:45:11.763 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/18' 2023-10-30T13:45:11.763 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/13' 2023-10-30T13:45:11.763 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/160/17' 2023-10-30T13:45:11.763 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/160' 2023-10-30T13:45:11.763 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/19' 2023-10-30T13:45:11.764 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/15' 2023-10-30T13:45:11.764 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/20' 2023-10-30T13:45:11.764 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/12' 2023-10-30T13:45:11.764 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/10' 2023-10-30T13:45:11.764 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/11' 2023-10-30T13:45:11.765 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/16' 2023-10-30T13:45:11.765 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/14' 2023-10-30T13:45:11.765 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/18' 2023-10-30T13:45:11.765 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/13' 2023-10-30T13:45:11.765 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/165/17' 2023-10-30T13:45:11.765 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/165' 2023-10-30T13:45:11.766 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/19' 2023-10-30T13:45:11.766 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/15' 2023-10-30T13:45:11.766 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/20' 2023-10-30T13:45:11.766 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/12' 2023-10-30T13:45:11.767 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/10' 2023-10-30T13:45:11.767 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/11' 2023-10-30T13:45:11.767 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/16' 2023-10-30T13:45:11.767 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/14' 2023-10-30T13:45:11.767 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/18' 2023-10-30T13:45:11.768 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/13' 2023-10-30T13:45:11.768 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/101/17' 2023-10-30T13:45:11.768 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/101' 2023-10-30T13:45:11.768 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/19' 2023-10-30T13:45:11.768 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/15' 2023-10-30T13:45:11.769 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/20' 2023-10-30T13:45:11.769 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/12' 2023-10-30T13:45:11.769 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/10' 2023-10-30T13:45:11.769 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/11' 2023-10-30T13:45:11.769 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/16' 2023-10-30T13:45:11.770 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/14' 2023-10-30T13:45:11.770 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/18' 2023-10-30T13:45:11.770 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/13' 2023-10-30T13:45:11.770 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/148/17' 2023-10-30T13:45:11.771 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/148' 2023-10-30T13:45:11.771 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/19' 2023-10-30T13:45:11.771 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/15' 2023-10-30T13:45:11.771 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/20' 2023-10-30T13:45:11.771 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/12' 2023-10-30T13:45:11.772 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/10' 2023-10-30T13:45:11.772 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/11' 2023-10-30T13:45:11.772 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/16' 2023-10-30T13:45:11.772 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/14' 2023-10-30T13:45:11.772 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/18' 2023-10-30T13:45:11.772 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/13' 2023-10-30T13:45:11.773 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/168/17' 2023-10-30T13:45:11.773 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/168' 2023-10-30T13:45:13.122 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/19' 2023-10-30T13:45:13.122 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/15' 2023-10-30T13:45:13.123 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/20' 2023-10-30T13:45:13.123 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/12' 2023-10-30T13:45:13.123 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/10' 2023-10-30T13:45:13.123 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/11' 2023-10-30T13:45:13.123 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/16' 2023-10-30T13:45:13.123 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/14' 2023-10-30T13:45:13.124 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/18' 2023-10-30T13:45:13.124 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/13' 2023-10-30T13:45:13.124 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/179/17' 2023-10-30T13:45:13.124 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/179' 2023-10-30T13:45:13.124 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/19' 2023-10-30T13:45:13.124 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/15' 2023-10-30T13:45:13.125 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/20' 2023-10-30T13:45:13.125 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/12' 2023-10-30T13:45:13.125 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/10' 2023-10-30T13:45:13.125 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/11' 2023-10-30T13:45:13.125 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/16' 2023-10-30T13:45:13.125 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/14' 2023-10-30T13:45:13.126 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/18' 2023-10-30T13:45:13.126 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/13' 2023-10-30T13:45:13.126 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/132/17' 2023-10-30T13:45:13.126 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/132' 2023-10-30T13:45:13.126 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/19' 2023-10-30T13:45:13.126 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/15' 2023-10-30T13:45:13.127 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/20' 2023-10-30T13:45:13.127 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/12' 2023-10-30T13:45:13.127 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/10' 2023-10-30T13:45:13.127 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/11' 2023-10-30T13:45:13.127 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/16' 2023-10-30T13:45:13.128 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/14' 2023-10-30T13:45:13.128 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/18' 2023-10-30T13:45:13.128 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/13' 2023-10-30T13:45:13.128 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/199/17' 2023-10-30T13:45:13.128 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/199' 2023-10-30T13:45:13.128 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/19' 2023-10-30T13:45:13.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/15' 2023-10-30T13:45:13.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/20' 2023-10-30T13:45:13.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/12' 2023-10-30T13:45:13.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/10' 2023-10-30T13:45:13.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/11' 2023-10-30T13:45:13.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/16' 2023-10-30T13:45:13.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/14' 2023-10-30T13:45:13.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/18' 2023-10-30T13:45:13.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/13' 2023-10-30T13:45:13.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/163/17' 2023-10-30T13:45:13.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/163' 2023-10-30T13:45:13.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/19' 2023-10-30T13:45:13.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/15' 2023-10-30T13:45:13.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/20' 2023-10-30T13:45:13.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/12' 2023-10-30T13:45:13.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/10' 2023-10-30T13:45:13.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/11' 2023-10-30T13:45:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/16' 2023-10-30T13:45:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/14' 2023-10-30T13:45:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/18' 2023-10-30T13:45:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/13' 2023-10-30T13:45:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/153/17' 2023-10-30T13:45:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/153' 2023-10-30T13:45:13.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/19' 2023-10-30T13:45:13.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/15' 2023-10-30T13:45:13.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/20' 2023-10-30T13:45:13.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/12' 2023-10-30T13:45:13.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/10' 2023-10-30T13:45:13.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/11' 2023-10-30T13:45:13.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/16' 2023-10-30T13:45:13.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/14' 2023-10-30T13:45:13.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/18' 2023-10-30T13:45:13.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/13' 2023-10-30T13:45:13.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/122/17' 2023-10-30T13:45:13.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/122' 2023-10-30T13:45:13.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/19' 2023-10-30T13:45:13.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/15' 2023-10-30T13:45:13.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/20' 2023-10-30T13:45:13.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/12' 2023-10-30T13:45:13.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/10' 2023-10-30T13:45:13.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/11' 2023-10-30T13:45:13.136 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/16' 2023-10-30T13:45:13.136 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/14' 2023-10-30T13:45:13.136 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/18' 2023-10-30T13:45:13.136 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/13' 2023-10-30T13:45:13.136 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/162/17' 2023-10-30T13:45:13.136 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/162' 2023-10-30T13:45:13.137 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/19' 2023-10-30T13:45:13.137 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/15' 2023-10-30T13:45:13.137 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/20' 2023-10-30T13:45:13.137 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/12' 2023-10-30T13:45:13.137 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/10' 2023-10-30T13:45:13.137 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/11' 2023-10-30T13:45:13.138 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/16' 2023-10-30T13:45:13.138 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/14' 2023-10-30T13:45:13.138 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/18' 2023-10-30T13:45:13.138 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/13' 2023-10-30T13:45:13.138 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/197/17' 2023-10-30T13:45:13.139 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/197' 2023-10-30T13:45:13.139 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/19' 2023-10-30T13:45:13.139 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/15' 2023-10-30T13:45:13.139 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/20' 2023-10-30T13:45:13.139 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/12' 2023-10-30T13:45:13.139 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/10' 2023-10-30T13:45:13.140 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/11' 2023-10-30T13:45:13.140 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/16' 2023-10-30T13:45:13.140 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/14' 2023-10-30T13:45:13.140 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/18' 2023-10-30T13:45:13.140 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/13' 2023-10-30T13:45:13.140 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/113/17' 2023-10-30T13:45:13.141 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/113' 2023-10-30T13:45:13.141 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/19' 2023-10-30T13:45:13.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/15' 2023-10-30T13:45:13.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/20' 2023-10-30T13:45:13.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/12' 2023-10-30T13:45:13.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/10' 2023-10-30T13:45:13.380 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/11' 2023-10-30T13:45:13.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/16' 2023-10-30T13:45:13.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/14' 2023-10-30T13:45:13.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/18' 2023-10-30T13:45:13.381 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/13' 2023-10-30T13:45:13.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/152/17' 2023-10-30T13:45:13.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/152' 2023-10-30T13:45:13.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/19' 2023-10-30T13:45:13.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/15' 2023-10-30T13:45:13.382 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/20' 2023-10-30T13:45:13.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/12' 2023-10-30T13:45:13.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/10' 2023-10-30T13:45:13.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/11' 2023-10-30T13:45:13.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/16' 2023-10-30T13:45:13.383 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/14' 2023-10-30T13:45:13.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/18' 2023-10-30T13:45:13.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/13' 2023-10-30T13:45:13.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/121/17' 2023-10-30T13:45:13.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/121' 2023-10-30T13:45:13.384 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/19' 2023-10-30T13:45:13.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/15' 2023-10-30T13:45:13.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/20' 2023-10-30T13:45:13.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/12' 2023-10-30T13:45:13.385 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/10' 2023-10-30T13:45:13.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/11' 2023-10-30T13:45:13.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/16' 2023-10-30T13:45:13.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/14' 2023-10-30T13:45:13.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/18' 2023-10-30T13:45:13.386 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/13' 2023-10-30T13:45:13.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/118/17' 2023-10-30T13:45:13.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/118' 2023-10-30T13:45:13.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/19' 2023-10-30T13:45:13.387 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/15' 2023-10-30T13:45:13.388 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/20' 2023-10-30T13:45:13.388 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/12' 2023-10-30T13:45:13.388 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/10' 2023-10-30T13:45:13.388 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/11' 2023-10-30T13:45:13.388 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/16' 2023-10-30T13:45:13.389 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/14' 2023-10-30T13:45:13.389 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/18' 2023-10-30T13:45:13.389 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/13' 2023-10-30T13:45:13.389 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/196/17' 2023-10-30T13:45:13.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/196' 2023-10-30T13:45:13.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/19' 2023-10-30T13:45:13.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/15' 2023-10-30T13:45:13.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/20' 2023-10-30T13:45:13.390 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/12' 2023-10-30T13:45:13.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/10' 2023-10-30T13:45:13.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/11' 2023-10-30T13:45:13.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/16' 2023-10-30T13:45:13.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/14' 2023-10-30T13:45:13.391 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/18' 2023-10-30T13:45:13.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/13' 2023-10-30T13:45:13.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/124/17' 2023-10-30T13:45:13.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/124' 2023-10-30T13:45:13.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/19' 2023-10-30T13:45:13.392 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/15' 2023-10-30T13:45:13.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/20' 2023-10-30T13:45:13.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/12' 2023-10-30T13:45:13.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/10' 2023-10-30T13:45:13.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/11' 2023-10-30T13:45:13.393 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/16' 2023-10-30T13:45:13.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/14' 2023-10-30T13:45:13.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/18' 2023-10-30T13:45:13.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/13' 2023-10-30T13:45:13.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/188/17' 2023-10-30T13:45:13.394 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/188' 2023-10-30T13:45:13.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/19' 2023-10-30T13:45:13.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/15' 2023-10-30T13:45:13.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/20' 2023-10-30T13:45:13.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/12' 2023-10-30T13:45:13.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/10' 2023-10-30T13:45:13.395 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/11' 2023-10-30T13:45:13.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/16' 2023-10-30T13:45:13.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/14' 2023-10-30T13:45:13.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/18' 2023-10-30T13:45:13.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/13' 2023-10-30T13:45:13.396 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/145/17' 2023-10-30T13:45:13.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/145' 2023-10-30T13:45:13.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/19' 2023-10-30T13:45:13.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/15' 2023-10-30T13:45:13.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/20' 2023-10-30T13:45:13.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/12' 2023-10-30T13:45:13.397 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/10' 2023-10-30T13:45:13.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/11' 2023-10-30T13:45:13.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/16' 2023-10-30T13:45:13.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/14' 2023-10-30T13:45:13.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/18' 2023-10-30T13:45:13.398 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/13' 2023-10-30T13:45:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/130/17' 2023-10-30T13:45:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/130' 2023-10-30T13:45:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/19' 2023-10-30T13:45:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/15' 2023-10-30T13:45:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/20' 2023-10-30T13:45:13.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/12' 2023-10-30T13:45:13.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/10' 2023-10-30T13:45:13.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/11' 2023-10-30T13:45:13.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/16' 2023-10-30T13:45:13.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/14' 2023-10-30T13:45:13.400 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/18' 2023-10-30T13:45:13.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/13' 2023-10-30T13:45:13.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/193/17' 2023-10-30T13:45:13.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/193' 2023-10-30T13:45:13.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/19' 2023-10-30T13:45:13.401 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/15' 2023-10-30T13:45:13.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/20' 2023-10-30T13:45:13.870 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/12' 2023-10-30T13:45:13.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/10' 2023-10-30T13:45:13.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/11' 2023-10-30T13:45:13.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/16' 2023-10-30T13:45:13.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/14' 2023-10-30T13:45:13.871 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/18' 2023-10-30T13:45:13.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/13' 2023-10-30T13:45:13.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/175/17' 2023-10-30T13:45:13.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/175' 2023-10-30T13:45:13.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/19' 2023-10-30T13:45:13.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/15' 2023-10-30T13:45:13.872 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/20' 2023-10-30T13:45:13.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/12' 2023-10-30T13:45:13.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/10' 2023-10-30T13:45:13.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/11' 2023-10-30T13:45:13.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/16' 2023-10-30T13:45:13.873 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/14' 2023-10-30T13:45:13.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/18' 2023-10-30T13:45:13.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/13' 2023-10-30T13:45:13.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/181/17' 2023-10-30T13:45:13.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/181' 2023-10-30T13:45:13.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/19' 2023-10-30T13:45:13.874 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/15' 2023-10-30T13:45:13.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/20' 2023-10-30T13:45:13.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/12' 2023-10-30T13:45:13.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/10' 2023-10-30T13:45:13.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/11' 2023-10-30T13:45:13.875 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/16' 2023-10-30T13:45:13.876 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/14' 2023-10-30T13:45:13.876 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/18' 2023-10-30T13:45:13.876 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/13' 2023-10-30T13:45:13.876 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/127/17' 2023-10-30T13:45:13.876 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/127' 2023-10-30T13:45:13.877 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/19' 2023-10-30T13:45:13.877 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/15' 2023-10-30T13:45:13.877 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/20' 2023-10-30T13:45:13.877 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/12' 2023-10-30T13:45:13.878 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/10' 2023-10-30T13:45:13.878 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/11' 2023-10-30T13:45:13.878 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/16' 2023-10-30T13:45:13.878 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/14' 2023-10-30T13:45:13.878 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/18' 2023-10-30T13:45:13.879 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/13' 2023-10-30T13:45:13.879 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/108/17' 2023-10-30T13:45:13.879 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/108' 2023-10-30T13:45:13.879 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/19' 2023-10-30T13:45:13.879 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/15' 2023-10-30T13:45:13.880 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/20' 2023-10-30T13:45:13.880 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/12' 2023-10-30T13:45:13.880 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/10' 2023-10-30T13:45:13.880 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/11' 2023-10-30T13:45:13.880 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/16' 2023-10-30T13:45:13.880 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/14' 2023-10-30T13:45:13.881 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/18' 2023-10-30T13:45:13.881 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/13' 2023-10-30T13:45:13.881 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/183/17' 2023-10-30T13:45:13.881 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/183' 2023-10-30T13:45:13.881 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/19' 2023-10-30T13:45:13.882 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/15' 2023-10-30T13:45:13.882 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/20' 2023-10-30T13:45:13.882 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/12' 2023-10-30T13:45:13.882 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/10' 2023-10-30T13:45:13.882 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/11' 2023-10-30T13:45:13.882 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/16' 2023-10-30T13:45:13.883 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/14' 2023-10-30T13:45:13.883 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/18' 2023-10-30T13:45:13.883 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/13' 2023-10-30T13:45:13.883 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/177/17' 2023-10-30T13:45:13.883 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/177' 2023-10-30T13:45:13.883 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/19' 2023-10-30T13:45:13.884 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/15' 2023-10-30T13:45:13.884 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/20' 2023-10-30T13:45:13.884 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/12' 2023-10-30T13:45:13.884 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/10' 2023-10-30T13:45:13.884 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/11' 2023-10-30T13:45:13.885 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/16' 2023-10-30T13:45:13.885 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/14' 2023-10-30T13:45:13.885 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/18' 2023-10-30T13:45:13.885 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/13' 2023-10-30T13:45:13.885 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/169/17' 2023-10-30T13:45:13.885 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/169' 2023-10-30T13:45:13.886 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/19' 2023-10-30T13:45:13.886 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/15' 2023-10-30T13:45:13.886 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/20' 2023-10-30T13:45:13.886 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/12' 2023-10-30T13:45:13.886 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/10' 2023-10-30T13:45:13.887 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/11' 2023-10-30T13:45:13.887 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/16' 2023-10-30T13:45:13.887 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/14' 2023-10-30T13:45:13.887 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/18' 2023-10-30T13:45:13.887 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/13' 2023-10-30T13:45:13.887 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/154/17' 2023-10-30T13:45:13.888 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/154' 2023-10-30T13:45:13.888 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/19' 2023-10-30T13:45:13.888 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/15' 2023-10-30T13:45:13.888 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/20' 2023-10-30T13:45:13.888 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/12' 2023-10-30T13:45:13.888 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/10' 2023-10-30T13:45:13.889 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/11' 2023-10-30T13:45:13.889 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/16' 2023-10-30T13:45:13.889 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/14' 2023-10-30T13:45:13.889 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/18' 2023-10-30T13:45:13.889 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/13' 2023-10-30T13:45:13.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/171/17' 2023-10-30T13:45:13.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/171' 2023-10-30T13:45:13.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/19' 2023-10-30T13:45:13.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/15' 2023-10-30T13:45:13.890 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/20' 2023-10-30T13:45:15.128 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/12' 2023-10-30T13:45:15.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/10' 2023-10-30T13:45:15.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/11' 2023-10-30T13:45:15.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/16' 2023-10-30T13:45:15.129 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/14' 2023-10-30T13:45:15.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/18' 2023-10-30T13:45:15.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/13' 2023-10-30T13:45:15.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/139/17' 2023-10-30T13:45:15.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/139' 2023-10-30T13:45:15.130 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/19' 2023-10-30T13:45:15.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/15' 2023-10-30T13:45:15.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/20' 2023-10-30T13:45:15.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/12' 2023-10-30T13:45:15.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/10' 2023-10-30T13:45:15.131 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/11' 2023-10-30T13:45:15.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/16' 2023-10-30T13:45:15.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/14' 2023-10-30T13:45:15.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/18' 2023-10-30T13:45:15.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/13' 2023-10-30T13:45:15.132 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/102/17' 2023-10-30T13:45:15.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/102' 2023-10-30T13:45:15.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/19' 2023-10-30T13:45:15.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/15' 2023-10-30T13:45:15.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/20' 2023-10-30T13:45:15.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/12' 2023-10-30T13:45:15.133 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/10' 2023-10-30T13:45:15.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/11' 2023-10-30T13:45:15.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/16' 2023-10-30T13:45:15.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/14' 2023-10-30T13:45:15.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/18' 2023-10-30T13:45:15.134 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/13' 2023-10-30T13:45:15.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed './copyofsnap1/test1/129/17' 2023-10-30T13:45:15.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1/129' 2023-10-30T13:45:15.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1/test1' 2023-10-30T13:45:15.135 INFO:tasks.workunit.client.0.smithi031.stdout:removed directory './copyofsnap1' 2023-10-30T13:45:15.135 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:45:15.136 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:45:15.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:45:15.170 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-30T13:45:15.170 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-30T13:45:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -e 2023-10-30T13:45:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:+ touch foo 2023-10-30T13:45:15.231 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod +x foo 2023-10-30T13:45:15.232 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir .snap/s 2023-10-30T13:45:15.251 INFO:tasks.workunit.client.0.smithi031.stderr:+ find .snap/s/foo -executable 2023-10-30T13:45:15.251 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep foo 2023-10-30T13:45:15.253 INFO:tasks.workunit.client.0.smithi031.stdout:.snap/s/foo 2023-10-30T13:45:15.254 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir .snap/s 2023-10-30T13:45:15.263 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm foo 2023-10-30T13:45:15.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2023-10-30T13:45:15.267 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:45:15.267 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:45:15.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:45:15.335 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-30T13:45:15.336 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-30T13:45:15.396 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -e 2023-10-30T13:45:15.397 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir foo 2023-10-30T13:45:15.399 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo x 2023-10-30T13:45:15.399 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:16.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir foo/.snap/ss 2023-10-30T13:45:16.532 INFO:tasks.workunit.client.0.smithi031.stderr:+ ln foo/x foo/xx 2023-10-30T13:45:16.534 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat foo/.snap/ss/x 2023-10-30T13:45:16.544 INFO:tasks.workunit.client.0.smithi031.stdout:x 2023-10-30T13:45:16.545 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir foo/.snap/ss 2023-10-30T13:45:16.555 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo a 2023-10-30T13:45:16.556 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo b 2023-10-30T13:45:16.556 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir foo/.snap/s 2023-10-30T13:45:16.572 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat foo/.snap/s/a 2023-10-30T13:45:16.584 INFO:tasks.workunit.client.0.smithi031.stderr:+ r=a 2023-10-30T13:45:16.584 INFO:tasks.workunit.client.0.smithi031.stderr:+ [ -z a ] 2023-10-30T13:45:16.584 INFO:tasks.workunit.client.0.smithi031.stderr:+ ln foo/b foo/b2 2023-10-30T13:45:16.586 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat foo/.snap/s/b 2023-10-30T13:45:17.211 INFO:tasks.workunit.client.0.smithi031.stdout:b 2023-10-30T13:45:17.212 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo this used to hang: 2023-10-30T13:45:17.212 INFO:tasks.workunit.client.0.smithi031.stdout:this used to hang: 2023-10-30T13:45:17.212 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo more 2023-10-30T13:45:17.213 INFO:tasks.workunit.client.0.smithi031.stdout:oh, it didn't hang! good job. 2023-10-30T13:45:17.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo oh, it didn't hang! good job. 2023-10-30T13:45:17.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat foo/b 2023-10-30T13:45:17.214 INFO:tasks.workunit.client.0.smithi031.stdout:b 2023-10-30T13:45:17.214 INFO:tasks.workunit.client.0.smithi031.stdout:more 2023-10-30T13:45:17.214 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir foo/.snap/s 2023-10-30T13:45:17.222 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm -r foo 2023-10-30T13:45:17.240 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2023-10-30T13:45:17.241 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:45:17.241 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:45:17.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:45:17.312 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-30T13:45:17.313 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-30T13:45:17.376 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -e 2023-10-30T13:45:17.376 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:17.378 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir .snap/bar 2023-10-30T13:45:17.394 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/foo 2023-10-30T13:45:17.406 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/foo 2023-10-30T13:45:17.821 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:17.824 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/bar/a 2023-10-30T13:45:17.828 INFO:tasks.workunit.client.0.smithi031.stdout: File: .snap/bar/a 2023-10-30T13:45:17.828 INFO:tasks.workunit.client.0.smithi031.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T13:45:17.829 INFO:tasks.workunit.client.0.smithi031.stdout:Device: fah/250d Inode: 1099511632959 Links: 2 2023-10-30T13:45:17.829 INFO:tasks.workunit.client.0.smithi031.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T13:45:17.829 INFO:tasks.workunit.client.0.smithi031.stdout:Access: 2023-10-30 13:45:17.374347566 +0000 2023-10-30T13:45:17.829 INFO:tasks.workunit.client.0.smithi031.stdout:Modify: 2023-10-30 13:45:17.374347566 +0000 2023-10-30T13:45:17.829 INFO:tasks.workunit.client.0.smithi031.stdout:Change: 2023-10-30 13:45:17.374347566 +0000 2023-10-30T13:45:17.830 INFO:tasks.workunit.client.0.smithi031.stdout: Birth: 2023-10-30 13:45:17.374347566 +0000 2023-10-30T13:45:17.830 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir .snap/bar 2023-10-30T13:45:17.838 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:45:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2023-10-30T13:45:17.840 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:45:17.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:45:17.913 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-30T13:45:17.913 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-30T13:45:17.972 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -e 2023-10-30T13:45:17.972 INFO:tasks.workunit.client.0.smithi031.stderr:+ seq 1 20 2023-10-30T13:45:17.973 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:17.976 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:17.976 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:17.986 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:17.992 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:17.994 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.177 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.192 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.193 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.193 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.193 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.193 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.194 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.194 INFO:tasks.workunit.client.0.smithi031.stdout: 51427 ? 00:00:00 ps 2023-10-30T13:45:18.194 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.204 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.207 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.212 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.215 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.216 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.224 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.230 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.232 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.316 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.317 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.317 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.317 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.317 INFO:tasks.workunit.client.0.smithi031.stdout: 51437 ? 00:00:00 ps 2023-10-30T13:45:18.318 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.327 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.329 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.334 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.337 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.338 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.346 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.352 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.354 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.401 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.416 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.416 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.416 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.416 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.417 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.417 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.417 INFO:tasks.workunit.client.0.smithi031.stdout: 51447 ? 00:00:00 ps 2023-10-30T13:45:18.417 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.427 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.429 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.434 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.437 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.438 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.447 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.453 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.453 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.492 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.507 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.508 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.508 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.508 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.508 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.508 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.509 INFO:tasks.workunit.client.0.smithi031.stdout: 51457 ? 00:00:00 ps 2023-10-30T13:45:18.509 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.527 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.529 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.530 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.538 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.543 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.544 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.585 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.600 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.600 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.600 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.600 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.601 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.601 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.601 INFO:tasks.workunit.client.0.smithi031.stdout: 51467 ? 00:00:00 ps 2023-10-30T13:45:18.601 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.611 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.613 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.618 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.621 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.622 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.630 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.636 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.638 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.676 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.691 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.692 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.692 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.692 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.692 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.692 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.692 INFO:tasks.workunit.client.0.smithi031.stdout: 51477 ? 00:00:00 ps 2023-10-30T13:45:18.693 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.703 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.705 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.710 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.713 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.714 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.722 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.727 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.729 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.768 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.783 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.783 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.783 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.783 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.784 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.784 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.784 INFO:tasks.workunit.client.0.smithi031.stdout: 51487 ? 00:00:00 ps 2023-10-30T13:45:18.784 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.794 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.796 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.802 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.805 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.806 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.816 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.824 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:18.879 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:18.894 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:18.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:18.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:18.895 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:18.895 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:18.895 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:18.895 INFO:tasks.workunit.client.0.smithi031.stdout: 51497 ? 00:00:00 ps 2023-10-30T13:45:18.896 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:18.906 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:18.908 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:18.913 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:18.916 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:18.917 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:18.926 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:18.937 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:18.938 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:19.002 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:19.017 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:19.018 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:19.018 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:19.018 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:19.018 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:19.019 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:19.019 INFO:tasks.workunit.client.0.smithi031.stdout: 51507 ? 00:00:00 ps 2023-10-30T13:45:19.019 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:19.030 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:19.032 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:19.039 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:19.042 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:19.044 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:19.052 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:19.058 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:19.059 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:19.111 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:19.138 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:19.139 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:19.139 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:19.139 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:19.139 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:19.139 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:19.139 INFO:tasks.workunit.client.0.smithi031.stdout: 51517 ? 00:00:00 ps 2023-10-30T13:45:19.140 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:19.169 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:19.175 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:19.194 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:19.199 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:19.200 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:19.235 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:19.264 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:19.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:19.495 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:19.509 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:19.510 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:19.510 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:19.510 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:19.510 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:19.510 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:19.511 INFO:tasks.workunit.client.0.smithi031.stdout: 51527 ? 00:00:00 ps 2023-10-30T13:45:19.760 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:19.771 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:19.773 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:19.779 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:19.781 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:19.782 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:19.791 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:19.796 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:19.798 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:19.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:19.977 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:19.977 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:19.977 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:19.977 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:19.977 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:19.978 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:19.978 INFO:tasks.workunit.client.0.smithi031.stdout: 51538 ? 00:00:00 ps 2023-10-30T13:45:20.700 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:20.908 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:20.910 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:20.916 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:20.919 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:20.920 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:21.616 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:21.623 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:21.624 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:21.779 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:21.795 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:21.796 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:21.796 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:21.796 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:21.796 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:21.796 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:21.797 INFO:tasks.workunit.client.0.smithi031.stdout: 51548 ? 00:00:00 ps 2023-10-30T13:45:21.797 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:21.808 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:21.811 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:21.818 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:21.822 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:21.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:21.834 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:21.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:21.842 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:21.878 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:21.893 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:21.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:21.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:21.894 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:21.894 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:21.894 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:21.894 INFO:tasks.workunit.client.0.smithi031.stdout: 51558 ? 00:00:00 ps 2023-10-30T13:45:21.895 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:21.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:21.907 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:21.913 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:21.916 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:21.916 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:21.926 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:21.933 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:21.934 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:21.986 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:22.001 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:22.002 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:22.002 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:22.002 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:22.002 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:22.002 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:22.003 INFO:tasks.workunit.client.0.smithi031.stdout: 51568 ? 00:00:00 ps 2023-10-30T13:45:22.003 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:22.013 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:22.015 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:22.021 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:22.024 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:22.025 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:22.285 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:22.291 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:22.292 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:22.421 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:22.436 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:22.436 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:22.436 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:22.437 INFO:tasks.workunit.client.0.smithi031.stdout: 51578 ? 00:00:00 ps 2023-10-30T13:45:22.438 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:22.447 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:22.449 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:22.454 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:22.457 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:22.458 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:22.467 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:22.473 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:22.475 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:22.529 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: 51588 ? 00:00:00 ps 2023-10-30T13:45:22.546 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:22.555 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:22.557 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:22.563 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:22.565 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:22.566 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:22.575 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:22.581 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:22.582 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:22.629 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:22.644 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:22.644 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:22.644 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:22.645 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:22.645 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:22.645 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:22.645 INFO:tasks.workunit.client.0.smithi031.stdout: 51598 ? 00:00:00 ps 2023-10-30T13:45:22.646 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:22.655 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:22.657 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:22.663 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:22.666 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:22.667 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:22.676 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:22.682 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:22.683 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:22.729 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:22.744 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:22.744 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:22.744 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:22.745 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:22.745 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:22.745 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:22.745 INFO:tasks.workunit.client.0.smithi031.stdout: 51608 ? 00:00:00 ps 2023-10-30T13:45:22.746 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:22.755 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:22.758 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:22.764 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a 2023-10-30T13:45:22.766 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/one 2023-10-30T13:45:22.767 INFO:tasks.workunit.client.0.smithi031.stderr:+ cat 2023-10-30T13:45:22.777 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir a/.snap/two 2023-10-30T13:45:22.783 INFO:tasks.workunit.client.0.smithi031.stderr:+ chmod 777 a/foo 2023-10-30T13:45:22.784 INFO:tasks.workunit.client.0.smithi031.stderr:+ sync 2023-10-30T13:45:22.821 INFO:tasks.workunit.client.0.smithi031.stderr:+ ps 2023-10-30T13:45:22.836 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2023-10-30T13:45:22.836 INFO:tasks.workunit.client.0.smithi031.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:45:22.836 INFO:tasks.workunit.client.0.smithi031.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:45:22.837 INFO:tasks.workunit.client.0.smithi031.stdout: 27854 ? 00:00:02 sshd 2023-10-30T13:45:22.837 INFO:tasks.workunit.client.0.smithi031.stdout: 51414 ? 00:00:00 timeout 2023-10-30T13:45:22.837 INFO:tasks.workunit.client.0.smithi031.stdout: 51417 ? 00:00:00 snaptest-double 2023-10-30T13:45:22.837 INFO:tasks.workunit.client.0.smithi031.stdout: 51618 ? 00:00:00 ps 2023-10-30T13:45:22.838 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:45:22.847 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm a/foo 2023-10-30T13:45:22.849 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir a 2023-10-30T13:45:22.855 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2023-10-30T13:45:22.856 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:45:22.856 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:45:22.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:45:22.925 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-30T13:45:22.926 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-30T13:45:23.669 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir .snap/foo 2023-10-30T13:45:23.686 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo We want ENOENT, not ESTALE, here. 2023-10-30T13:45:23.686 INFO:tasks.workunit.client.0.smithi031.stdout:We want ENOENT, not ESTALE, here. 2023-10-30T13:45:23.687 INFO:tasks.workunit.client.0.smithi031.stderr:+ seq 1 100 2023-10-30T13:45:23.688 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/1 2023-10-30T13:45:23.688 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.613 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-10-30T13:45:24.613 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/2 2023-10-30T13:45:24.614 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.615 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-10-30T13:45:24.616 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/3 2023-10-30T13:45:24.616 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.617 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-10-30T13:45:24.619 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/4 2023-10-30T13:45:24.619 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.620 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-10-30T13:45:24.621 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/5 2023-10-30T13:45:24.621 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.622 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-10-30T13:45:24.624 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/6 2023-10-30T13:45:24.624 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.625 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-10-30T13:45:24.626 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/7 2023-10-30T13:45:24.626 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.628 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-10-30T13:45:24.629 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/8 2023-10-30T13:45:24.629 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.630 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-10-30T13:45:24.631 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/9 2023-10-30T13:45:24.631 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.633 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-10-30T13:45:24.634 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/10 2023-10-30T13:45:24.634 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.635 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-10-30T13:45:24.636 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/11 2023-10-30T13:45:24.637 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.638 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-10-30T13:45:24.639 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/12 2023-10-30T13:45:24.639 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.640 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-10-30T13:45:24.641 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/13 2023-10-30T13:45:24.641 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.643 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-10-30T13:45:24.644 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/14 2023-10-30T13:45:24.644 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.645 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-10-30T13:45:24.646 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/15 2023-10-30T13:45:24.647 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.648 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-10-30T13:45:24.649 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/16 2023-10-30T13:45:24.649 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.650 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-10-30T13:45:24.651 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/17 2023-10-30T13:45:24.652 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.653 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-10-30T13:45:24.654 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/18 2023-10-30T13:45:24.654 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.655 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-10-30T13:45:24.656 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/19 2023-10-30T13:45:24.657 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.658 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-10-30T13:45:24.659 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/20 2023-10-30T13:45:24.659 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.660 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-10-30T13:45:24.661 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/21 2023-10-30T13:45:24.661 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.663 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-10-30T13:45:24.664 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/22 2023-10-30T13:45:24.664 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.665 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-10-30T13:45:24.666 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/23 2023-10-30T13:45:24.666 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.668 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-10-30T13:45:24.668 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/24 2023-10-30T13:45:24.669 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.671 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-10-30T13:45:24.672 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/25 2023-10-30T13:45:24.672 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.675 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-10-30T13:45:24.678 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/26 2023-10-30T13:45:24.681 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.682 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-10-30T13:45:24.684 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/27 2023-10-30T13:45:24.684 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.688 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-10-30T13:45:24.690 INFO:tasks.workunit.client.0.smithi031.stderr:+ + grep No such file 2023-10-30T13:45:24.690 INFO:tasks.workunit.client.0.smithi031.stderr:stat .snap/foo/28 2023-10-30T13:45:24.692 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-10-30T13:45:24.695 INFO:tasks.workunit.client.0.smithi031.stderr:+ + stat .snap/foo/29 2023-10-30T13:45:24.700 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-10-30T13:45:24.701 INFO:tasks.workunit.client.0.smithi031.stderr:grep No such file 2023-10-30T13:45:24.702 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/30 2023-10-30T13:45:24.704 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.706 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-10-30T13:45:24.708 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/31 2023-10-30T13:45:24.708 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.711 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-10-30T13:45:24.714 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/32 2023-10-30T13:45:24.715 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.717 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-10-30T13:45:24.721 INFO:tasks.workunit.client.0.smithi031.stderr:+ + grep No such file 2023-10-30T13:45:24.721 INFO:tasks.workunit.client.0.smithi031.stderr:stat .snap/foo/33 2023-10-30T13:45:24.726 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-10-30T13:45:24.728 INFO:tasks.workunit.client.0.smithi031.stderr:+ + stat .snap/foo/34 2023-10-30T13:45:24.728 INFO:tasks.workunit.client.0.smithi031.stderr:grep No such file 2023-10-30T13:45:24.732 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-10-30T13:45:24.733 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/35 2023-10-30T13:45:24.733 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.738 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-10-30T13:45:24.739 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/36 2023-10-30T13:45:24.739 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.742 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-10-30T13:45:24.743 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/37 2023-10-30T13:45:24.743 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.747 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-10-30T13:45:24.747 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/38 2023-10-30T13:45:24.748 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.751 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-10-30T13:45:24.751 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/39 2023-10-30T13:45:24.752 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.754 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-10-30T13:45:24.755 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/40 2023-10-30T13:45:24.755 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.756 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-10-30T13:45:24.757 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/41 2023-10-30T13:45:24.757 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.759 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-10-30T13:45:24.760 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/42 2023-10-30T13:45:24.760 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.761 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-10-30T13:45:24.762 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/43 2023-10-30T13:45:24.762 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.763 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-10-30T13:45:24.764 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/44 2023-10-30T13:45:24.764 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.766 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-10-30T13:45:24.766 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/45 2023-10-30T13:45:24.767 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.768 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-10-30T13:45:24.769 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/46 2023-10-30T13:45:24.769 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.770 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-10-30T13:45:24.771 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/47 2023-10-30T13:45:24.771 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.772 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-10-30T13:45:24.773 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/48 2023-10-30T13:45:24.774 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.775 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-10-30T13:45:24.776 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/49 2023-10-30T13:45:24.776 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.777 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-10-30T13:45:24.778 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/50 2023-10-30T13:45:24.778 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.779 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-10-30T13:45:24.780 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/51 2023-10-30T13:45:24.781 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.782 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-10-30T13:45:24.782 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/52 2023-10-30T13:45:24.783 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.784 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-10-30T13:45:24.785 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/53 2023-10-30T13:45:24.785 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.786 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-10-30T13:45:24.787 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/54 2023-10-30T13:45:24.788 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.789 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-10-30T13:45:24.789 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/55 2023-10-30T13:45:24.790 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.791 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-10-30T13:45:24.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/56 2023-10-30T13:45:24.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.793 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-10-30T13:45:24.794 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/57 2023-10-30T13:45:24.794 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.796 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-10-30T13:45:24.797 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/58 2023-10-30T13:45:24.797 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.798 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-10-30T13:45:24.799 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/59 2023-10-30T13:45:24.799 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.801 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-10-30T13:45:24.802 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/60 2023-10-30T13:45:24.802 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.803 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-10-30T13:45:24.804 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/61 2023-10-30T13:45:24.804 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.805 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-10-30T13:45:24.806 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/62 2023-10-30T13:45:24.806 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.808 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-10-30T13:45:24.809 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/63 2023-10-30T13:45:24.809 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.810 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-10-30T13:45:24.811 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/64 2023-10-30T13:45:24.811 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.812 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-10-30T13:45:24.813 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/65 2023-10-30T13:45:24.813 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.815 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-10-30T13:45:24.816 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/66 2023-10-30T13:45:24.816 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.817 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-10-30T13:45:24.818 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/67 2023-10-30T13:45:24.818 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.819 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-10-30T13:45:24.820 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/68 2023-10-30T13:45:24.821 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.822 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-10-30T13:45:24.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/69 2023-10-30T13:45:24.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.824 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-10-30T13:45:24.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/70 2023-10-30T13:45:24.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.826 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-10-30T13:45:24.827 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/71 2023-10-30T13:45:24.828 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.829 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-10-30T13:45:24.830 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/72 2023-10-30T13:45:24.830 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.831 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-10-30T13:45:24.832 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/73 2023-10-30T13:45:24.832 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.833 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-10-30T13:45:24.834 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/74 2023-10-30T13:45:24.835 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.836 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-10-30T13:45:24.836 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/75 2023-10-30T13:45:24.837 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.838 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-10-30T13:45:24.839 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/76 2023-10-30T13:45:24.839 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.840 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-10-30T13:45:24.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/77 2023-10-30T13:45:24.842 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.843 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-10-30T13:45:24.844 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/78 2023-10-30T13:45:24.844 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.845 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-10-30T13:45:24.846 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/79 2023-10-30T13:45:24.846 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.847 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-10-30T13:45:24.848 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/80 2023-10-30T13:45:24.849 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.850 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-10-30T13:45:24.851 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/81 2023-10-30T13:45:24.851 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.852 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-10-30T13:45:24.853 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/82 2023-10-30T13:45:24.853 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.854 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-10-30T13:45:24.855 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/83 2023-10-30T13:45:24.856 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.857 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-10-30T13:45:24.858 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/84 2023-10-30T13:45:24.858 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.859 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-10-30T13:45:24.861 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/85 2023-10-30T13:45:24.861 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.862 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-10-30T13:45:24.863 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/86 2023-10-30T13:45:24.863 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.864 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-10-30T13:45:24.865 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/87 2023-10-30T13:45:24.865 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.866 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-10-30T13:45:24.867 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/88 2023-10-30T13:45:24.867 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.869 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-10-30T13:45:24.870 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/89 2023-10-30T13:45:24.870 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.871 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-10-30T13:45:24.872 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/90 2023-10-30T13:45:24.872 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.873 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-10-30T13:45:24.874 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/91 2023-10-30T13:45:24.874 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.876 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-10-30T13:45:24.876 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/92 2023-10-30T13:45:24.877 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.878 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-10-30T13:45:24.879 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/93 2023-10-30T13:45:24.879 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.880 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-10-30T13:45:24.881 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/94 2023-10-30T13:45:24.881 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.882 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-10-30T13:45:24.883 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/95 2023-10-30T13:45:24.884 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.885 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-10-30T13:45:24.886 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/96 2023-10-30T13:45:24.886 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.887 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-10-30T13:45:24.888 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/97 2023-10-30T13:45:24.888 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.889 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-10-30T13:45:24.890 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/98 2023-10-30T13:45:24.891 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.892 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-10-30T13:45:24.893 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/99 2023-10-30T13:45:24.893 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.894 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-10-30T13:45:24.895 INFO:tasks.workunit.client.0.smithi031.stderr:+ stat .snap/foo/100 2023-10-30T13:45:24.895 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep No such file 2023-10-30T13:45:24.897 INFO:tasks.workunit.client.0.smithi031.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-10-30T13:45:24.897 INFO:tasks.workunit.client.0.smithi031.stderr:+ rmdir .snap/foo 2023-10-30T13:45:24.904 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2023-10-30T13:45:24.904 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2023-10-30T13:45:24.905 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:45:24.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:45:24.928 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-30T13:45:24.929 DEBUG:teuthology.orchestra.run.smithi031:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-30T13:45:24.987 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -e 2023-10-30T13:45:24.987 INFO:tasks.workunit.client.0.smithi031.stderr:+ sudo git config --global http.sslVerify false 2023-10-30T13:45:25.002 INFO:tasks.workunit.client.0.smithi031.stderr:+ sudo git config --global http.postBuffer 1048576000 2023-10-30T13:45:25.016 INFO:tasks.workunit.client.0.smithi031.stderr:+ retried=false 2023-10-30T13:45:25.016 INFO:tasks.workunit.client.0.smithi031.stderr:+ trap -- retry EXIT 2023-10-30T13:45:25.016 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm -rf ceph 2023-10-30T13:45:25.017 INFO:tasks.workunit.client.0.smithi031.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-30T13:45:25.021 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into 'ceph'... 2023-10-30T13:45:26.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:26.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:56.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:56.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:26.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:26.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:53.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc58bb3e640 1 Processor -- start 2023-10-30T13:46:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc58bb3e640 1 -- start start 2023-10-30T13:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc58bb3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584159ec0 0x7fc5840c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc58bb3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58415cda0 con 0x7fc5840c5f80 2023-10-30T13:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc58bb3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58406f490 con 0x7fc5840c5ba0 2023-10-30T13:46:53.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc58bb3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58415d990 con 0x7fc584159ec0 2023-10-30T13:46:53.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc5890d9640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc5840c5f80 0x7fc5841597b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:47402/0 (socket says 172.21.15.31:47402) 2023-10-30T13:46:53.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc5890d9640 1 -- 172.21.15.31:0/346815626 learned_addr learned my addr 172.21.15.31:0/346815626 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:46:53.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.945+0000 7fc58a0db640 1 --2- 172.21.15.31:0/346815626 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584159ec0 0x7fc5840c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:53.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023816244 0 0) 0x7fc58415cda0 con 0x7fc5840c5f80 2023-10-30T13:46:53.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc570003530 con 0x7fc5840c5f80 2023-10-30T13:46:53.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307627526 0 0) 0x7fc58406f490 con 0x7fc5840c5ba0 2023-10-30T13:46:53.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58415cda0 con 0x7fc5840c5ba0 2023-10-30T13:46:53.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2757381164 0 0) 0x7fc570003530 con 0x7fc5840c5f80 2023-10-30T13:46:53.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc58a0db640 1 -- 172.21.15.31:0/346815626 >> v1:172.21.15.33:6789/0 conn(0x7fc5840c5ba0 legacy=0x7fc584069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:53.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc58a0db640 1 -- 172.21.15.31:0/346815626 >> v1:172.21.15.31:6789/0 conn(0x7fc5840c5f80 legacy=0x7fc5841597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:53.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.949+0000 7fc58a0db640 1 -- 172.21.15.31:0/346815626 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58415cff0 con 0x7fc584159ec0 2023-10-30T13:46:53.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58a0db640 1 --2- 172.21.15.31:0/346815626 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584159ec0 0x7fc5840c70f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009dc0 tx=0x7fc57c01c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f947bb8bc3ce787 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:53.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574001680 con 0x7fc584159ec0 2023-10-30T13:46:53.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c0a64e0 con 0x7fc584159ec0 2023-10-30T13:46:53.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc5888d8640 1 -- 172.21.15.31:0/346815626 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0a6870 con 0x7fc584159ec0 2023-10-30T13:46:53.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58bb3e640 1 -- 172.21.15.31:0/346815626 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584159ec0 msgr2=0x7fc5840c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:53.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/346815626 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584159ec0 0x7fc5840c70f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009dc0 tx=0x7fc57c01c890 comp rx=0 tx=0).stop 2023-10-30T13:46:53.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58bb3e640 1 -- 172.21.15.31:0/346815626 shutdown_connections 2023-10-30T13:46:53.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/346815626 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584159ec0 0x7fc5840c70f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:53.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58bb3e640 1 -- 172.21.15.31:0/346815626 >> 172.21.15.31:0/346815626 conn(0x7fc58406afa0 msgr2=0x7fc58406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:53.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58bb3e640 1 -- 172.21.15.31:0/346815626 shutdown_connections 2023-10-30T13:46:53.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.953+0000 7fc58bb3e640 1 -- 172.21.15.31:0/346815626 wait complete. 2023-10-30T13:46:53.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 Processor -- start 2023-10-30T13:46:53.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 -- start start 2023-10-30T13:46:53.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5840c5ba0 0x7fc5841f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:53.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5840c5f80 0x7fc58414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:53.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584151490 0x7fc584152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:53.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584152930 con 0x7fc5840c5ba0 2023-10-30T13:46:53.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5841539b0 con 0x7fc5840c5f80 2023-10-30T13:46:53.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58bb3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5841f5f60 con 0x7fc584151490 2023-10-30T13:46:53.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5840c5ba0 0x7fc5841f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:53.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5840c5ba0 0x7fc5841f5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39124/0 (socket says 172.21.15.31:39124) 2023-10-30T13:46:53.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 -- 172.21.15.31:0/2154211618 learned_addr learned my addr 172.21.15.31:0/2154211618 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:46:53.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc58a0db640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584151490 0x7fc584152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:53.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5890d9640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5840c5f80 0x7fc58414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 -- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584151490 msgr2=0x7fc584152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584151490 0x7fc584152dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 -- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5840c5f80 msgr2=0x7fc58414fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5840c5f80 0x7fc58414fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.957+0000 7fc5898da640 1 -- 172.21.15.31:0/2154211618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c009a70 con 0x7fc5840c5ba0 2023-10-30T13:46:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc5898da640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5840c5ba0 0x7fc5841f5a20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c430 tx=0x7fc58000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc57a7fc640 1 -- 172.21.15.31:0/2154211618 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5800052d0 con 0x7fc5840c5ba0 2023-10-30T13:46:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc57a7fc640 1 -- 172.21.15.31:0/2154211618 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc580005450 con 0x7fc5840c5ba0 2023-10-30T13:46:53.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc57a7fc640 1 -- 172.21.15.31:0/2154211618 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580003840 con 0x7fc5840c5ba0 2023-10-30T13:46:53.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc584151940 con 0x7fc5840c5ba0 2023-10-30T13:46:53.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc584151f00 con 0x7fc5840c5ba0 2023-10-30T13:46:53.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58404e170 con 0x7fc5840c5ba0 2023-10-30T13:46:53.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc57a7fc640 1 -- 172.21.15.31:0/2154211618 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5800039a0 con 0x7fc5840c5ba0 2023-10-30T13:46:53.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc57a7fc640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5680664d0 0x7fc568068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.961+0000 7fc57a7fc640 1 -- 172.21.15.31:0/2154211618 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fc5800836d0 con 0x7fc5840c5ba0 2023-10-30T13:46:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.965+0000 7fc5890d9640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5680664d0 0x7fc568068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.965+0000 7fc5890d9640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5680664d0 0x7fc568068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc58415cae0 tx=0x7fc574001d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:53.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:53.969+0000 7fc57a7fc640 1 -- 172.21.15.31:0/2154211618 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc580050900 con 0x7fc5840c5ba0 2023-10-30T13:46:54.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.197+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5840c7c00 con 0x7fc5840c5ba0 2023-10-30T13:46:54.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.197+0000 7fc57a7fc640 1 -- 172.21.15.31:0/2154211618 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7fc580053f20 con 0x7fc5840c5ba0 2023-10-30T13:46:54.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:46:54.200 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:54.200 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:46:54.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5680664d0 msgr2=0x7fc568068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5680664d0 0x7fc568068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc58415cae0 tx=0x7fc574001d80 comp rx=0 tx=0).stop 2023-10-30T13:46:54.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5840c5ba0 msgr2=0x7fc5841f5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5840c5ba0 0x7fc5841f5a20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c430 tx=0x7fc58000c8f0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 shutdown_connections 2023-10-30T13:46:54.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5680664d0 0x7fc568068990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc584151490 0x7fc584152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5840c5f80 0x7fc58414fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 --2- 172.21.15.31:0/2154211618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5840c5ba0 0x7fc5841f5a20 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 >> 172.21.15.31:0/2154211618 conn(0x7fc58406afa0 msgr2=0x7fc5840c8700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:54.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 shutdown_connections 2023-10-30T13:46:54.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.201+0000 7fc58bb3e640 1 -- 172.21.15.31:0/2154211618 wait complete. 2023-10-30T13:46:54.221 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:46:54.222 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:46:54.222 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:46:54.245 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:46:54.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:54.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdfd3e640 1 Processor -- start 2023-10-30T13:46:54.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdfd3e640 1 -- start start 2023-10-30T13:46:54.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdfd3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8fd80c4a10 0x7f8fd815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdfd3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd80c42a0 0x7f8fd80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdfd3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd8159f20 con 0x7f8fd8153c80 2023-10-30T13:46:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdfd3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd815dec0 con 0x7f8fd80c4a10 2023-10-30T13:46:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdfd3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd81599a0 con 0x7f8fd80c42a0 2023-10-30T13:46:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fddada640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8fd8153c80 0x7f8fd815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:47412/0 (socket says 172.21.15.31:47412) 2023-10-30T13:46:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fddada640 1 -- 172.21.15.31:0/2333286607 learned_addr learned my addr 172.21.15.31:0/2333286607 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:46:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdd2d9640 1 --2- 172.21.15.31:0/2333286607 >> v2:172.21.15.31:3300/0 conn(0x7f8fd80c4a10 0x7f8fd815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fde2db640 1 --2- 172.21.15.31:0/2333286607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd80c42a0 0x7f8fd80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdcad8640 1 -- 172.21.15.31:0/2333286607 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635040187 0 0) 0x7f8fd8159f20 con 0x7f8fd8153c80 2023-10-30T13:46:54.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdcad8640 1 -- 172.21.15.31:0/2333286607 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc4003530 con 0x7f8fd8153c80 2023-10-30T13:46:54.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdd2d9640 1 -- 172.21.15.31:0/2333286607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd80c42a0 msgr2=0x7f8fd80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:54.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdd2d9640 1 --2- 172.21.15.31:0/2333286607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd80c42a0 0x7f8fd80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdd2d9640 1 -- 172.21.15.31:0/2333286607 >> v1:172.21.15.31:6789/0 conn(0x7f8fd8153c80 legacy=0x7f8fd815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.432+0000 7f8fdd2d9640 1 -- 172.21.15.31:0/2333286607 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd815d620 con 0x7f8fd80c4a10 2023-10-30T13:46:54.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdd2d9640 1 --2- 172.21.15.31:0/2333286607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c4a10 0x7f8fd815cf90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80097c0 tx=0x7f8fc801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8d9d5462730e13 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:54.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdcad8640 1 -- 172.21.15.31:0/2333286607 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80a8970 con 0x7f8fd80c4a10 2023-10-30T13:46:54.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdcad8640 1 -- 172.21.15.31:0/2333286607 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc80a8ad0 con 0x7f8fd80c4a10 2023-10-30T13:46:54.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdcad8640 1 -- 172.21.15.31:0/2333286607 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80a8e40 con 0x7f8fd80c4a10 2023-10-30T13:46:54.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/2333286607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c4a10 msgr2=0x7f8fd815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/2333286607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c4a10 0x7f8fd815cf90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80097c0 tx=0x7f8fc801ec70 comp rx=0 tx=0).stop 2023-10-30T13:46:54.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/2333286607 shutdown_connections 2023-10-30T13:46:54.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/2333286607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd80c42a0 0x7f8fd80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/2333286607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c4a10 0x7f8fd815cf90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/2333286607 >> 172.21.15.31:0/2333286607 conn(0x7f8fd80715c0 msgr2=0x7f8fd80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:54.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/2333286607 shutdown_connections 2023-10-30T13:46:54.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.436+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/2333286607 wait complete. 2023-10-30T13:46:54.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 Processor -- start 2023-10-30T13:46:54.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 -- start start 2023-10-30T13:46:54.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c42a0 0x7f8fd81f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:54.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fd80c4a10 0x7f8fd80c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:54.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd8153c80 0x7f8fd815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:54.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd81ef4b0 con 0x7f8fd80c42a0 2023-10-30T13:46:54.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd80c5840 con 0x7f8fd80c4a10 2023-10-30T13:46:54.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd814adf0 con 0x7f8fd8153c80 2023-10-30T13:46:54.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c42a0 0x7f8fd81f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:54.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c42a0 0x7f8fd81f2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39140/0 (socket says 172.21.15.31:39140) 2023-10-30T13:46:54.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 -- 172.21.15.31:0/357241674 learned_addr learned my addr 172.21.15.31:0/357241674 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:46:54.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fde2db640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd8153c80 0x7f8fd815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:54.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdd2d9640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fd80c4a10 0x7f8fd80c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:54.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 -- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd8153c80 msgr2=0x7f8fd815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd8153c80 0x7f8fd815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 -- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fd80c4a10 msgr2=0x7f8fd80c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fd80c4a10 0x7f8fd80c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 -- 172.21.15.31:0/357241674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc8009420 con 0x7f8fd80c42a0 2023-10-30T13:46:54.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdd2d9640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fd80c4a10 0x7f8fd80c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:54.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fddada640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c42a0 0x7f8fd81f2930 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c3c0 tx=0x7f8fd400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:54.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fce7fc640 1 -- 172.21.15.31:0/357241674 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4005cf0 con 0x7f8fd80c42a0 2023-10-30T13:46:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fce7fc640 1 -- 172.21.15.31:0/357241674 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd4005e50 con 0x7f8fd80c42a0 2023-10-30T13:46:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fce7fc640 1 -- 172.21.15.31:0/357241674 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd40072f0 con 0x7f8fd80c42a0 2023-10-30T13:46:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd814b0d0 con 0x7f8fd80c42a0 2023-10-30T13:46:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.440+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd81eee60 con 0x7f8fd80c42a0 2023-10-30T13:46:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.444+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd804e170 con 0x7f8fd80c42a0 2023-10-30T13:46:54.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.444+0000 7f8fce7fc640 1 -- 172.21.15.31:0/357241674 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8fd4007450 con 0x7f8fd80c42a0 2023-10-30T13:46:54.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.444+0000 7f8fce7fc640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fbc066470 0x7f8fbc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:54.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.444+0000 7f8fce7fc640 1 -- 172.21.15.31:0/357241674 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f8fd4083770 con 0x7f8fd80c42a0 2023-10-30T13:46:54.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.444+0000 7f8fdd2d9640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fbc066470 0x7f8fbc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:54.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.444+0000 7f8fdd2d9640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fbc066470 0x7f8fbc068930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80b2000 tx=0x7f8fc80aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.448+0000 7f8fce7fc640 1 -- 172.21.15.31:0/357241674 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8fd40508f0 con 0x7f8fd80c42a0 2023-10-30T13:46:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fd80cad40 con 0x7f8fd80c42a0 2023-10-30T13:46:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fce7fc640 1 -- 172.21.15.31:0/357241674 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f8fd4017090 con 0x7f8fd80c42a0 2023-10-30T13:46:54.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:46:54.645 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:54.645 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:46:54.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fbc066470 msgr2=0x7f8fbc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fbc066470 0x7f8fbc068930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80b2000 tx=0x7f8fc80aa070 comp rx=0 tx=0).stop 2023-10-30T13:46:54.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c42a0 msgr2=0x7f8fd81f2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:54.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c42a0 0x7f8fd81f2930 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c3c0 tx=0x7f8fd400c880 comp rx=0 tx=0).stop 2023-10-30T13:46:54.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 shutdown_connections 2023-10-30T13:46:54.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fbc066470 0x7f8fbc068930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8fd8153c80 0x7f8fd815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fd80c4a10 0x7f8fd80c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 --2- 172.21.15.31:0/357241674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fd80c42a0 0x7f8fd81f2930 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:54.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 >> 172.21.15.31:0/357241674 conn(0x7f8fd80715c0 msgr2=0x7f8fd80cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:54.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 shutdown_connections 2023-10-30T13:46:54.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:54.644+0000 7f8fdfd3e640 1 -- 172.21.15.31:0/357241674 wait complete. 2023-10-30T13:46:54.665 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:46:56.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:56.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:56.720 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:56.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9fde46640 1 Processor -- start 2023-10-30T13:46:56.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9fde46640 1 -- start start 2023-10-30T13:46:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9fde46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f8153c80 0x7fa9f815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9fde46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa9f80c42a0 0x7fa9f80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9fde46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f8159f20 con 0x7fa9f80c4a10 2023-10-30T13:46:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9fde46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f815dec0 con 0x7fa9f80c42a0 2023-10-30T13:46:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9fde46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f81599a0 con 0x7fa9f8153c80 2023-10-30T13:46:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f77fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f8153c80 0x7fa9f815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f7fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa9f80c42a0 0x7fa9f80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f7fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa9f80c42a0 0x7fa9f80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:37390/0 (socket says 172.21.15.31:37390) 2023-10-30T13:46:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f7fff640 1 -- 172.21.15.31:0/1986459742 learned_addr learned my addr 172.21.15.31:0/1986459742 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:46:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f77fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f8153c80 0x7fa9f815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:55484/0 (socket says 172.21.15.31:55484) 2023-10-30T13:46:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f7fff640 1 -- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f8153c80 msgr2=0x7fa9f815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f7fff640 1 --2- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f8153c80 0x7fa9f815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f7fff640 1 -- 172.21.15.31:0/1986459742 >> v1:172.21.15.33:6789/0 conn(0x7fa9f80c4a10 legacy=0x7fa9f815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f7fff640 1 -- 172.21.15.31:0/1986459742 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f815d620 con 0x7fa9f80c42a0 2023-10-30T13:46:56.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.812+0000 7fa9f77fe640 1 --2- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f8153c80 0x7fa9f815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:46:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9f7fff640 1 --2- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f80c42a0 0x7fa9f80ca1e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009e50 tx=0x7fa9f001f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee483c72d7d418cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9f67fc640 1 -- 172.21.15.31:0/1986459742 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00af400 con 0x7fa9f80c42a0 2023-10-30T13:46:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9f67fc640 1 -- 172.21.15.31:0/1986459742 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f00ab050 con 0x7fa9f80c42a0 2023-10-30T13:46:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9f67fc640 1 -- 172.21.15.31:0/1986459742 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00af780 con 0x7fa9f80c42a0 2023-10-30T13:46:56.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9fde46640 1 -- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f80c42a0 msgr2=0x7fa9f80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:56.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9fde46640 1 --2- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f80c42a0 0x7fa9f80ca1e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009e50 tx=0x7fa9f001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:46:56.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9fde46640 1 -- 172.21.15.31:0/1986459742 shutdown_connections 2023-10-30T13:46:56.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9fde46640 1 --2- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f80c42a0 0x7fa9f80ca1e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:56.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9fde46640 1 --2- 172.21.15.31:0/1986459742 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f8153c80 0x7fa9f815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:56.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.816+0000 7fa9fde46640 1 -- 172.21.15.31:0/1986459742 >> 172.21.15.31:0/1986459742 conn(0x7fa9f80715c0 msgr2=0x7fa9f80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:56.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 -- 172.21.15.31:0/1986459742 shutdown_connections 2023-10-30T13:46:56.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 -- 172.21.15.31:0/1986459742 wait complete. 2023-10-30T13:46:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 Processor -- start 2023-10-30T13:46:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 -- start start 2023-10-30T13:46:56.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9f80c42a0 0x7fa9f814b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:56.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f80c4a10 0x7fa9f80c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:56.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f8153c80 0x7fa9f8149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:56.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f814bbe0 con 0x7fa9f80c42a0 2023-10-30T13:46:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f81f1ee0 con 0x7fa9f8153c80 2023-10-30T13:46:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9fde46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f81f2180 con 0x7fa9f80c4a10 2023-10-30T13:46:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9f80c42a0 0x7fa9f814b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9f80c42a0 0x7fa9f814b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55208/0 (socket says 172.21.15.31:55208) 2023-10-30T13:46:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 -- 172.21.15.31:0/3871694127 learned_addr learned my addr 172.21.15.31:0/3871694127 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:46:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f6ffd640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f80c4a10 0x7fa9f80c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f7fff640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f8153c80 0x7fa9f8149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 -- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f80c4a10 msgr2=0x7fa9f80c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f80c4a10 0x7fa9f80c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 -- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f8153c80 msgr2=0x7fa9f8149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f8153c80 0x7fa9f8149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 -- 172.21.15.31:0/3871694127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8009420 con 0x7fa9f80c42a0 2023-10-30T13:46:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f6ffd640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f80c4a10 0x7fa9f80c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.820+0000 7fa9f77fe640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9f80c42a0 0x7fa9f814b6a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e980 tx=0x7fa9e800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9dffff640 1 -- 172.21.15.31:0/3871694127 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800ca00 con 0x7fa9f80c42a0 2023-10-30T13:46:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9dffff640 1 -- 172.21.15.31:0/3871694127 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e800cb60 con 0x7fa9f80c42a0 2023-10-30T13:46:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9dffff640 1 -- 172.21.15.31:0/3871694127 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8013470 con 0x7fa9f80c42a0 2023-10-30T13:46:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f0009b30 con 0x7fa9f80c42a0 2023-10-30T13:46:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f81f2740 con 0x7fa9f80c42a0 2023-10-30T13:46:56.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f804e170 con 0x7fa9f80c42a0 2023-10-30T13:46:56.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9dffff640 1 -- 172.21.15.31:0/3871694127 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9e80135d0 con 0x7fa9f80c42a0 2023-10-30T13:46:56.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9dffff640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa9e40664d0 0x7fa9e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:56.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9dffff640 1 -- 172.21.15.31:0/3871694127 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fa9e8084490 con 0x7fa9f80c42a0 2023-10-30T13:46:56.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.824+0000 7fa9f6ffd640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa9e40664d0 0x7fa9e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:56.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.828+0000 7fa9f6ffd640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa9e40664d0 0x7fa9e4068990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa9f814e4f0 tx=0x7fa9ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:56.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:56.828+0000 7fa9dffff640 1 -- 172.21.15.31:0/3871694127 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9e8051580 con 0x7fa9f80c42a0 2023-10-30T13:46:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9f8159b80 con 0x7fa9f80c42a0 2023-10-30T13:46:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9dffff640 1 -- 172.21.15.31:0/3871694127 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7fa9e8054ba0 con 0x7fa9f80c42a0 2023-10-30T13:46:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:46:57.021 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:46:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa9e40664d0 msgr2=0x7fa9e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa9e40664d0 0x7fa9e4068990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa9f814e4f0 tx=0x7fa9ec009040 comp rx=0 tx=0).stop 2023-10-30T13:46:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9f80c42a0 msgr2=0x7fa9f814b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9f80c42a0 0x7fa9f814b6a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e980 tx=0x7fa9e800ee40 comp rx=0 tx=0).stop 2023-10-30T13:46:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 shutdown_connections 2023-10-30T13:46:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa9e40664d0 0x7fa9e4068990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9f8153c80 0x7fa9f8149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa9f80c4a10 0x7fa9f80c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 --2- 172.21.15.31:0/3871694127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9f80c42a0 0x7fa9f814b6a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:57.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 >> 172.21.15.31:0/3871694127 conn(0x7fa9f80715c0 msgr2=0x7fa9f8157c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:57.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 shutdown_connections 2023-10-30T13:46:57.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:57.020+0000 7fa9fde46640 1 -- 172.21.15.31:0/3871694127 wait complete. 2023-10-30T13:46:57.042 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:46:59.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:59.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0163a09640 1 Processor -- start 2023-10-30T13:46:59.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0163a09640 1 -- start start 2023-10-30T13:46:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0163a09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0c8060 0x7f015c06f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0163a09640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f015c0cb8b0 0x7f015c15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0163a09640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c0ca180 con 0x7f015c0ca9b0 2023-10-30T13:46:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0163a09640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c150d00 con 0x7f015c0cb8b0 2023-10-30T13:46:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0163a09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0ca740 con 0x7f015c0c8060 2023-10-30T13:46:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0161fa6640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f015c0ca9b0 0x7f015c0c8730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52740/0 (socket says 172.21.15.31:52740) 2023-10-30T13:46:59.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0161fa6640 1 -- 172.21.15.31:0/3778456675 learned_addr learned my addr 172.21.15.31:0/3778456675 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:46:59.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f01617a5640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0c8060 0x7f015c06f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0160fa4640 1 --2- 172.21.15.31:0/3778456675 >> v2:172.21.15.33:3300/0 conn(0x7f015c0cb8b0 0x7f015c15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828067506 0 0) 0x7f015c0ca180 con 0x7f015c0ca9b0 2023-10-30T13:46:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0148005740 con 0x7f015c0ca9b0 2023-10-30T13:46:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19497316 0 0) 0x7f0148005740 con 0x7f015c0ca9b0 2023-10-30T13:46:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0c8060 msgr2=0x7f015c06f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0c8060 0x7f015c06f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 >> v2:172.21.15.33:3300/0 conn(0x7f015c0cb8b0 msgr2=0x7f015c15a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:59.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 --2- 172.21.15.31:0/3778456675 >> v2:172.21.15.33:3300/0 conn(0x7f015c0cb8b0 0x7f015c15a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c0ca460 con 0x7f015c0ca9b0 2023-10-30T13:46:59.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f0154001a00 con 0x7f015c0ca9b0 2023-10-30T13:46:59.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 >> v1:172.21.15.31:6789/0 conn(0x7f015c0ca9b0 legacy=0x7f015c0c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:59.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0148005b90 0x7f0148005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0148006580 0x7f0148006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01480070d0 0x7f014800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c150d00 con 0x7f0148006580 2023-10-30T13:46:59.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0ca740 con 0x7f01480070d0 2023-10-30T13:46:59.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014800e0e0 con 0x7f0148005b90 2023-10-30T13:46:59.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0160fa4640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0148006580 0x7f0148006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f01617a5640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0148005b90 0x7f0148005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.200+0000 7f0161fa6640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01480070d0 0x7f014800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0160fa4640 1 -- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0148005b90 msgr2=0x7f0148005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:59.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0160fa4640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0148005b90 0x7f0148005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0160fa4640 1 -- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01480070d0 msgr2=0x7f014800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:59.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0160fa4640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01480070d0 0x7f014800dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0160fa4640 1 -- 172.21.15.31:0/3778456675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0148005740 con 0x7f0148006580 2023-10-30T13:46:59.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0160fa4640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0148006580 0x7f0148006a60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f014c009ae0 tx=0x7f014c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0fc28cad1dc8ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:59.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c0a5680 con 0x7f0148006580 2023-10-30T13:46:59.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c0a57e0 con 0x7f0148006580 2023-10-30T13:46:59.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0153fff640 1 -- 172.21.15.31:0/3778456675 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c0a5b90 con 0x7f0148006580 2023-10-30T13:46:59.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0163a09640 1 -- 172.21.15.31:0/3778456675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0148006580 msgr2=0x7f0148006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:59.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0163a09640 1 --2- 172.21.15.31:0/3778456675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0148006580 0x7f0148006a60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f014c009ae0 tx=0x7f014c01ca90 comp rx=0 tx=0).stop 2023-10-30T13:46:59.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0161fa6640 1 -- 172.21.15.31:0/3778456675 reap_dead start 2023-10-30T13:46:59.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0163a09640 1 -- 172.21.15.31:0/3778456675 shutdown_connections 2023-10-30T13:46:59.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.204+0000 7f0163a09640 1 -- 172.21.15.31:0/3778456675 >> 172.21.15.31:0/3778456675 conn(0x7f015c06afa0 msgr2=0x7f015c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:59.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 -- 172.21.15.31:0/3778456675 shutdown_connections 2023-10-30T13:46:59.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 -- 172.21.15.31:0/3778456675 wait complete. 2023-10-30T13:46:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 Processor -- start 2023-10-30T13:46:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 -- start start 2023-10-30T13:46:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148006580 0x7f015c156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0b1b00 0x7f015c071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f015c1eff00 0x7f015c1f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c153730 con 0x7f015c1eff00 2023-10-30T13:46:59.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1550f0 con 0x7f0148006580 2023-10-30T13:46:59.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0163a09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1f08c0 con 0x7f015c0b1b00 2023-10-30T13:46:59.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f015c1eff00 0x7f015c1f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f015c1eff00 0x7f015c1f0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55220/0 (socket says 172.21.15.31:55220) 2023-10-30T13:46:59.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 -- 172.21.15.31:0/3630123 learned_addr learned my addr 172.21.15.31:0/3630123 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:46:59.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f01617a5640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148006580 0x7f015c156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0160fa4640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0b1b00 0x7f015c071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 -- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0b1b00 msgr2=0x7f015c071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:59.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0b1b00 0x7f015c071900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 -- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148006580 msgr2=0x7f015c156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148006580 0x7f015c156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 -- 172.21.15.31:0/3630123 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c154200 con 0x7f015c1eff00 2023-10-30T13:46:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.208+0000 7f0161fa6640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f015c1eff00 0x7f015c1f0380 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f015c06d720 tx=0x7f0154012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0151ffb640 1 -- 172.21.15.31:0/3630123 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015401b070 con 0x7f015c1eff00 2023-10-30T13:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0151ffb640 1 -- 172.21.15.31:0/3630123 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154001260 con 0x7f015c1eff00 2023-10-30T13:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c1544e0 con 0x7f015c1eff00 2023-10-30T13:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0151ffb640 1 -- 172.21.15.31:0/3630123 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154007b20 con 0x7f015c1eff00 2023-10-30T13:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c154aa0 con 0x7f015c1eff00 2023-10-30T13:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c06e790 con 0x7f015c1eff00 2023-10-30T13:46:59.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0151ffb640 1 -- 172.21.15.31:0/3630123 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0154016660 con 0x7f015c1eff00 2023-10-30T13:46:59.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0151ffb640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01400664a0 0x7f0140068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:59.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f0151ffb640 1 -- 172.21.15.31:0/3630123 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f0154087af0 con 0x7f015c1eff00 2023-10-30T13:46:59.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.212+0000 7f01617a5640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01400664a0 0x7f0140068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:59.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.216+0000 7f01617a5640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01400664a0 0x7f0140068960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f015c0c9ea0 tx=0x7f0158009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:59.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.216+0000 7f0151ffb640 1 -- 172.21.15.31:0/3630123 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f015c06e790 con 0x7f015c1eff00 2023-10-30T13:46:59.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.408+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f015c06e790 con 0x7f015c1eff00 2023-10-30T13:46:59.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.408+0000 7f0151ffb640 1 -- 172.21.15.31:0/3630123 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f015c06e790 con 0x7f015c1eff00 2023-10-30T13:46:59.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:46:59.410 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:59.411 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:46:59.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01400664a0 msgr2=0x7f0140068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:59.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01400664a0 0x7f0140068960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f015c0c9ea0 tx=0x7f0158009190 comp rx=0 tx=0).stop 2023-10-30T13:46:59.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f015c1eff00 msgr2=0x7f015c1f0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:59.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f015c1eff00 0x7f015c1f0380 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f015c06d720 tx=0x7f0154012920 comp rx=0 tx=0).stop 2023-10-30T13:46:59.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 shutdown_connections 2023-10-30T13:46:59.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01400664a0 0x7f0140068960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f015c1eff00 0x7f015c1f0380 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f015c0b1b00 0x7f015c071900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 --2- 172.21.15.31:0/3630123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148006580 0x7f015c156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:59.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 >> 172.21.15.31:0/3630123 conn(0x7f015c06afa0 msgr2=0x7f015c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:59.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 shutdown_connections 2023-10-30T13:46:59.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:46:59.412+0000 7f0163a09640 1 -- 172.21.15.31:0/3630123 wait complete. 2023-10-30T13:46:59.432 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:47:01.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:01.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04c30f640 1 Processor -- start 2023-10-30T13:47:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04c30f640 1 -- start start 2023-10-30T13:47:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04c30f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff04406bd10 0x7ff0440c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04c30f640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440c5030 con 0x7ff044151c20 2023-10-30T13:47:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04c30f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff044157830 con 0x7ff044151840 2023-10-30T13:47:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04c30f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff044157b10 con 0x7ff04406bd10 2023-10-30T13:47:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff0498aa640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff044151c20 0x7ff04406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52750/0 (socket says 172.21.15.31:52750) 2023-10-30T13:47:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff0498aa640 1 -- 172.21.15.31:0/1464093813 learned_addr learned my addr 172.21.15.31:0/1464093813 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04a8ac640 1 --2- 172.21.15.31:0/1464093813 >> v2:172.21.15.31:3300/0 conn(0x7ff04406bd10 0x7ff0440c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff0490a9640 1 -- 172.21.15.31:0/1464093813 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64269927 0 0) 0x7ff0440c5030 con 0x7ff044151c20 2023-10-30T13:47:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff0490a9640 1 -- 172.21.15.31:0/1464093813 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff030003530 con 0x7ff044151c20 2023-10-30T13:47:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04a8ac640 1 -- 172.21.15.31:0/1464093813 >> v1:172.21.15.33:6789/0 conn(0x7ff044151840 legacy=0x7ff044072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04a8ac640 1 -- 172.21.15.31:0/1464093813 >> v1:172.21.15.31:6789/0 conn(0x7ff044151c20 legacy=0x7ff04406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.588+0000 7ff04a8ac640 1 -- 172.21.15.31:0/1464093813 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044157df0 con 0x7ff04406bd10 2023-10-30T13:47:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04a8ac640 1 --2- 172.21.15.31:0/1464093813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff0440c49a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009d90 tx=0x7ff03c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e00c0080e7e4993 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff0490a9640 1 -- 172.21.15.31:0/1464093813 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0af600 con 0x7ff04406bd10 2023-10-30T13:47:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff0490a9640 1 -- 172.21.15.31:0/1464093813 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c0ab050 con 0x7ff04406bd10 2023-10-30T13:47:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff0490a9640 1 -- 172.21.15.31:0/1464093813 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0af980 con 0x7ff04406bd10 2023-10-30T13:47:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04c30f640 1 -- 172.21.15.31:0/1464093813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 msgr2=0x7ff0440c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04c30f640 1 --2- 172.21.15.31:0/1464093813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff0440c49a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009d90 tx=0x7ff03c01f130 comp rx=0 tx=0).stop 2023-10-30T13:47:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04c30f640 1 -- 172.21.15.31:0/1464093813 shutdown_connections 2023-10-30T13:47:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04c30f640 1 --2- 172.21.15.31:0/1464093813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff0440c49a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04c30f640 1 -- 172.21.15.31:0/1464093813 >> 172.21.15.31:0/1464093813 conn(0x7ff0440cac70 msgr2=0x7ff0440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04c30f640 1 -- 172.21.15.31:0/1464093813 shutdown_connections 2023-10-30T13:47:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.592+0000 7ff04c30f640 1 -- 172.21.15.31:0/1464093813 wait complete. 2023-10-30T13:47:01.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 Processor -- start 2023-10-30T13:47:01.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 -- start start 2023-10-30T13:47:01.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff044155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:01.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff044151840 0x7ff04414e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:01.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff044151c20 0x7ff0441ec930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:01.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441e19a0 con 0x7ff04406bd10 2023-10-30T13:47:01.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441efc30 con 0x7ff044151c20 2023-10-30T13:47:01.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441efe80 con 0x7ff044151840 2023-10-30T13:47:01.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff044155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:01.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff044155590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55244/0 (socket says 172.21.15.31:55244) 2023-10-30T13:47:01.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 -- 172.21.15.31:0/4260524201 learned_addr learned my addr 172.21.15.31:0/4260524201 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:01.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff0498aa640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff044151840 0x7ff04414e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:01.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a8ac640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff044151c20 0x7ff0441ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:01.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 -- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff044151840 msgr2=0x7ff04414e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:01.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff044151840 0x7ff04414e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 -- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff044151c20 msgr2=0x7ff0441ec930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:01.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff044151c20 0x7ff0441ec930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 -- 172.21.15.31:0/4260524201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c009a70 con 0x7ff04406bd10 2023-10-30T13:47:01.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04a0ab640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff044155590 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c500 tx=0x7ff04000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:01.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff03affd640 1 -- 172.21.15.31:0/4260524201 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040005220 con 0x7ff04406bd10 2023-10-30T13:47:01.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff03affd640 1 -- 172.21.15.31:0/4260524201 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040005380 con 0x7ff04406bd10 2023-10-30T13:47:01.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff03affd640 1 -- 172.21.15.31:0/4260524201 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040007270 con 0x7ff04406bd10 2023-10-30T13:47:01.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0441f0160 con 0x7ff04406bd10 2023-10-30T13:47:01.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.596+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0441f0720 con 0x7ff04406bd10 2023-10-30T13:47:01.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.600+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0441ec6c0 con 0x7ff04406bd10 2023-10-30T13:47:01.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.600+0000 7ff03affd640 1 -- 172.21.15.31:0/4260524201 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0400073d0 con 0x7ff04406bd10 2023-10-30T13:47:01.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.600+0000 7ff03affd640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0280664a0 0x7ff028068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:01.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.600+0000 7ff03affd640 1 -- 172.21.15.31:0/4260524201 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7ff040083a60 con 0x7ff04406bd10 2023-10-30T13:47:01.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.600+0000 7ff0498aa640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0280664a0 0x7ff028068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:01.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.600+0000 7ff0498aa640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0280664a0 0x7ff028068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff0441e1d50 tx=0x7ff034009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.604+0000 7ff03affd640 1 -- 172.21.15.31:0/4260524201 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff040050bf0 con 0x7ff04406bd10 2023-10-30T13:47:01.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.792+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff04404e170 con 0x7ff04406bd10 2023-10-30T13:47:01.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff03affd640 1 -- 172.21.15.31:0/4260524201 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7ff040054210 con 0x7ff04406bd10 2023-10-30T13:47:01.796 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:01.796 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:01.796 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:47:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0280664a0 msgr2=0x7ff028068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0280664a0 0x7ff028068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff0441e1d50 tx=0x7ff034009040 comp rx=0 tx=0).stop 2023-10-30T13:47:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 msgr2=0x7ff044155590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff044155590 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c500 tx=0x7ff04000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 shutdown_connections 2023-10-30T13:47:01.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0280664a0 0x7ff028068960 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff044151c20 0x7ff0441ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff044151840 0x7ff04414e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 --2- 172.21.15.31:0/4260524201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff04406bd10 0x7ff044155590 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 >> 172.21.15.31:0/4260524201 conn(0x7ff0440cac70 msgr2=0x7ff0440c63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 shutdown_connections 2023-10-30T13:47:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:01.796+0000 7ff04c30f640 1 -- 172.21.15.31:0/4260524201 wait complete. 2023-10-30T13:47:01.817 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:47:03.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b72ea640 1 Processor -- start 2023-10-30T13:47:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b72ea640 1 -- start start 2023-10-30T13:47:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b72ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe7b006bb40 0x7fe7b0157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b72ea640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe7b006c420 0x7fe7b0157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b72ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b0071e00 con 0x7fe7b006b760 2023-10-30T13:47:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b72ea640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b0070020 con 0x7fe7b006c420 2023-10-30T13:47:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b72ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b0070f10 con 0x7fe7b006bb40 2023-10-30T13:47:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b5887640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe7b006c420 0x7fe7b0157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:03.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b5887640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe7b006c420 0x7fe7b0157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55250/0 (socket says 172.21.15.31:55250) 2023-10-30T13:47:03.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b5887640 1 -- 172.21.15.31:0/3636008436 learned_addr learned my addr 172.21.15.31:0/3636008436 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:03.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.972+0000 7fe7b4885640 1 --2- 172.21.15.31:0/3636008436 >> v2:172.21.15.33:3300/0 conn(0x7fe7b006bb40 0x7fe7b0157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:03.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b5887640 1 -- 172.21.15.31:0/3636008436 >> v2:172.21.15.33:3300/0 conn(0x7fe7b006bb40 msgr2=0x7fe7b0157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b5887640 1 --2- 172.21.15.31:0/3636008436 >> v2:172.21.15.33:3300/0 conn(0x7fe7b006bb40 0x7fe7b0157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b5887640 1 -- 172.21.15.31:0/3636008436 >> v1:172.21.15.33:6789/0 conn(0x7fe7b006b760 legacy=0x7fe7b015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b5887640 1 -- 172.21.15.31:0/3636008436 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00711f0 con 0x7fe7b006c420 2023-10-30T13:47:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b4885640 1 --2- 172.21.15.31:0/3636008436 >> v2:172.21.15.33:3300/0 conn(0x7fe7b006bb40 0x7fe7b0157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:03.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b5887640 1 --2- 172.21.15.31:0/3636008436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006c420 0x7fe7b0157f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe7b006fa40 tx=0x7fe7a801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c833420219a7444 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:03.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7a7fff640 1 -- 172.21.15.31:0/3636008436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80a5860 con 0x7fe7b006c420 2023-10-30T13:47:03.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7a7fff640 1 -- 172.21.15.31:0/3636008436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a80a59c0 con 0x7fe7b006c420 2023-10-30T13:47:03.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7a7fff640 1 -- 172.21.15.31:0/3636008436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80a5d50 con 0x7fe7b006c420 2023-10-30T13:47:03.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b72ea640 1 -- 172.21.15.31:0/3636008436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006c420 msgr2=0x7fe7b0157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/3636008436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006c420 0x7fe7b0157f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe7b006fa40 tx=0x7fe7a801c9a0 comp rx=0 tx=0).stop 2023-10-30T13:47:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b72ea640 1 -- 172.21.15.31:0/3636008436 shutdown_connections 2023-10-30T13:47:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/3636008436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006c420 0x7fe7b0157f60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/3636008436 >> v2:172.21.15.33:3300/0 conn(0x7fe7b006bb40 0x7fe7b0157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.976+0000 7fe7b72ea640 1 -- 172.21.15.31:0/3636008436 >> 172.21.15.31:0/3636008436 conn(0x7fe7b00cac70 msgr2=0x7fe7b00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 -- 172.21.15.31:0/3636008436 shutdown_connections 2023-10-30T13:47:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 -- 172.21.15.31:0/3636008436 wait complete. 2023-10-30T13:47:03.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 Processor -- start 2023-10-30T13:47:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 -- start start 2023-10-30T13:47:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7b006b760 0x7fe7b015d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006bb40 0x7fe7b00c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7b006c420 0x7fe7b015ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b015b530 con 0x7fe7b006bb40 2023-10-30T13:47:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b015df70 con 0x7fe7b006c420 2023-10-30T13:47:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b72ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b015b200 con 0x7fe7b006b760 2023-10-30T13:47:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006bb40 0x7fe7b00c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006bb40 0x7fe7b00c3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55254/0 (socket says 172.21.15.31:55254) 2023-10-30T13:47:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 -- 172.21.15.31:0/449177011 learned_addr learned my addr 172.21.15.31:0/449177011 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b5887640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7b006c420 0x7fe7b015ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b5086640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7b006b760 0x7fe7b015d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 -- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7b006b760 msgr2=0x7fe7b015d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7b006b760 0x7fe7b015d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 -- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7b006c420 msgr2=0x7fe7b015ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7b006c420 0x7fe7b015ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 -- 172.21.15.31:0/449177011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a8009b30 con 0x7fe7b006bb40 2023-10-30T13:47:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.980+0000 7fe7b4885640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006bb40 0x7fe7b00c3a20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e7e0 tx=0x7fe7a000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7a5ffb640 1 -- 172.21.15.31:0/449177011 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000c6d0 con 0x7fe7b006bb40 2023-10-30T13:47:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7a5ffb640 1 -- 172.21.15.31:0/449177011 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a000c830 con 0x7fe7b006bb40 2023-10-30T13:47:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7a5ffb640 1 -- 172.21.15.31:0/449177011 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000cbe0 con 0x7fe7b006bb40 2023-10-30T13:47:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b01fa6d0 con 0x7fe7b006bb40 2023-10-30T13:47:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b015dc90 con 0x7fe7b006bb40 2023-10-30T13:47:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b015b6e0 con 0x7fe7b006bb40 2023-10-30T13:47:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7a5ffb640 1 -- 172.21.15.31:0/449177011 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7a000cd40 con 0x7fe7b006bb40 2023-10-30T13:47:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7a5ffb640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7940664d0 0x7fe794068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7a5ffb640 1 -- 172.21.15.31:0/449177011 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fe7a0083fd0 con 0x7fe7b006bb40 2023-10-30T13:47:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.984+0000 7fe7b5086640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7940664d0 0x7fe794068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.988+0000 7fe7b5086640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7940664d0 0x7fe794068990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac000f70 tx=0x7fe7ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:03.988+0000 7fe7a5ffb640 1 -- 172.21.15.31:0/449177011 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7a00511e0 con 0x7fe7b006bb40 2023-10-30T13:47:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.176+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7b015b340 con 0x7fe7b006bb40 2023-10-30T13:47:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7a5ffb640 1 -- 172.21.15.31:0/449177011 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7fe7a0054800 con 0x7fe7b006bb40 2023-10-30T13:47:04.180 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:04.180 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:04.180 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:47:04.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7940664d0 msgr2=0x7fe794068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:04.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7940664d0 0x7fe794068990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac000f70 tx=0x7fe7ac009040 comp rx=0 tx=0).stop 2023-10-30T13:47:04.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006bb40 msgr2=0x7fe7b00c3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:04.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006bb40 0x7fe7b00c3a20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e7e0 tx=0x7fe7a000eca0 comp rx=0 tx=0).stop 2023-10-30T13:47:04.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 shutdown_connections 2023-10-30T13:47:04.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7940664d0 0x7fe794068990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:04.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7b006c420 0x7fe7b015ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:04.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7b006bb40 0x7fe7b00c3a20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:04.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 --2- 172.21.15.31:0/449177011 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe7b006b760 0x7fe7b015d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:04.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 >> 172.21.15.31:0/449177011 conn(0x7fe7b00cac70 msgr2=0x7fe7b0072550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:04.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 shutdown_connections 2023-10-30T13:47:04.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:04.180+0000 7fe7b72ea640 1 -- 172.21.15.31:0/449177011 wait complete. 2023-10-30T13:47:04.201 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:47:06.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:06.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.356+0000 7fdabcf94640 1 Processor -- start 2023-10-30T13:47:06.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdabcf94640 1 -- start start 2023-10-30T13:47:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdabcf94640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab8150830 0x7fdab806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdabcf94640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fdab8070520 0x7fdab80709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdabcf94640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab80c2310 con 0x7fdab80cb7d0 2023-10-30T13:47:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdabcf94640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab8156710 con 0x7fdab8070520 2023-10-30T13:47:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdabcf94640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80c4360 con 0x7fdab8150830 2023-10-30T13:47:06.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab659c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab8150830 0x7fdab806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:06.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab6d9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fdab8070520 0x7fdab80709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:06.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab6d9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fdab8070520 0x7fdab80709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:35446/0 (socket says 172.21.15.31:35446) 2023-10-30T13:47:06.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab659c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab8150830 0x7fdab806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:41452/0 (socket says 172.21.15.31:41452) 2023-10-30T13:47:06.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab6d9d640 1 -- 172.21.15.31:0/3888709197 learned_addr learned my addr 172.21.15.31:0/3888709197 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:06.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab5d9b640 1 -- 172.21.15.31:0/3888709197 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544785891 0 0) 0x7fdab80c2310 con 0x7fdab80cb7d0 2023-10-30T13:47:06.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab5d9b640 1 -- 172.21.15.31:0/3888709197 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa4003530 con 0x7fdab80cb7d0 2023-10-30T13:47:06.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab659c640 1 -- 172.21.15.31:0/3888709197 >> v2:172.21.15.33:3300/0 conn(0x7fdab8070520 msgr2=0x7fdab80709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:06.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab659c640 1 --2- 172.21.15.31:0/3888709197 >> v2:172.21.15.33:3300/0 conn(0x7fdab8070520 0x7fdab80709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab659c640 1 -- 172.21.15.31:0/3888709197 >> v1:172.21.15.33:6789/0 conn(0x7fdab80cb7d0 legacy=0x7fdab80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:06.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.360+0000 7fdab659c640 1 -- 172.21.15.31:0/3888709197 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab81569b0 con 0x7fdab8150830 2023-10-30T13:47:06.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdab659c640 1 --2- 172.21.15.31:0/3888709197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab8150830 0x7fdab806b500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800ba60 tx=0x7fdaa8021340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef4f2df0b16920a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:06.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdab5d9b640 1 -- 172.21.15.31:0/3888709197 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80b1600 con 0x7fdab8150830 2023-10-30T13:47:06.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdab5d9b640 1 -- 172.21.15.31:0/3888709197 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa80ad050 con 0x7fdab8150830 2023-10-30T13:47:06.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdab5d9b640 1 -- 172.21.15.31:0/3888709197 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80b1980 con 0x7fdab8150830 2023-10-30T13:47:06.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 -- 172.21.15.31:0/3888709197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab8150830 msgr2=0x7fdab806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3888709197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab8150830 0x7fdab806b500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800ba60 tx=0x7fdaa8021340 comp rx=0 tx=0).stop 2023-10-30T13:47:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 -- 172.21.15.31:0/3888709197 shutdown_connections 2023-10-30T13:47:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3888709197 >> v2:172.21.15.33:3300/0 conn(0x7fdab8070520 0x7fdab80709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3888709197 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab8150830 0x7fdab806b500 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 -- 172.21.15.31:0/3888709197 >> 172.21.15.31:0/3888709197 conn(0x7fdab80cac70 msgr2=0x7fdab80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:06.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 -- 172.21.15.31:0/3888709197 shutdown_connections 2023-10-30T13:47:06.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.364+0000 7fdabcf94640 1 -- 172.21.15.31:0/3888709197 wait complete. 2023-10-30T13:47:06.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 Processor -- start 2023-10-30T13:47:06.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 -- start start 2023-10-30T13:47:06.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdab8070520 0x7fdab8155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:06.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab80cb7d0 0x7fdab815a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:06.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdab8150830 0x7fdab80c1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:06.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab81ef240 con 0x7fdab8070520 2023-10-30T13:47:06.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab81ef520 con 0x7fdab8150830 2023-10-30T13:47:06.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdabcf94640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80c9d90 con 0x7fdab80cb7d0 2023-10-30T13:47:06.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdab8070520 0x7fdab8155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:06.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdab8070520 0x7fdab8155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38632/0 (socket says 172.21.15.31:38632) 2023-10-30T13:47:06.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 -- 172.21.15.31:0/3601612182 learned_addr learned my addr 172.21.15.31:0/3601612182 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:06.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab6d9d640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdab8150830 0x7fdab80c1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:06.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdaaffff640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab80cb7d0 0x7fdab815a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:06.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 -- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab80cb7d0 msgr2=0x7fdab815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:06.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab80cb7d0 0x7fdab815a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 -- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdab8150830 msgr2=0x7fdab80c1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:06.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdab8150830 0x7fdab80c1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 -- 172.21.15.31:0/3601612182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa800b6c0 con 0x7fdab8070520 2023-10-30T13:47:06.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdab659c640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdab8070520 0x7fdab8155320 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdaa801ef70 tx=0x7fdaa801e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:06.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdaaeffd640 1 -- 172.21.15.31:0/3601612182 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80ba420 con 0x7fdab8070520 2023-10-30T13:47:06.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdaaeffd640 1 -- 172.21.15.31:0/3601612182 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa80ba5a0 con 0x7fdab8070520 2023-10-30T13:47:06.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.368+0000 7fdaaeffd640 1 -- 172.21.15.31:0/3601612182 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80ba950 con 0x7fdab8070520 2023-10-30T13:47:06.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.372+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab80ca520 con 0x7fdab8070520 2023-10-30T13:47:06.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.372+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab80ca050 con 0x7fdab8070520 2023-10-30T13:47:06.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.372+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab8066510 con 0x7fdab8070520 2023-10-30T13:47:06.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.372+0000 7fdaaeffd640 1 -- 172.21.15.31:0/3601612182 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdaa80b8070 con 0x7fdab8070520 2023-10-30T13:47:06.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.376+0000 7fdaaeffd640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fda980664a0 0x7fda98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:06.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.376+0000 7fdaaeffd640 1 -- 172.21.15.31:0/3601612182 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fdaa80aae40 con 0x7fdab8070520 2023-10-30T13:47:06.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.376+0000 7fdaaffff640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fda980664a0 0x7fda98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:06.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.376+0000 7fdaaffff640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fda980664a0 0x7fda98068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdab814f4a0 tx=0x7fdaa0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:06.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.380+0000 7fdaaeffd640 1 -- 172.21.15.31:0/3601612182 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdaa80ee420 con 0x7fdab8070520 2023-10-30T13:47:06.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.580+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdab8073db0 con 0x7fdab8070520 2023-10-30T13:47:06.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.580+0000 7fdaaeffd640 1 -- 172.21.15.31:0/3601612182 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7fdaa80b6090 con 0x7fdab8070520 2023-10-30T13:47:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:06.582 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:06.583 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:47:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fda980664a0 msgr2=0x7fda98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fda980664a0 0x7fda98068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdab814f4a0 tx=0x7fdaa0009040 comp rx=0 tx=0).stop 2023-10-30T13:47:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdab8070520 msgr2=0x7fdab8155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdab8070520 0x7fdab8155320 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdaa801ef70 tx=0x7fdaa801e0e0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 shutdown_connections 2023-10-30T13:47:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fda980664a0 0x7fda98068960 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdab8150830 0x7fdab80c1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdab80cb7d0 0x7fdab815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 --2- 172.21.15.31:0/3601612182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdab8070520 0x7fdab8155320 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 >> 172.21.15.31:0/3601612182 conn(0x7fdab80cac70 msgr2=0x7fdab80c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 shutdown_connections 2023-10-30T13:47:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:06.584+0000 7fdabcf94640 1 -- 172.21.15.31:0/3601612182 wait complete. 2023-10-30T13:47:06.603 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:47:08.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:08.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dfc6b640 1 Processor -- start 2023-10-30T13:47:08.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dfc6b640 1 -- start start 2023-10-30T13:47:08.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dfc6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d80701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dfc6b640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f60d815a2e0 0x7f60d80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dfc6b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d80c2aa0 con 0x7f60d80cd4a0 2023-10-30T13:47:08.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dfc6b640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d80c3ab0 con 0x7f60d815a2e0 2023-10-30T13:47:08.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dfc6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80706e0 con 0x7f60d81537f0 2023-10-30T13:47:08.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dda07640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d80701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dd206640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f60d815a2e0 0x7f60d80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dda07640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d80701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:41478/0 (socket says 172.21.15.31:41478) 2023-10-30T13:47:08.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dd206640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f60d815a2e0 0x7f60d80ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:35470/0 (socket says 172.21.15.31:35470) 2023-10-30T13:47:08.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dda07640 1 -- 172.21.15.31:0/2272023428 learned_addr learned my addr 172.21.15.31:0/2272023428 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:08.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60de208640 1 --1- 172.21.15.31:0/2272023428 >> v1:172.21.15.33:6789/0 conn(0x7f60d80cd4a0 0x7f60d806b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:53902/0 (socket says 172.21.15.31:53902) 2023-10-30T13:47:08.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dd206640 1 -- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 msgr2=0x7f60d80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:08.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dd206640 1 --2- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d80701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dd206640 1 -- 172.21.15.31:0/2272023428 >> v1:172.21.15.33:6789/0 conn(0x7f60d80cd4a0 legacy=0x7f60d806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dd206640 1 -- 172.21.15.31:0/2272023428 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d80708e0 con 0x7f60d815a2e0 2023-10-30T13:47:08.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dda07640 1 --2- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d80701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:47:08.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.760+0000 7f60dca05640 1 -- 172.21.15.31:0/2272023428 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008972228 0 0) 0x7f60d80c2aa0 con 0x7f60d80cd4a0 2023-10-30T13:47:08.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.764+0000 7f60dd206640 1 --2- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d815a2e0 0x7f60d80ccf60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60c80097c0 tx=0x7f60c801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c03be5b1b7dd8f7b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.764+0000 7f60dca05640 1 -- 172.21.15.31:0/2272023428 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80a8970 con 0x7f60d815a2e0 2023-10-30T13:47:08.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.764+0000 7f60dca05640 1 -- 172.21.15.31:0/2272023428 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c80a8ad0 con 0x7f60d815a2e0 2023-10-30T13:47:08.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.764+0000 7f60dca05640 1 -- 172.21.15.31:0/2272023428 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80a8e60 con 0x7f60d815a2e0 2023-10-30T13:47:08.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.764+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d815a2e0 msgr2=0x7f60d80ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.764+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d815a2e0 0x7f60d80ccf60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60c80097c0 tx=0x7f60c801ec70 comp rx=0 tx=0).stop 2023-10-30T13:47:08.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2272023428 shutdown_connections 2023-10-30T13:47:08.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d815a2e0 0x7f60d80ccf60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2272023428 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d80701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2272023428 >> 172.21.15.31:0/2272023428 conn(0x7f60d8071fb0 msgr2=0x7f60d80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:08.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2272023428 shutdown_connections 2023-10-30T13:47:08.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2272023428 wait complete. 2023-10-30T13:47:08.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 Processor -- start 2023-10-30T13:47:08.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- start start 2023-10-30T13:47:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80cd4a0 0x7f60d80cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d8146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d815a2e0 0x7f60d8146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8157a10 con 0x7f60d815a2e0 2023-10-30T13:47:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8152e60 con 0x7f60d80cd4a0 2023-10-30T13:47:08.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dfc6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80cccf0 con 0x7f60d81537f0 2023-10-30T13:47:08.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60de208640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d815a2e0 0x7f60d8146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60de208640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d815a2e0 0x7f60d8146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38648/0 (socket says 172.21.15.31:38648) 2023-10-30T13:47:08.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60de208640 1 -- 172.21.15.31:0/2323121855 learned_addr learned my addr 172.21.15.31:0/2323121855 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:08.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dda07640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80cd4a0 0x7f60d80cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60dd206640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d8146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60de208640 1 -- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 msgr2=0x7f60d8146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60de208640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d8146aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60de208640 1 -- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80cd4a0 msgr2=0x7f60d80cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.768+0000 7f60de208640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80cd4a0 0x7f60d80cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60de208640 1 -- 172.21.15.31:0/2323121855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40094d0 con 0x7f60d815a2e0 2023-10-30T13:47:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60de208640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d815a2e0 0x7f60d8146fe0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f60d000c730 tx=0x7f60d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60ce7fc640 1 -- 172.21.15.31:0/2323121855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d000ce40 con 0x7f60d815a2e0 2023-10-30T13:47:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60ce7fc640 1 -- 172.21.15.31:0/2323121855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d000a640 con 0x7f60d815a2e0 2023-10-30T13:47:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60ce7fc640 1 -- 172.21.15.31:0/2323121855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d000aa30 con 0x7f60d815a2e0 2023-10-30T13:47:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c8009420 con 0x7f60d815a2e0 2023-10-30T13:47:08.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d8158c00 con 0x7f60d815a2e0 2023-10-30T13:47:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d804e170 con 0x7f60d815a2e0 2023-10-30T13:47:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60ce7fc640 1 -- 172.21.15.31:0/2323121855 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60d000ab90 con 0x7f60d815a2e0 2023-10-30T13:47:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60ce7fc640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60ce7fc640 1 -- 172.21.15.31:0/2323121855 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f60d00835a0 con 0x7f60d815a2e0 2023-10-30T13:47:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.772+0000 7f60dda07640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.776+0000 7f60dda07640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006cb0 tx=0x7f60d4009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.776+0000 7f60ce7fc640 1 -- 172.21.15.31:0/2323121855 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60d00507d0 con 0x7f60d815a2e0 2023-10-30T13:47:08.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d80cca10 con 0x7f60d815a2e0 2023-10-30T13:47:08.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60ce7fc640 1 -- 172.21.15.31:0/2323121855 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f60d0018090 con 0x7f60d815a2e0 2023-10-30T13:47:08.973 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:08.973 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:47:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 msgr2=0x7f60bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006cb0 tx=0x7f60d4009240 comp rx=0 tx=0).stop 2023-10-30T13:47:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d815a2e0 msgr2=0x7f60d8146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d815a2e0 0x7f60d8146fe0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f60d000c730 tx=0x7f60d000cbf0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 shutdown_connections 2023-10-30T13:47:08.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f60bc0664a0 0x7f60bc068960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60d815a2e0 0x7f60d8146fe0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f60d81537f0 0x7f60d8146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 --2- 172.21.15.31:0/2323121855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80cd4a0 0x7f60d80cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 >> 172.21.15.31:0/2323121855 conn(0x7f60d8071fb0 msgr2=0x7f60d814e540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:08.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 shutdown_connections 2023-10-30T13:47:08.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:08.972+0000 7f60dfc6b640 1 -- 172.21.15.31:0/2323121855 wait complete. 2023-10-30T13:47:08.994 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:47:10.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:11.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.148+0000 7f6b9d2d6640 1 Processor -- start 2023-10-30T13:47:11.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.148+0000 7f6b9d2d6640 1 -- start start 2023-10-30T13:47:11.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.148+0000 7f6b9d2d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6b98150830 0x7f6b9806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.148+0000 7f6b9d2d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f6b980cb7d0 0x7f6b980c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.148+0000 7f6b9d2d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b980c2310 con 0x7f6b980c3e40 2023-10-30T13:47:11.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.148+0000 7f6b9d2d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b98156710 con 0x7f6b98150830 2023-10-30T13:47:11.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.148+0000 7f6b9d2d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b980c4360 con 0x7f6b980cb7d0 2023-10-30T13:47:11.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6b98150830 0x7f6b9806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6b98150830 0x7f6b9806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38660/0 (socket says 172.21.15.31:38660) 2023-10-30T13:47:11.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 -- 172.21.15.31:0/1456118809 learned_addr learned my addr 172.21.15.31:0/1456118809 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:11.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b8ffff640 1 --2- 172.21.15.31:0/1456118809 >> v2:172.21.15.33:3300/0 conn(0x7f6b980cb7d0 0x7f6b980c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 -- 172.21.15.31:0/1456118809 >> v2:172.21.15.33:3300/0 conn(0x7f6b980cb7d0 msgr2=0x7f6b980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:11.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 --2- 172.21.15.31:0/1456118809 >> v2:172.21.15.33:3300/0 conn(0x7f6b980cb7d0 0x7f6b980c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 -- 172.21.15.31:0/1456118809 >> v1:172.21.15.33:6789/0 conn(0x7f6b980c3e40 legacy=0x7f6b98070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 -- 172.21.15.31:0/1456118809 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b981569b0 con 0x7f6b98150830 2023-10-30T13:47:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b96ffd640 1 --2- 172.21.15.31:0/1456118809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98150830 0x7f6b9806b500 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6b9806f9f0 tx=0x7f6b8801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4683d04fd5df8b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:11.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b967fc640 1 -- 172.21.15.31:0/1456118809 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b880a89f0 con 0x7f6b98150830 2023-10-30T13:47:11.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b967fc640 1 -- 172.21.15.31:0/1456118809 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b880a8b50 con 0x7f6b98150830 2023-10-30T13:47:11.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.152+0000 7f6b967fc640 1 -- 172.21.15.31:0/1456118809 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b880b0440 con 0x7f6b98150830 2023-10-30T13:47:11.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/1456118809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98150830 msgr2=0x7f6b9806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/1456118809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98150830 0x7f6b9806b500 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6b9806f9f0 tx=0x7f6b8801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/1456118809 shutdown_connections 2023-10-30T13:47:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/1456118809 >> v2:172.21.15.33:3300/0 conn(0x7f6b980cb7d0 0x7f6b980c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/1456118809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98150830 0x7f6b9806b500 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/1456118809 >> 172.21.15.31:0/1456118809 conn(0x7f6b980cac70 msgr2=0x7f6b980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/1456118809 shutdown_connections 2023-10-30T13:47:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/1456118809 wait complete. 2023-10-30T13:47:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 Processor -- start 2023-10-30T13:47:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- start start 2023-10-30T13:47:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b980c3e40 0x7f6b980ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b980cb7d0 0x7f6b981f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6b98150830 0x7f6b9813e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98148d50 con 0x7f6b980cb7d0 2023-10-30T13:47:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981473d0 con 0x7f6b980c3e40 2023-10-30T13:47:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b9d2d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9813fdd0 con 0x7f6b98150830 2023-10-30T13:47:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b980cb7d0 0x7f6b981f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b980cb7d0 0x7f6b981f23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38676/0 (socket says 172.21.15.31:38676) 2023-10-30T13:47:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 -- 172.21.15.31:0/585489576 learned_addr learned my addr 172.21.15.31:0/585489576 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b977fe640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6b98150830 0x7f6b9813e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b96ffd640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b980c3e40 0x7f6b980ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 -- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6b98150830 msgr2=0x7f6b9813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6b98150830 0x7f6b9813e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 -- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b980c3e40 msgr2=0x7f6b980ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b980c3e40 0x7f6b980ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b8ffff640 1 -- 172.21.15.31:0/585489576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b880094d0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.156+0000 7f6b96ffd640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b980c3e40 0x7f6b980ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b8ffff640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b980cb7d0 0x7f6b981f23a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000d800 tx=0x7f6b8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b8f7fe640 1 -- 172.21.15.31:0/585489576 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000b6c0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b8f7fe640 1 -- 172.21.15.31:0/585489576 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8000b860 con 0x7f6b980cb7d0 2023-10-30T13:47:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b8f7fe640 1 -- 172.21.15.31:0/585489576 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000bc10 con 0x7f6b980cb7d0 2023-10-30T13:47:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9813f5f0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b98147620 con 0x7f6b980cb7d0 2023-10-30T13:47:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b980c93f0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b8f7fe640 1 -- 172.21.15.31:0/585489576 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b8001e460 con 0x7f6b980cb7d0 2023-10-30T13:47:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b8f7fe640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6b780664a0 0x7f6b78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b8f7fe640 1 -- 172.21.15.31:0/585489576 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f6b800847d0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.160+0000 7f6b96ffd640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6b780664a0 0x7f6b78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.164+0000 7f6b96ffd640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6b780664a0 0x7f6b78068960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6b880b2000 tx=0x7f6b88008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.164+0000 7f6b8f7fe640 1 -- 172.21.15.31:0/585489576 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b980c93f0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.356+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b980c93f0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.356+0000 7f6b8f7fe640 1 -- 172.21.15.31:0/585489576 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f6b980c93f0 con 0x7f6b980cb7d0 2023-10-30T13:47:11.358 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:11.358 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5842,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:44:48.581423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5448},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":28,"addr":"172.21.15.31:6837/349618961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":349618961},{"type":"v1","addr":"172.21.15.31:6837","nonce":349618961}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:11.358 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2023-10-30T13:47:11.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6b780664a0 msgr2=0x7f6b78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6b780664a0 0x7f6b78068960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6b880b2000 tx=0x7f6b88008040 comp rx=0 tx=0).stop 2023-10-30T13:47:11.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b980cb7d0 msgr2=0x7f6b981f23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b980cb7d0 0x7f6b981f23a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000d800 tx=0x7f6b8000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 shutdown_connections 2023-10-30T13:47:11.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6b780664a0 0x7f6b78068960 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6b98150830 0x7f6b9813e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b980cb7d0 0x7f6b981f23a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 --2- 172.21.15.31:0/585489576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b980c3e40 0x7f6b980ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 >> 172.21.15.31:0/585489576 conn(0x7f6b980cac70 msgr2=0x7f6b98149e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:11.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 shutdown_connections 2023-10-30T13:47:11.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:11.360+0000 7f6b9d2d6640 1 -- 172.21.15.31:0/585489576 wait complete. 2023-10-30T13:47:11.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:47:13.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.532+0000 7ff09646b640 1 Processor -- start 2023-10-30T13:47:13.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.532+0000 7ff09646b640 1 -- start start 2023-10-30T13:47:13.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.532+0000 7ff09646b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff090153bb0 0x7ff0900c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.532+0000 7ff09646b640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff09006c420 0x7ff0900c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.532+0000 7ff09646b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900c5f80 con 0x7ff0900c7170 2023-10-30T13:47:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.532+0000 7ff09646b640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0900c5690 con 0x7ff09006c420 2023-10-30T13:47:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.532+0000 7ff09646b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09015a580 con 0x7ff090153bb0 2023-10-30T13:47:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff090153bb0 0x7ff0900c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08ffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff090153bb0 0x7ff0900c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:41498/0 (socket says 172.21.15.31:41498) 2023-10-30T13:47:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08ffff640 1 -- 172.21.15.31:0/759769389 learned_addr learned my addr 172.21.15.31:0/759769389 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08f7fe640 1 --2- 172.21.15.31:0/759769389 >> v2:172.21.15.33:3300/0 conn(0x7ff09006c420 0x7ff0900c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08ffff640 1 -- 172.21.15.31:0/759769389 >> v2:172.21.15.33:3300/0 conn(0x7ff09006c420 msgr2=0x7ff0900c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:13.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08ffff640 1 --2- 172.21.15.31:0/759769389 >> v2:172.21.15.33:3300/0 conn(0x7ff09006c420 0x7ff0900c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08ffff640 1 -- 172.21.15.31:0/759769389 >> v1:172.21.15.33:6789/0 conn(0x7ff0900c7170 legacy=0x7ff090159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.536+0000 7ff08ffff640 1 -- 172.21.15.31:0/759769389 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0900c5930 con 0x7ff090153bb0 2023-10-30T13:47:13.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff08ffff640 1 --2- 172.21.15.31:0/759769389 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff090153bb0 0x7ff0900c6410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff080009870 tx=0x7ff08001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ccc22ca48d00d3a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff08effd640 1 -- 172.21.15.31:0/759769389 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800a8a30 con 0x7ff090153bb0 2023-10-30T13:47:13.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff08effd640 1 -- 172.21.15.31:0/759769389 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0800a8b90 con 0x7ff090153bb0 2023-10-30T13:47:13.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff08effd640 1 -- 172.21.15.31:0/759769389 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800b0440 con 0x7ff090153bb0 2023-10-30T13:47:13.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 -- 172.21.15.31:0/759769389 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff090153bb0 msgr2=0x7ff0900c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 --2- 172.21.15.31:0/759769389 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff090153bb0 0x7ff0900c6410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff080009870 tx=0x7ff08001ed30 comp rx=0 tx=0).stop 2023-10-30T13:47:13.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 -- 172.21.15.31:0/759769389 shutdown_connections 2023-10-30T13:47:13.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 --2- 172.21.15.31:0/759769389 >> v2:172.21.15.33:3300/0 conn(0x7ff09006c420 0x7ff0900c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 --2- 172.21.15.31:0/759769389 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff090153bb0 0x7ff0900c6410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 -- 172.21.15.31:0/759769389 >> 172.21.15.31:0/759769389 conn(0x7ff0900cac70 msgr2=0x7ff0900cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:13.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 -- 172.21.15.31:0/759769389 shutdown_connections 2023-10-30T13:47:13.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 -- 172.21.15.31:0/759769389 wait complete. 2023-10-30T13:47:13.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.540+0000 7ff09646b640 1 Processor -- start 2023-10-30T13:47:13.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 -- start start 2023-10-30T13:47:13.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff09006c420 0x7ff0900be800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0900c7170 0x7ff0901f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff090153bb0 0x7ff0901eff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff090154240 con 0x7ff090153bb0 2023-10-30T13:47:13.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09014d5c0 con 0x7ff09006c420 2023-10-30T13:47:13.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900714b0 con 0x7ff0900c7170 2023-10-30T13:47:13.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff090153bb0 0x7ff0901eff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff08ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff09006c420 0x7ff0900be800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff090153bb0 0x7ff0901eff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38686/0 (socket says 172.21.15.31:38686) 2023-10-30T13:47:13.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 -- 172.21.15.31:0/2986707423 learned_addr learned my addr 172.21.15.31:0/2986707423 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:13.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff08f7fe640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0900c7170 0x7ff0901f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 -- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0900c7170 msgr2=0x7ff0901f2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0900c7170 0x7ff0901f2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 -- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff09006c420 msgr2=0x7ff0900be800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff09006c420 0x7ff0900be800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 -- 172.21.15.31:0/2986707423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0800094d0 con 0x7ff090153bb0 2023-10-30T13:47:13.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff094a08640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff090153bb0 0x7ff0901eff20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c760 tx=0x7ff08800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff08cff9640 1 -- 172.21.15.31:0/2986707423 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088003b80 con 0x7ff090153bb0 2023-10-30T13:47:13.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff08cff9640 1 -- 172.21.15.31:0/2986707423 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff088003ce0 con 0x7ff090153bb0 2023-10-30T13:47:13.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff08cff9640 1 -- 172.21.15.31:0/2986707423 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08800aa30 con 0x7ff090153bb0 2023-10-30T13:47:13.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0901f2c30 con 0x7ff090153bb0 2023-10-30T13:47:13.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0901f19e0 con 0x7ff090153bb0 2023-10-30T13:47:13.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.544+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff090159140 con 0x7ff090153bb0 2023-10-30T13:47:13.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.548+0000 7ff08cff9640 1 -- 172.21.15.31:0/2986707423 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff08800ab90 con 0x7ff090153bb0 2023-10-30T13:47:13.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.548+0000 7ff08cff9640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff07c0664d0 0x7ff07c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.548+0000 7ff08cff9640 1 -- 172.21.15.31:0/2986707423 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7ff088083d70 con 0x7ff090153bb0 2023-10-30T13:47:13.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.548+0000 7ff08ffff640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff07c0664d0 0x7ff07c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.548+0000 7ff08ffff640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff07c0664d0 0x7ff07c068990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0800b2000 tx=0x7ff0800aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.552+0000 7ff08cff9640 1 -- 172.21.15.31:0/2986707423 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff090159140 con 0x7ff090153bb0 2023-10-30T13:47:13.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.740+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff090159140 con 0x7ff090153bb0 2023-10-30T13:47:13.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.740+0000 7ff08cff9640 1 -- 172.21.15.31:0/2986707423 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5695 (secure 0 0 0) 0x7ff090159140 con 0x7ff090153bb0 2023-10-30T13:47:13.742 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:13.742 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:11.952302+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 60 2023-10-30T13:47:13.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff07c0664d0 msgr2=0x7ff07c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff07c0664d0 0x7ff07c068990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0800b2000 tx=0x7ff0800aa070 comp rx=0 tx=0).stop 2023-10-30T13:47:13.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff090153bb0 msgr2=0x7ff0901eff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff090153bb0 0x7ff0901eff20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c760 tx=0x7ff08800cc20 comp rx=0 tx=0).stop 2023-10-30T13:47:13.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 shutdown_connections 2023-10-30T13:47:13.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff07c0664d0 0x7ff07c068990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff090153bb0 0x7ff0901eff20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0900c7170 0x7ff0901f2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 --2- 172.21.15.31:0/2986707423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff09006c420 0x7ff0900be800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 >> 172.21.15.31:0/2986707423 conn(0x7ff0900cac70 msgr2=0x7ff090071680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:13.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 shutdown_connections 2023-10-30T13:47:13.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.744+0000 7ff09646b640 1 -- 172.21.15.31:0/2986707423 wait complete. 2023-10-30T13:47:13.763 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:47:13.763 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:47:13.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:13.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.904+0000 7f0dfabf5640 1 Processor -- start 2023-10-30T13:47:13.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.904+0000 7f0dfabf5640 1 -- start start 2023-10-30T13:47:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0dfabf5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 0x7f0df40709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0dfabf5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df40c2310 con 0x7f0df4150830 2023-10-30T13:47:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0dfabf5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df4156710 con 0x7f0df40cb7d0 2023-10-30T13:47:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0dfabf5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df406bc10 con 0x7f0df4070520 2023-10-30T13:47:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f0df4150830 0x7f0df406b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48394/0 (socket says 172.21.15.31:48394) 2023-10-30T13:47:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 -- 172.21.15.31:0/2702294805 learned_addr learned my addr 172.21.15.31:0/2702294805 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df9192640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 0x7f0df40709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001990730 0 0) 0x7f0df40c2310 con 0x7f0df4150830 2023-10-30T13:47:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de4003530 con 0x7f0df4150830 2023-10-30T13:47:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1814047230 0 0) 0x7f0de4003530 con 0x7f0df4150830 2023-10-30T13:47:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 msgr2=0x7f0df40709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 0x7f0df40709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 >> v1:172.21.15.33:6789/0 conn(0x7f0df40cb7d0 legacy=0x7f0df40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df40c4360 con 0x7f0df4150830 2023-10-30T13:47:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f0de8001680 con 0x7f0df4150830 2023-10-30T13:47:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 >> v1:172.21.15.31:6789/0 conn(0x7f0df4150830 legacy=0x7f0df406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0de4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de4004360 0x7f0de4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 0x7f0de4005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df406bc10 con 0x7f0de40039b0 2023-10-30T13:47:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de400b990 con 0x7f0de4004360 2023-10-30T13:47:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de400dd60 con 0x7f0df4070520 2023-10-30T13:47:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0de4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df3fff640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de4004360 0x7f0de4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df9192640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 0x7f0de4005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 -- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 msgr2=0x7f0de4005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 0x7f0de4005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 -- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de4004360 msgr2=0x7f0de4004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de4004360 0x7f0de4004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df8991640 1 -- 172.21.15.31:0/2702294805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de4003530 con 0x7f0de40039b0 2023-10-30T13:47:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.908+0000 7f0df9192640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df4070520 0x7f0de4005090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0df8991640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0de4003d90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0de800c210 tx=0x7f0de801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee83b88a5985fe6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80a5b60 con 0x7f0de40039b0 2023-10-30T13:47:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80a5ce0 con 0x7f0de40039b0 2023-10-30T13:47:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0df37fe640 1 -- 172.21.15.31:0/2702294805 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8005b20 con 0x7f0de40039b0 2023-10-30T13:47:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0dfabf5640 1 -- 172.21.15.31:0/2702294805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 msgr2=0x7f0de4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0dfabf5640 1 --2- 172.21.15.31:0/2702294805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0de4003d90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0de800c210 tx=0x7f0de801cc60 comp rx=0 tx=0).stop 2023-10-30T13:47:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0df9192640 1 -- 172.21.15.31:0/2702294805 reap_dead start 2023-10-30T13:47:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0dfabf5640 1 -- 172.21.15.31:0/2702294805 shutdown_connections 2023-10-30T13:47:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0dfabf5640 1 -- 172.21.15.31:0/2702294805 >> 172.21.15.31:0/2702294805 conn(0x7f0df40cac70 msgr2=0x7f0df40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0dfabf5640 1 -- 172.21.15.31:0/2702294805 shutdown_connections 2023-10-30T13:47:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.912+0000 7f0dfabf5640 1 -- 172.21.15.31:0/2702294805 wait complete. 2023-10-30T13:47:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 Processor -- start 2023-10-30T13:47:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 -- start start 2023-10-30T13:47:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0df41449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df40c64e0 0x7f0df41f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df41f26b0 0x7f0df414a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df41577c0 con 0x7f0de40039b0 2023-10-30T13:47:13.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df4158060 con 0x7f0df41f26b0 2023-10-30T13:47:13.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df4144f00 con 0x7f0df40c64e0 2023-10-30T13:47:13.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0df41449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0df41449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38704/0 (socket says 172.21.15.31:38704) 2023-10-30T13:47:13.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 -- 172.21.15.31:0/519554179 learned_addr learned my addr 172.21.15.31:0/519554179 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:13.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df3fff640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df40c64e0 0x7f0df41f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df9192640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df41f26b0 0x7f0df414a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 -- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df40c64e0 msgr2=0x7f0df41f2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df40c64e0 0x7f0df41f2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 -- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df41f26b0 msgr2=0x7f0df414a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df41f26b0 0x7f0df414a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 -- 172.21.15.31:0/519554179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df40c6a20 con 0x7f0de40039b0 2023-10-30T13:47:13.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df3fff640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df40c64e0 0x7f0df41f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:13.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df9192640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df41f26b0 0x7f0df414a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:13.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df8991640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0df41449c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0de8098680 tx=0x7f0de4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df17fa640 1 -- 172.21.15.31:0/519554179 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80b4070 con 0x7f0de40039b0 2023-10-30T13:47:13.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df17fa640 1 -- 172.21.15.31:0/519554179 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80a5b70 con 0x7f0de40039b0 2023-10-30T13:47:13.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0df17fa640 1 -- 172.21.15.31:0/519554179 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8001640 con 0x7f0de40039b0 2023-10-30T13:47:13.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df40c6cc0 con 0x7f0de40039b0 2023-10-30T13:47:13.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df4157cf0 con 0x7f0de40039b0 2023-10-30T13:47:13.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.916+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df4157eb0 con 0x7f0de40039b0 2023-10-30T13:47:13.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.920+0000 7f0df17fa640 1 -- 172.21.15.31:0/519554179 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0de8007070 con 0x7f0de40039b0 2023-10-30T13:47:13.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.920+0000 7f0df17fa640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0dd80664d0 0x7f0dd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.920+0000 7f0df17fa640 1 -- 172.21.15.31:0/519554179 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f0de811e160 con 0x7f0de40039b0 2023-10-30T13:47:13.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.920+0000 7f0df3fff640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0dd80664d0 0x7f0dd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.920+0000 7f0df3fff640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0dd80664d0 0x7f0dd8068990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0de0007990 tx=0x7f0de0007900 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:13.924+0000 7f0df17fa640 1 -- 172.21.15.31:0/519554179 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0de80eb3d0 con 0x7f0de40039b0 2023-10-30T13:47:14.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.112+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0df4066510 con 0x7f0de40039b0 2023-10-30T13:47:14.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.112+0000 7f0df17fa640 1 -- 172.21.15.31:0/519554179 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5695 (secure 0 0 0) 0x7f0de80ee9f0 con 0x7f0de40039b0 2023-10-30T13:47:14.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:14.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:11.952302+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:14.115 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 60 2023-10-30T13:47:14.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0dd80664d0 msgr2=0x7f0dd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:14.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0dd80664d0 0x7f0dd8068990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0de0007990 tx=0x7f0de0007900 comp rx=0 tx=0).stop 2023-10-30T13:47:14.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 msgr2=0x7f0df41449c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:14.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0df41449c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0de8098680 tx=0x7f0de4005800 comp rx=0 tx=0).stop 2023-10-30T13:47:14.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 shutdown_connections 2023-10-30T13:47:14.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0dd80664d0 0x7f0dd8068990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:14.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df41f26b0 0x7f0df414a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:14.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0df40c64e0 0x7f0df41f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:14.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 --2- 172.21.15.31:0/519554179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0de40039b0 0x7f0df41449c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:14.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 >> 172.21.15.31:0/519554179 conn(0x7f0df40cac70 msgr2=0x7f0df40c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:14.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 shutdown_connections 2023-10-30T13:47:14.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:14.116+0000 7f0dfabf5640 1 -- 172.21.15.31:0/519554179 wait complete. 2023-10-30T13:47:14.136 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 60, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 60, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5842": { "addr": "172.21.15.33:6836/15857146", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 15857146, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 15857146, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5842, "incarnation": 60, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:47:11.952302+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5842 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:47:16.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f1000c09640 1 Processor -- start 2023-10-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f1000c09640 1 -- start start 2023-10-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f1000c09640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0ffc0cb7d0 0x7f0ffc0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f1000c09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f1000c09640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ffc0c2310 con 0x7f0ffc150830 2023-10-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f1000c09640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ffc06bc10 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f1000c09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc156710 con 0x7f0ffc0c3e40 2023-10-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ff3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0ffc0cb7d0 0x7f0ffc0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ff3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0ffc0cb7d0 0x7f0ffc0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49616/0 (socket says 172.21.15.31:49616) 2023-10-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ff3fff640 1 -- 172.21.15.31:0/3291417131 learned_addr learned my addr 172.21.15.31:0/3291417131 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ffad9d640 1 --2- 172.21.15.31:0/3291417131 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ff3fff640 1 -- 172.21.15.31:0/3291417131 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 msgr2=0x7f0ffc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ff3fff640 1 --2- 172.21.15.31:0/3291417131 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ff3fff640 1 -- 172.21.15.31:0/3291417131 >> v1:172.21.15.33:6789/0 conn(0x7f0ffc150830 legacy=0x7f0ffc06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.292+0000 7f0ff3fff640 1 -- 172.21.15.31:0/3291417131 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0c4360 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f0ff3fff640 1 --2- 172.21.15.31:0/3291417131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc0c3900 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009870 tx=0x7f0fe4020f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fdebf209245b6c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f0ff9d9b640 1 -- 172.21.15.31:0/3291417131 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40aabc0 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f0ff9d9b640 1 -- 172.21.15.31:0/3291417131 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe40aad20 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f0ff9d9b640 1 -- 172.21.15.31:0/3291417131 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40b2660 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 -- 172.21.15.31:0/3291417131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 msgr2=0x7f0ffc0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 --2- 172.21.15.31:0/3291417131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc0c3900 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009870 tx=0x7f0fe4020f00 comp rx=0 tx=0).stop 2023-10-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 -- 172.21.15.31:0/3291417131 shutdown_connections 2023-10-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 --2- 172.21.15.31:0/3291417131 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 --2- 172.21.15.31:0/3291417131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc0c3900 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 -- 172.21.15.31:0/3291417131 >> 172.21.15.31:0/3291417131 conn(0x7f0ffc0cac70 msgr2=0x7f0ffc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 -- 172.21.15.31:0/3291417131 shutdown_connections 2023-10-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.296+0000 7f1000c09640 1 -- 172.21.15.31:0/3291417131 wait complete. 2023-10-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 Processor -- start 2023-10-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 -- start start 2023-10-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc14eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ffc150830 0x7f0ffc158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc15a8c0 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc151180 con 0x7f0ffc150830 2023-10-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc156510 con 0x7f0ffc0c3e40 2023-10-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49618/0 (socket says 172.21.15.31:49618) 2023-10-30T13:47:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 -- 172.21.15.31:0/459292980 learned_addr learned my addr 172.21.15.31:0/459292980 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ffa59c640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc14eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ffad9d640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ffc150830 0x7f0ffc158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:16.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 -- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 msgr2=0x7f0ffc14eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc14eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 -- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ffc150830 msgr2=0x7f0ffc158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ffc150830 0x7f0ffc158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 -- 172.21.15.31:0/459292980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe40094d0 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff3fff640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc158a60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0fe40b5040 tx=0x7f0fe401dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff2ffd640 1 -- 172.21.15.31:0/459292980 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40b2e20 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff2ffd640 1 -- 172.21.15.31:0/459292980 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe40aae90 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc1585d0 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f0ff2ffd640 1 -- 172.21.15.31:0/459292980 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40bb670 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.300+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ffc155c70 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.304+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ffc066510 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.304+0000 7f0ff2ffd640 1 -- 172.21.15.31:0/459292980 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fe40bb7d0 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.304+0000 7f0ff2ffd640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fdc0664a0 0x7f0fdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.304+0000 7f0ff2ffd640 1 -- 172.21.15.31:0/459292980 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f0fe41214a0 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.304+0000 7f0ffa59c640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fdc0664a0 0x7f0fdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.304+0000 7f0ffa59c640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fdc0664a0 0x7f0fdc068960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00b810 tx=0x7f0fec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.308+0000 7f0ff2ffd640 1 -- 172.21.15.31:0/459292980 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fe40ee5e0 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.496+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ffc158760 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.496+0000 7f0ff2ffd640 1 -- 172.21.15.31:0/459292980 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5695 (secure 0 0 0) 0x7f0fe40f1c00 con 0x7f0ffc0cb7d0 2023-10-30T13:47:16.498 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:16.498 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:11.952302+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 60 2023-10-30T13:47:16.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.496+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fdc0664a0 msgr2=0x7f0fdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:16.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.496+0000 7f1000c09640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fdc0664a0 0x7f0fdc068960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00b810 tx=0x7f0fec008040 comp rx=0 tx=0).stop 2023-10-30T13:47:16.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.496+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 msgr2=0x7f0ffc158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:16.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.496+0000 7f1000c09640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc158a60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0fe40b5040 tx=0x7f0fe401dca0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 shutdown_connections 2023-10-30T13:47:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fdc0664a0 0x7f0fdc068960 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ffc150830 0x7f0ffc158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ffc0cb7d0 0x7f0ffc158a60 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 --2- 172.21.15.31:0/459292980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0ffc0c3e40 0x7f0ffc14eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 >> 172.21.15.31:0/459292980 conn(0x7f0ffc0cac70 msgr2=0x7f0ffc06bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 shutdown_connections 2023-10-30T13:47:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:16.500+0000 7f1000c09640 1 -- 172.21.15.31:0/459292980 wait complete. 2023-10-30T13:47:16.519 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:18.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adf0d6640 1 Processor -- start 2023-10-30T13:47:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adf0d6640 1 -- start start 2023-10-30T13:47:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adf0d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad80c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adf0d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f6ad80c9b40 0x7f6ad81597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adf0d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad815cda0 con 0x7f6ad80c5f80 2023-10-30T13:47:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adf0d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad815d990 con 0x7f6ad80c9b40 2023-10-30T13:47:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adf0d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad8153f60 con 0x7f6ad80c5ba0 2023-10-30T13:47:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6add673640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f6ad80c9b40 0x7f6ad81597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6ad7fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f6ad80c5f80 0x7f6ad8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:41376/0 (socket says 172.21.15.31:41376) 2023-10-30T13:47:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6adce72640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad80c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.676+0000 7f6ad7fff640 1 -- 172.21.15.31:0/3425949564 learned_addr learned my addr 172.21.15.31:0/3425949564 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6ad77fe640 1 -- 172.21.15.31:0/3425949564 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710164819 0 0) 0x7f6ad815cda0 con 0x7f6ad80c5f80 2023-10-30T13:47:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6ad77fe640 1 -- 172.21.15.31:0/3425949564 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac8003530 con 0x7f6ad80c5f80 2023-10-30T13:47:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6add673640 1 -- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 msgr2=0x7f6ad80c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6add673640 1 --2- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad80c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6add673640 1 -- 172.21.15.31:0/3425949564 >> v1:172.21.15.33:6789/0 conn(0x7f6ad80c5f80 legacy=0x7f6ad8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6add673640 1 -- 172.21.15.31:0/3425949564 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad806f490 con 0x7f6ad80c9b40 2023-10-30T13:47:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6adce72640 1 --2- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad80c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:47:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6add673640 1 --2- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c9b40 0x7f6ad81597b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009dc0 tx=0x7f6ad001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e62b2ab29f8076 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6ad77fe640 1 -- 172.21.15.31:0/3425949564 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad00a57f0 con 0x7f6ad80c9b40 2023-10-30T13:47:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6ad77fe640 1 -- 172.21.15.31:0/3425949564 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad00a5950 con 0x7f6ad80c9b40 2023-10-30T13:47:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.680+0000 7f6ad77fe640 1 -- 172.21.15.31:0/3425949564 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad00a5ce0 con 0x7f6ad80c9b40 2023-10-30T13:47:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c9b40 msgr2=0x7f6ad81597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c9b40 0x7f6ad81597b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009dc0 tx=0x7f6ad001c930 comp rx=0 tx=0).stop 2023-10-30T13:47:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- 172.21.15.31:0/3425949564 shutdown_connections 2023-10-30T13:47:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c9b40 0x7f6ad81597b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/3425949564 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad80c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- 172.21.15.31:0/3425949564 >> 172.21.15.31:0/3425949564 conn(0x7f6ad806afa0 msgr2=0x7f6ad806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:18.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- 172.21.15.31:0/3425949564 shutdown_connections 2023-10-30T13:47:18.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- 172.21.15.31:0/3425949564 wait complete. 2023-10-30T13:47:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 Processor -- start 2023-10-30T13:47:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- start start 2023-10-30T13:47:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad8153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c5f80 0x7f6ad8150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad80c9b40 0x7f6ad8150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad81f5d70 con 0x7f6ad80c9b40 2023-10-30T13:47:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad80c0b40 con 0x7f6ad80c5f80 2023-10-30T13:47:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adf0d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad80c1050 con 0x7f6ad80c5ba0 2023-10-30T13:47:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6add673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad80c9b40 0x7f6ad8150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6add673640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad80c9b40 0x7f6ad8150d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49634/0 (socket says 172.21.15.31:49634) 2023-10-30T13:47:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6add673640 1 -- 172.21.15.31:0/1854003480 learned_addr learned my addr 172.21.15.31:0/1854003480 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6ad7fff640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c5f80 0x7f6ad8150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.684+0000 7f6adce72640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad8153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6add673640 1 -- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 msgr2=0x7f6ad8153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6add673640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad8153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6add673640 1 -- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c5f80 msgr2=0x7f6ad8150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6add673640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c5f80 0x7f6ad8150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6add673640 1 -- 172.21.15.31:0/1854003480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc009420 con 0x7f6ad80c9b40 2023-10-30T13:47:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6add673640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad80c9b40 0x7f6ad8150d70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6ad001a560 tx=0x7f6ad00196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6ad57fa640 1 -- 172.21.15.31:0/1854003480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad00b9650 con 0x7f6ad80c9b40 2023-10-30T13:47:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6ad57fa640 1 -- 172.21.15.31:0/1854003480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad00b97b0 con 0x7f6ad80c9b40 2023-10-30T13:47:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6ad57fa640 1 -- 172.21.15.31:0/1854003480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad00b9b40 con 0x7f6ad80c9b40 2023-10-30T13:47:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad0009a70 con 0x7f6ad80c9b40 2023-10-30T13:47:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad8159760 con 0x7f6ad80c9b40 2023-10-30T13:47:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad804e170 con 0x7f6ad80c9b40 2023-10-30T13:47:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6ad57fa640 1 -- 172.21.15.31:0/1854003480 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ad00b9ca0 con 0x7f6ad80c9b40 2023-10-30T13:47:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6ad57fa640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6abc0664a0 0x7f6abc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.688+0000 7f6ad57fa640 1 -- 172.21.15.31:0/1854003480 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f6ad011f1b0 con 0x7f6ad80c9b40 2023-10-30T13:47:18.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.692+0000 7f6adce72640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6abc0664a0 0x7f6abc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.692+0000 7f6adce72640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6abc0664a0 0x7f6abc068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80bf4a0 tx=0x7f6acc009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:18.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.696+0000 7f6ad57fa640 1 -- 172.21.15.31:0/1854003480 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ad00ec3a0 con 0x7f6ad80c9b40 2023-10-30T13:47:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.884+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ad80c1190 con 0x7f6ad80c9b40 2023-10-30T13:47:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.884+0000 7f6ad57fa640 1 -- 172.21.15.31:0/1854003480 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5695 (secure 0 0 0) 0x7f6ad00ef9c0 con 0x7f6ad80c9b40 2023-10-30T13:47:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:18.886 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:11.952302+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:18.886 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 60 2023-10-30T13:47:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6abc0664a0 msgr2=0x7f6abc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6abc0664a0 0x7f6abc068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80bf4a0 tx=0x7f6acc009190 comp rx=0 tx=0).stop 2023-10-30T13:47:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad80c9b40 msgr2=0x7f6ad8150d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad80c9b40 0x7f6ad8150d70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6ad001a560 tx=0x7f6ad00196d0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 shutdown_connections 2023-10-30T13:47:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6abc0664a0 0x7f6abc068960 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad80c9b40 0x7f6ad8150d70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ad80c5f80 0x7f6ad8150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 --2- 172.21.15.31:0/1854003480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ad80c5ba0 0x7f6ad8153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 >> 172.21.15.31:0/1854003480 conn(0x7f6ad806afa0 msgr2=0x7f6ad80723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:18.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 shutdown_connections 2023-10-30T13:47:18.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:18.888+0000 7f6adf0d6640 1 -- 172.21.15.31:0/1854003480 wait complete. 2023-10-30T13:47:18.907 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:20.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:21.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946b488640 1 Processor -- start 2023-10-30T13:47:21.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946b488640 1 -- start start 2023-10-30T13:47:21.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946b488640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f94640c8eb0 0x7f946415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:21.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946b488640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f946415ac70 0x7f946415a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:21.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946b488640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946415dcc0 con 0x7f946406aeb0 2023-10-30T13:47:21.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946b488640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9464159800 con 0x7f94640c8eb0 2023-10-30T13:47:21.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946b488640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9464159d80 con 0x7f946415ac70 2023-10-30T13:47:21.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946a486640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f94640c8eb0 0x7f946415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946a486640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f94640c8eb0 0x7f946415cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49644/0 (socket says 172.21.15.31:49644) 2023-10-30T13:47:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946a486640 1 -- 172.21.15.31:0/2384507295 learned_addr learned my addr 172.21.15.31:0/2384507295 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f9469c85640 1 --2- 172.21.15.31:0/2384507295 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f946415ac70 0x7f946415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946a486640 1 -- 172.21.15.31:0/2384507295 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f946415ac70 msgr2=0x7f946415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946a486640 1 --2- 172.21.15.31:0/2384507295 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f946415ac70 0x7f946415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946a486640 1 -- 172.21.15.31:0/2384507295 >> v1:172.21.15.33:6789/0 conn(0x7f946406aeb0 legacy=0x7f9464069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:21.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f946a486640 1 -- 172.21.15.31:0/2384507295 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946415a750 con 0x7f94640c8eb0 2023-10-30T13:47:21.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.064+0000 7f9469c85640 1 --2- 172.21.15.31:0/2384507295 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f946415ac70 0x7f946415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946a486640 1 --2- 172.21.15.31:0/2384507295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94640c8eb0 0x7f946415cd70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9460009870 tx=0x7f946001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3b681f4375af46 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f9469484640 1 -- 172.21.15.31:0/2384507295 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600a8a30 con 0x7f94640c8eb0 2023-10-30T13:47:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f9469484640 1 -- 172.21.15.31:0/2384507295 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94600a8b90 con 0x7f94640c8eb0 2023-10-30T13:47:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f9469484640 1 -- 172.21.15.31:0/2384507295 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600b0440 con 0x7f94640c8eb0 2023-10-30T13:47:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 -- 172.21.15.31:0/2384507295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94640c8eb0 msgr2=0x7f946415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 --2- 172.21.15.31:0/2384507295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94640c8eb0 0x7f946415cd70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9460009870 tx=0x7f946001ed30 comp rx=0 tx=0).stop 2023-10-30T13:47:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 -- 172.21.15.31:0/2384507295 shutdown_connections 2023-10-30T13:47:21.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 --2- 172.21.15.31:0/2384507295 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f946415ac70 0x7f946415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 --2- 172.21.15.31:0/2384507295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94640c8eb0 0x7f946415cd70 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 -- 172.21.15.31:0/2384507295 >> 172.21.15.31:0/2384507295 conn(0x7f94640cef00 msgr2=0x7f94640cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:21.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 -- 172.21.15.31:0/2384507295 shutdown_connections 2023-10-30T13:47:21.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.068+0000 7f946b488640 1 -- 172.21.15.31:0/2384507295 wait complete. 2023-10-30T13:47:21.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 Processor -- start 2023-10-30T13:47:21.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 -- start start 2023-10-30T13:47:21.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f946406aeb0 0x7f94640cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:21.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f94640c8eb0 0x7f94640c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:21.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f946415ac70 0x7f946415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:21.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946415c500 con 0x7f946406aeb0 2023-10-30T13:47:21.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946415cf90 con 0x7f946415ac70 2023-10-30T13:47:21.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946415c220 con 0x7f94640c8eb0 2023-10-30T13:47:21.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f946406aeb0 0x7f94640cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:21.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f946406aeb0 0x7f94640cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49658/0 (socket says 172.21.15.31:49658) 2023-10-30T13:47:21.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 -- 172.21.15.31:0/818728597 learned_addr learned my addr 172.21.15.31:0/818728597 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:21.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f9469c85640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f94640c8eb0 0x7f94640c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:21.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946ac87640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f946415ac70 0x7f946415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:21.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 -- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f94640c8eb0 msgr2=0x7f94640c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f94640c8eb0 0x7f94640c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 -- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f946415ac70 msgr2=0x7f946415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f946415ac70 0x7f946415b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 -- 172.21.15.31:0/818728597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94600094d0 con 0x7f946406aeb0 2023-10-30T13:47:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946a486640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f946406aeb0 0x7f94640cdf90 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f946001d3e0 tx=0x7f946001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f945affd640 1 -- 172.21.15.31:0/818728597 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600ba600 con 0x7f946406aeb0 2023-10-30T13:47:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f945affd640 1 -- 172.21.15.31:0/818728597 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94600ba780 con 0x7f946406aeb0 2023-10-30T13:47:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946415c9d0 con 0x7f946406aeb0 2023-10-30T13:47:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f945affd640 1 -- 172.21.15.31:0/818728597 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600bab30 con 0x7f946406aeb0 2023-10-30T13:47:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94640c7d00 con 0x7f946406aeb0 2023-10-30T13:47:21.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.072+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946404e170 con 0x7f946406aeb0 2023-10-30T13:47:21.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.076+0000 7f945affd640 1 -- 172.21.15.31:0/818728597 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94600bac90 con 0x7f946406aeb0 2023-10-30T13:47:21.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.076+0000 7f945affd640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94480664d0 0x7f9448068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:21.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.076+0000 7f945affd640 1 -- 172.21.15.31:0/818728597 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f9460122760 con 0x7f946406aeb0 2023-10-30T13:47:21.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.076+0000 7f9469c85640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94480664d0 0x7f9448068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:21.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.076+0000 7f9469c85640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94480664d0 0x7f9448068990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f94640ced90 tx=0x7f9454008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:21.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.080+0000 7f945affd640 1 -- 172.21.15.31:0/818728597 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94600ef810 con 0x7f946406aeb0 2023-10-30T13:47:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94640c7ef0 con 0x7f946406aeb0 2023-10-30T13:47:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f945affd640 1 -- 172.21.15.31:0/818728597 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f94600f2e30 con 0x7f946406aeb0 2023-10-30T13:47:21.273 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:21.273 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:19.124500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2023-10-30T13:47:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94480664d0 msgr2=0x7f9448068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94480664d0 0x7f9448068990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f94640ced90 tx=0x7f9454008040 comp rx=0 tx=0).stop 2023-10-30T13:47:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f946406aeb0 msgr2=0x7f94640cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f946406aeb0 0x7f94640cdf90 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f946001d3e0 tx=0x7f946001bad0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 shutdown_connections 2023-10-30T13:47:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94480664d0 0x7f9448068990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f946415ac70 0x7f946415b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f94640c8eb0 0x7f94640c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 --2- 172.21.15.31:0/818728597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f946406aeb0 0x7f94640cdf90 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:21.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 >> 172.21.15.31:0/818728597 conn(0x7f94640cef00 msgr2=0x7f9464152020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:21.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 shutdown_connections 2023-10-30T13:47:21.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:21.272+0000 7f946b488640 1 -- 172.21.15.31:0/818728597 wait complete. 2023-10-30T13:47:21.293 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:23.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.448+0000 7f700d4c9640 1 Processor -- start 2023-10-30T13:47:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.448+0000 7f700d4c9640 1 -- start start 2023-10-30T13:47:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f700d4c9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f70080c4a10 0x7f700815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f700d4c9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f70080c42a0 0x7f70080ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f700d4c9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7008159f20 con 0x7f7008153c80 2023-10-30T13:47:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f700d4c9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700815dec0 con 0x7f70080c4a10 2023-10-30T13:47:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f700d4c9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70081599a0 con 0x7f70080c42a0 2023-10-30T13:47:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f7006ffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7008153c80 0x7f700815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:38858/0 (socket says 172.21.15.31:38858) 2023-10-30T13:47:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f7006ffd640 1 -- 172.21.15.31:0/899343483 learned_addr learned my addr 172.21.15.31:0/899343483 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f6ffffff640 1 --2- 172.21.15.31:0/899343483 >> v2:172.21.15.31:3300/0 conn(0x7f70080c4a10 0x7f700815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f70077fe640 1 --2- 172.21.15.31:0/899343483 >> v2:172.21.15.33:3300/0 conn(0x7f70080c42a0 0x7f70080ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f70067fc640 1 -- 172.21.15.31:0/899343483 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129461042 0 0) 0x7f7008159f20 con 0x7f7008153c80 2023-10-30T13:47:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f70067fc640 1 -- 172.21.15.31:0/899343483 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff4003530 con 0x7f7008153c80 2023-10-30T13:47:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f6ffffff640 1 -- 172.21.15.31:0/899343483 >> v2:172.21.15.33:3300/0 conn(0x7f70080c42a0 msgr2=0x7f70080ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f6ffffff640 1 --2- 172.21.15.31:0/899343483 >> v2:172.21.15.33:3300/0 conn(0x7f70080c42a0 0x7f70080ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f6ffffff640 1 -- 172.21.15.31:0/899343483 >> v1:172.21.15.31:6789/0 conn(0x7f7008153c80 legacy=0x7f700815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f6ffffff640 1 -- 172.21.15.31:0/899343483 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700815d620 con 0x7f70080c4a10 2023-10-30T13:47:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f70077fe640 1 --2- 172.21.15.31:0/899343483 >> v2:172.21.15.33:3300/0 conn(0x7f70080c42a0 0x7f70080ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:23.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f6ffffff640 1 --2- 172.21.15.31:0/899343483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f700815cf90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f70080b1d40 tx=0x7f6ff001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7813e2dfe2d9f82a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:23.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f70067fc640 1 -- 172.21.15.31:0/899343483 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00a8950 con 0x7f70080c4a10 2023-10-30T13:47:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.452+0000 7f70067fc640 1 -- 172.21.15.31:0/899343483 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff00a8ab0 con 0x7f70080c4a10 2023-10-30T13:47:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f70067fc640 1 -- 172.21.15.31:0/899343483 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00a8e40 con 0x7f70080c4a10 2023-10-30T13:47:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- 172.21.15.31:0/899343483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 msgr2=0x7f700815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 --2- 172.21.15.31:0/899343483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f700815cf90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f70080b1d40 tx=0x7f6ff001ec50 comp rx=0 tx=0).stop 2023-10-30T13:47:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- 172.21.15.31:0/899343483 shutdown_connections 2023-10-30T13:47:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 --2- 172.21.15.31:0/899343483 >> v2:172.21.15.33:3300/0 conn(0x7f70080c42a0 0x7f70080ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 --2- 172.21.15.31:0/899343483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f700815cf90 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- 172.21.15.31:0/899343483 >> 172.21.15.31:0/899343483 conn(0x7f70080715c0 msgr2=0x7f70080719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:23.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- 172.21.15.31:0/899343483 shutdown_connections 2023-10-30T13:47:23.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- 172.21.15.31:0/899343483 wait complete. 2023-10-30T13:47:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 Processor -- start 2023-10-30T13:47:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- start start 2023-10-30T13:47:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70080c42a0 0x7f7008149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f70080c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7008153c80 0x7f70080c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70081ef5d0 con 0x7f70080c4a10 2023-10-30T13:47:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70081eea50 con 0x7f7008153c80 2023-10-30T13:47:23.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f700d4c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70081eed30 con 0x7f70080c42a0 2023-10-30T13:47:23.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f70080c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:23.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f70080c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49678/0 (socket says 172.21.15.31:49678) 2023-10-30T13:47:23.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f6ffffff640 1 -- 172.21.15.31:0/3035144300 learned_addr learned my addr 172.21.15.31:0/3035144300 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:23.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f7006ffd640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70080c42a0 0x7f7008149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:23.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f70077fe640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7008153c80 0x7f70080c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:23.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f6ffffff640 1 -- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70080c42a0 msgr2=0x7f7008149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f6ffffff640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70080c42a0 0x7f7008149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f6ffffff640 1 -- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7008153c80 msgr2=0x7f70080c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.456+0000 7f6ffffff640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7008153c80 0x7f70080c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6ffffff640 1 -- 172.21.15.31:0/3035144300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0009420 con 0x7f70080c4a10 2023-10-30T13:47:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6ffffff640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f70080c8770 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00b3040 tx=0x7f6ff001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6fff7fe640 1 -- 172.21.15.31:0/3035144300 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00b0df0 con 0x7f70080c4a10 2023-10-30T13:47:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6fff7fe640 1 -- 172.21.15.31:0/3035144300 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff00b9410 con 0x7f70080c4a10 2023-10-30T13:47:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6fff7fe640 1 -- 172.21.15.31:0/3035144300 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00b97a0 con 0x7f70080c4a10 2023-10-30T13:47:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70081ef010 con 0x7f70080c4a10 2023-10-30T13:47:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700814e0d0 con 0x7f70080c4a10 2023-10-30T13:47:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700814df30 con 0x7f70080c4a10 2023-10-30T13:47:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6fff7fe640 1 -- 172.21.15.31:0/3035144300 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ff00b7040 con 0x7f70080c4a10 2023-10-30T13:47:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6fff7fe640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6fe80664a0 0x7f6fe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f6fff7fe640 1 -- 172.21.15.31:0/3035144300 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f6ff0121d20 con 0x7f70080c4a10 2023-10-30T13:47:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.460+0000 7f7006ffd640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6fe80664a0 0x7f6fe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.464+0000 7f7006ffd640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6fe80664a0 0x7f6fe8068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80011f0 tx=0x7f6ff8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.464+0000 7f6fff7fe640 1 -- 172.21.15.31:0/3035144300 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f700814df30 con 0x7f70080c4a10 2023-10-30T13:47:23.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7008159b90 con 0x7f70080c4a10 2023-10-30T13:47:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f6fff7fe640 1 -- 172.21.15.31:0/3035144300 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f7008159b90 con 0x7f70080c4a10 2023-10-30T13:47:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:23.658 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:19.124500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:23.660 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2023-10-30T13:47:23.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6fe80664a0 msgr2=0x7f6fe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6fe80664a0 0x7f6fe8068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80011f0 tx=0x7f6ff8009040 comp rx=0 tx=0).stop 2023-10-30T13:47:23.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 msgr2=0x7f70080c8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f70080c8770 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00b3040 tx=0x7f6ff001b9f0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 shutdown_connections 2023-10-30T13:47:23.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6fe80664a0 0x7f6fe8068960 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7008153c80 0x7f70080c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70080c4a10 0x7f70080c8770 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 --2- 172.21.15.31:0/3035144300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70080c42a0 0x7f7008149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 >> 172.21.15.31:0/3035144300 conn(0x7f70080715c0 msgr2=0x7f7008158480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:23.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 shutdown_connections 2023-10-30T13:47:23.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:23.656+0000 7f700d4c9640 1 -- 172.21.15.31:0/3035144300 wait complete. 2023-10-30T13:47:23.680 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:25.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:25.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.832+0000 7ff413846640 1 Processor -- start 2023-10-30T13:47:25.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.832+0000 7ff413846640 1 -- start start 2023-10-30T13:47:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff413846640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff40c06b760 0x7ff40c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff413846640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff40c06bb40 0x7ff40c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff413846640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40c071e00 con 0x7ff40c06c420 2023-10-30T13:47:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff413846640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c070020 con 0x7ff40c06bb40 2023-10-30T13:47:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff413846640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c15eb90 con 0x7ff40c06b760 2023-10-30T13:47:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff411de3640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff40c06c420 0x7ff40c0c57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:53316/0 (socket says 172.21.15.31:53316) 2023-10-30T13:47:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff411de3640 1 -- 172.21.15.31:0/943059928 learned_addr learned my addr 172.21.15.31:0/943059928 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff410de1640 1 --2- 172.21.15.31:0/943059928 >> v2:172.21.15.31:3300/0 conn(0x7ff40c06bb40 0x7ff40c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff4115e2640 1 --2- 172.21.15.31:0/943059928 >> v2:172.21.15.33:3300/0 conn(0x7ff40c06b760 0x7ff40c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff403fff640 1 -- 172.21.15.31:0/943059928 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411305589 0 0) 0x7ff40c071e00 con 0x7ff40c06c420 2023-10-30T13:47:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff403fff640 1 -- 172.21.15.31:0/943059928 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8003530 con 0x7ff40c06c420 2023-10-30T13:47:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff410de1640 1 -- 172.21.15.31:0/943059928 >> v2:172.21.15.33:3300/0 conn(0x7ff40c06b760 msgr2=0x7ff40c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff410de1640 1 --2- 172.21.15.31:0/943059928 >> v2:172.21.15.33:3300/0 conn(0x7ff40c06b760 0x7ff40c15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff410de1640 1 -- 172.21.15.31:0/943059928 >> v1:172.21.15.31:6789/0 conn(0x7ff40c06c420 legacy=0x7ff40c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.836+0000 7ff410de1640 1 -- 172.21.15.31:0/943059928 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c070f10 con 0x7ff40c06bb40 2023-10-30T13:47:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff410de1640 1 --2- 172.21.15.31:0/943059928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06bb40 0x7ff40c157a20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0cb840 tx=0x7ff3fc01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c811a95f5ec9f02b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff403fff640 1 -- 172.21.15.31:0/943059928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc0ab650 con 0x7ff40c06bb40 2023-10-30T13:47:25.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff403fff640 1 -- 172.21.15.31:0/943059928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc0ab7b0 con 0x7ff40c06bb40 2023-10-30T13:47:25.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff403fff640 1 -- 172.21.15.31:0/943059928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc0abb40 con 0x7ff40c06bb40 2023-10-30T13:47:25.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 -- 172.21.15.31:0/943059928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06bb40 msgr2=0x7ff40c157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 --2- 172.21.15.31:0/943059928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06bb40 0x7ff40c157a20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0cb840 tx=0x7ff3fc01c430 comp rx=0 tx=0).stop 2023-10-30T13:47:25.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 -- 172.21.15.31:0/943059928 shutdown_connections 2023-10-30T13:47:25.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 --2- 172.21.15.31:0/943059928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06bb40 0x7ff40c157a20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 --2- 172.21.15.31:0/943059928 >> v2:172.21.15.33:3300/0 conn(0x7ff40c06b760 0x7ff40c15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 -- 172.21.15.31:0/943059928 >> 172.21.15.31:0/943059928 conn(0x7ff40c0cac70 msgr2=0x7ff40c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:25.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 -- 172.21.15.31:0/943059928 shutdown_connections 2023-10-30T13:47:25.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 -- 172.21.15.31:0/943059928 wait complete. 2023-10-30T13:47:25.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 Processor -- start 2023-10-30T13:47:25.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.840+0000 7ff413846640 1 -- start start 2023-10-30T13:47:25.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06b760 0x7ff40c149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff40c06bb40 0x7ff40c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff40c06c420 0x7ff40c15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c15dc30 con 0x7ff40c06b760 2023-10-30T13:47:25.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c15c200 con 0x7ff40c06bb40 2023-10-30T13:47:25.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c14a550 con 0x7ff40c06c420 2023-10-30T13:47:25.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06b760 0x7ff40c149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06b760 0x7ff40c149200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49066/0 (socket says 172.21.15.31:49066) 2023-10-30T13:47:25.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 -- 172.21.15.31:0/407187853 learned_addr learned my addr 172.21.15.31:0/407187853 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:25.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff410de1640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff40c06bb40 0x7ff40c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff411de3640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff40c06c420 0x7ff40c15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 -- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff40c06c420 msgr2=0x7ff40c15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff40c06c420 0x7ff40c15cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 -- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff40c06bb40 msgr2=0x7ff40c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff40c06bb40 0x7ff40c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 -- 172.21.15.31:0/407187853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc009420 con 0x7ff40c06b760 2023-10-30T13:47:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff4115e2640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06b760 0x7ff40c149200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff40800e930 tx=0x7ff40800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff401ffb640 1 -- 172.21.15.31:0/407187853 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff408009690 con 0x7ff40c06b760 2023-10-30T13:47:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff401ffb640 1 -- 172.21.15.31:0/407187853 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40800ca00 con 0x7ff40c06b760 2023-10-30T13:47:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff401ffb640 1 -- 172.21.15.31:0/407187853 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800cd70 con 0x7ff40c06b760 2023-10-30T13:47:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40c15d670 con 0x7ff40c06b760 2023-10-30T13:47:25.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40c14a9d0 con 0x7ff40c06b760 2023-10-30T13:47:25.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40c04e170 con 0x7ff40c06b760 2023-10-30T13:47:25.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.844+0000 7ff401ffb640 1 -- 172.21.15.31:0/407187853 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff408013470 con 0x7ff40c06b760 2023-10-30T13:47:25.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.848+0000 7ff401ffb640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff3f00664a0 0x7ff3f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.848+0000 7ff401ffb640 1 -- 172.21.15.31:0/407187853 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7ff408084480 con 0x7ff40c06b760 2023-10-30T13:47:25.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.848+0000 7ff410de1640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff3f00664a0 0x7ff3f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.848+0000 7ff410de1640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff3f00664a0 0x7ff3f0068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0af000 tx=0x7ff3fc0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:25.852+0000 7ff401ffb640 1 -- 172.21.15.31:0/407187853 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4080515f0 con 0x7ff40c06b760 2023-10-30T13:47:26.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.040+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff40c15b580 con 0x7ff40c06b760 2023-10-30T13:47:26.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.040+0000 7ff401ffb640 1 -- 172.21.15.31:0/407187853 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5696 (secure 0 0 0) 0x7ff408054c10 con 0x7ff40c06b760 2023-10-30T13:47:26.042 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:26.042 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:24.289033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:26.043 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 62 2023-10-30T13:47:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff3f00664a0 msgr2=0x7ff3f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff3f00664a0 0x7ff3f0068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0af000 tx=0x7ff3fc0a7000 comp rx=0 tx=0).stop 2023-10-30T13:47:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06b760 msgr2=0x7ff40c149200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06b760 0x7ff40c149200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff40800e930 tx=0x7ff40800edf0 comp rx=0 tx=0).stop 2023-10-30T13:47:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 shutdown_connections 2023-10-30T13:47:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff3f00664a0 0x7ff3f0068960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff40c06c420 0x7ff40c15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff40c06bb40 0x7ff40c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 --2- 172.21.15.31:0/407187853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff40c06b760 0x7ff40c149200 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 >> 172.21.15.31:0/407187853 conn(0x7ff40c0cac70 msgr2=0x7ff40c1550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 shutdown_connections 2023-10-30T13:47:26.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:26.044+0000 7ff413846640 1 -- 172.21.15.31:0/407187853 wait complete. 2023-10-30T13:47:26.063 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5842": { "addr": "172.21.15.33:6836/15857146", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 15857146, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 15857146, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5842, "incarnation": 60, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:47:24.289033+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5842 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:47:26.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:26.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:28.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:28.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.216+0000 7f74124ea640 1 Processor -- start 2023-10-30T13:47:28.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.216+0000 7f74124ea640 1 -- start start 2023-10-30T13:47:28.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f74124ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f740c0c5c10 0x7f740c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f74124ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f74124ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740c15cd50 con 0x7f740c15da00 2023-10-30T13:47:28.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f74124ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740c15d0a0 con 0x7f740c0c5c10 2023-10-30T13:47:28.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f74124ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740c0ca000 con 0x7f740c159760 2023-10-30T13:47:28.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740bfff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f740c0c5c10 0x7f740c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740b7fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740b7fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c15d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:47192/0 (socket says 172.21.15.31:47192) 2023-10-30T13:47:28.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740b7fe640 1 -- 172.21.15.31:0/1020222504 learned_addr learned my addr 172.21.15.31:0/1020222504 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:28.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f7410a87640 1 --1- 172.21.15.31:0/1020222504 >> v1:172.21.15.33:6789/0 conn(0x7f740c15da00 0x7f740c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:41720/0 (socket says 172.21.15.31:41720) 2023-10-30T13:47:28.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740b7fe640 1 -- 172.21.15.31:0/1020222504 >> v2:172.21.15.33:3300/0 conn(0x7f740c0c5c10 msgr2=0x7f740c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:28.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740b7fe640 1 --2- 172.21.15.31:0/1020222504 >> v2:172.21.15.33:3300/0 conn(0x7f740c0c5c10 0x7f740c06eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740b7fe640 1 -- 172.21.15.31:0/1020222504 >> v1:172.21.15.33:6789/0 conn(0x7f740c15da00 legacy=0x7f740c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740b7fe640 1 -- 172.21.15.31:0/1020222504 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c159cf0 con 0x7f740c159760 2023-10-30T13:47:28.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740affd640 1 -- 172.21.15.31:0/1020222504 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800312621 0 0) 0x7f740c15cd50 con 0x7f740c15da00 2023-10-30T13:47:28.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.220+0000 7f740bfff640 1 --2- 172.21.15.31:0/1020222504 >> v2:172.21.15.33:3300/0 conn(0x7f740c0c5c10 0x7f740c06eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:47:28.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f740b7fe640 1 --2- 172.21.15.31:0/1020222504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c15d4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7400009870 tx=0x7f740001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72b001d9b7d4d4e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:28.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f740affd640 1 -- 172.21.15.31:0/1020222504 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74000aa670 con 0x7f740c159760 2023-10-30T13:47:28.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f740affd640 1 -- 172.21.15.31:0/1020222504 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000aa7d0 con 0x7f740c159760 2023-10-30T13:47:28.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f740affd640 1 -- 172.21.15.31:0/1020222504 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74000aab60 con 0x7f740c159760 2023-10-30T13:47:28.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 -- 172.21.15.31:0/1020222504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 msgr2=0x7f740c15d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 --2- 172.21.15.31:0/1020222504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c15d4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7400009870 tx=0x7f740001c510 comp rx=0 tx=0).stop 2023-10-30T13:47:28.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 -- 172.21.15.31:0/1020222504 shutdown_connections 2023-10-30T13:47:28.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 --2- 172.21.15.31:0/1020222504 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c15d4c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 --2- 172.21.15.31:0/1020222504 >> v2:172.21.15.33:3300/0 conn(0x7f740c0c5c10 0x7f740c06eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 -- 172.21.15.31:0/1020222504 >> 172.21.15.31:0/1020222504 conn(0x7f740c06afa0 msgr2=0x7f740c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:28.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 -- 172.21.15.31:0/1020222504 shutdown_connections 2023-10-30T13:47:28.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 -- 172.21.15.31:0/1020222504 wait complete. 2023-10-30T13:47:28.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 Processor -- start 2023-10-30T13:47:28.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.224+0000 7f74124ea640 1 -- start start 2023-10-30T13:47:28.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f740c0c5c10 0x7f740c15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f740c15da00 0x7f740c0c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c14e550 con 0x7f740c15da00 2023-10-30T13:47:28.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c14efb0 con 0x7f740c0c5c10 2023-10-30T13:47:28.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740c14ecf0 con 0x7f740c159760 2023-10-30T13:47:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f740c15da00 0x7f740c0c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f740c15da00 0x7f740c0c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49074/0 (socket says 172.21.15.31:49074) 2023-10-30T13:47:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 -- 172.21.15.31:0/2865458152 learned_addr learned my addr 172.21.15.31:0/2865458152 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f740bfff640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f740c0c5c10 0x7f740c15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f740b7fe640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 -- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 msgr2=0x7f740c148fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c148fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 -- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f740c0c5c10 msgr2=0x7f740c15c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f740c0c5c10 0x7f740c15c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 -- 172.21.15.31:0/2865458152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc009420 con 0x7f740c15da00 2023-10-30T13:47:28.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7410a87640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f740c15da00 0x7f740c0c34a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f740400c780 tx=0x7f740400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:28.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7408ff9640 1 -- 172.21.15.31:0/2865458152 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740400ce30 con 0x7f740c15da00 2023-10-30T13:47:28.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7408ff9640 1 -- 172.21.15.31:0/2865458152 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740400a640 con 0x7f740c15da00 2023-10-30T13:47:28.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74000094d0 con 0x7f740c15da00 2023-10-30T13:47:28.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7408ff9640 1 -- 172.21.15.31:0/2865458152 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740400aa30 con 0x7f740c15da00 2023-10-30T13:47:28.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740c14ea10 con 0x7f740c15da00 2023-10-30T13:47:28.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740c04e170 con 0x7f740c15da00 2023-10-30T13:47:28.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.228+0000 7f7408ff9640 1 -- 172.21.15.31:0/2865458152 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f740400ab90 con 0x7f740c15da00 2023-10-30T13:47:28.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.232+0000 7f7408ff9640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73f80664d0 0x7f73f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.232+0000 7f7408ff9640 1 -- 172.21.15.31:0/2865458152 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f7404083930 con 0x7f740c15da00 2023-10-30T13:47:28.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.232+0000 7f740bfff640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73f80664d0 0x7f73f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.232+0000 7f740bfff640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73f80664d0 0x7f73f8068990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009c50 tx=0x7f73fc009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:28.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.236+0000 7f7408ff9640 1 -- 172.21.15.31:0/2865458152 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7404050b20 con 0x7f740c15da00 2023-10-30T13:47:28.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.424+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f740c06b520 con 0x7f740c15da00 2023-10-30T13:47:28.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f7408ff9640 1 -- 172.21.15.31:0/2865458152 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5696 (secure 0 0 0) 0x7f7404018090 con 0x7f740c15da00 2023-10-30T13:47:28.427 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:28.428 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:24.289033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:28.428 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 62 2023-10-30T13:47:28.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73f80664d0 msgr2=0x7f73f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73f80664d0 0x7f73f8068990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009c50 tx=0x7f73fc009190 comp rx=0 tx=0).stop 2023-10-30T13:47:28.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f740c15da00 msgr2=0x7f740c0c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f740c15da00 0x7f740c0c34a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f740400c780 tx=0x7f740400cc40 comp rx=0 tx=0).stop 2023-10-30T13:47:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 shutdown_connections 2023-10-30T13:47:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73f80664d0 0x7f73f8068990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f740c15da00 0x7f740c0c34a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f740c159760 0x7f740c148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 --2- 172.21.15.31:0/2865458152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f740c0c5c10 0x7f740c15c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 >> 172.21.15.31:0/2865458152 conn(0x7f740c06afa0 msgr2=0x7f740c072600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 shutdown_connections 2023-10-30T13:47:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:28.428+0000 7f74124ea640 1 -- 172.21.15.31:0/2865458152 wait complete. 2023-10-30T13:47:28.449 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:30.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:30.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a46e54640 1 Processor -- start 2023-10-30T13:47:30.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a46e54640 1 -- start start 2023-10-30T13:47:30.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a46e54640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0a4006b3b0 0x7f0a4006b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a46e54640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a400c7b00 con 0x7f0a4015d840 2023-10-30T13:47:30.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a46e54640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4006bd20 con 0x7f0a40153aa0 2023-10-30T13:47:30.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a46e54640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a40070a50 con 0x7f0a4006b3b0 2023-10-30T13:47:30.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a3ffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f0a4015d840 0x7f0a400ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:53324/0 (socket says 172.21.15.31:53324) 2023-10-30T13:47:30.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a3ffff640 1 -- 172.21.15.31:0/929065089 learned_addr learned my addr 172.21.15.31:0/929065089 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:30.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.604+0000 7f0a453f1640 1 --2- 172.21.15.31:0/929065089 >> v2:172.21.15.31:3300/0 conn(0x7f0a4006b3b0 0x7f0a4006b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a3f7fe640 1 -- 172.21.15.31:0/929065089 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143687892 0 0) 0x7f0a400c7b00 con 0x7f0a4015d840 2023-10-30T13:47:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a3f7fe640 1 -- 172.21.15.31:0/929065089 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a30003530 con 0x7f0a4015d840 2023-10-30T13:47:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a453f1640 1 -- 172.21.15.31:0/929065089 >> v1:172.21.15.33:6789/0 conn(0x7f0a40153aa0 legacy=0x7f0a400701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a453f1640 1 -- 172.21.15.31:0/929065089 >> v1:172.21.15.31:6789/0 conn(0x7f0a4015d840 legacy=0x7f0a400ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a453f1640 1 -- 172.21.15.31:0/929065089 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a400c2050 con 0x7f0a4006b3b0 2023-10-30T13:47:30.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a453f1640 1 --2- 172.21.15.31:0/929065089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a4006b7e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009d90 tx=0x7f0a3801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab338f68df7570a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:30.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a3f7fe640 1 -- 172.21.15.31:0/929065089 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a380af600 con 0x7f0a4006b3b0 2023-10-30T13:47:30.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a3f7fe640 1 -- 172.21.15.31:0/929065089 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a380ab050 con 0x7f0a4006b3b0 2023-10-30T13:47:30.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a3f7fe640 1 -- 172.21.15.31:0/929065089 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a380af980 con 0x7f0a4006b3b0 2023-10-30T13:47:30.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a46e54640 1 -- 172.21.15.31:0/929065089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 msgr2=0x7f0a4006b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.608+0000 7f0a46e54640 1 --2- 172.21.15.31:0/929065089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a4006b7e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009d90 tx=0x7f0a3801f130 comp rx=0 tx=0).stop 2023-10-30T13:47:30.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- 172.21.15.31:0/929065089 shutdown_connections 2023-10-30T13:47:30.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 --2- 172.21.15.31:0/929065089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a4006b7e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- 172.21.15.31:0/929065089 >> 172.21.15.31:0/929065089 conn(0x7f0a40071fb0 msgr2=0x7f0a400723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- 172.21.15.31:0/929065089 shutdown_connections 2023-10-30T13:47:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- 172.21.15.31:0/929065089 wait complete. 2023-10-30T13:47:30.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 Processor -- start 2023-10-30T13:47:30.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- start start 2023-10-30T13:47:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a40150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a40153aa0 0x7f0a401514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0a4015d840 0x7f0a40156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40157d60 con 0x7f0a4006b3b0 2023-10-30T13:47:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a401503c0 con 0x7f0a40153aa0 2023-10-30T13:47:30.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a46e54640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40158910 con 0x7f0a4015d840 2023-10-30T13:47:30.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a40150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a40150a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49092/0 (socket says 172.21.15.31:49092) 2023-10-30T13:47:30.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 -- 172.21.15.31:0/864988333 learned_addr learned my addr 172.21.15.31:0/864988333 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:30.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a3ffff640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a40153aa0 0x7f0a401514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a453f1640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0a4015d840 0x7f0a40156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 -- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0a4015d840 msgr2=0x7f0a40156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0a4015d840 0x7f0a40156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 -- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a40153aa0 msgr2=0x7f0a401514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a40153aa0 0x7f0a401514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 -- 172.21.15.31:0/864988333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38009a70 con 0x7f0a4006b3b0 2023-10-30T13:47:30.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.612+0000 7f0a44bf0640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a40150a50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c530 tx=0x7f0a3400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:30.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a3d7fa640 1 -- 172.21.15.31:0/864988333 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34005220 con 0x7f0a4006b3b0 2023-10-30T13:47:30.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a3d7fa640 1 -- 172.21.15.31:0/864988333 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a34005380 con 0x7f0a4006b3b0 2023-10-30T13:47:30.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4014f460 con 0x7f0a4006b3b0 2023-10-30T13:47:30.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a3d7fa640 1 -- 172.21.15.31:0/864988333 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34007270 con 0x7f0a4006b3b0 2023-10-30T13:47:30.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4014fa20 con 0x7f0a4006b3b0 2023-10-30T13:47:30.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4004e170 con 0x7f0a4006b3b0 2023-10-30T13:47:30.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a3d7fa640 1 -- 172.21.15.31:0/864988333 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a340073d0 con 0x7f0a4006b3b0 2023-10-30T13:47:30.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a3d7fa640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0a240664a0 0x7f0a24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a3d7fa640 1 -- 172.21.15.31:0/864988333 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f0a34083990 con 0x7f0a4006b3b0 2023-10-30T13:47:30.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.616+0000 7f0a3ffff640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0a240664a0 0x7f0a24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.620+0000 7f0a3ffff640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0a240664a0 0x7f0a24068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0a4006b960 tx=0x7f0a2c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:30.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.620+0000 7f0a3d7fa640 1 -- 172.21.15.31:0/864988333 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a34050c00 con 0x7f0a4006b3b0 2023-10-30T13:47:30.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a400b2fd0 con 0x7f0a4006b3b0 2023-10-30T13:47:30.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a3d7fa640 1 -- 172.21.15.31:0/864988333 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5696 (secure 0 0 0) 0x7f0a34054220 con 0x7f0a4006b3b0 2023-10-30T13:47:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:30.813 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:24.289033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:30.813 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 62 2023-10-30T13:47:30.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0a240664a0 msgr2=0x7f0a24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0a240664a0 0x7f0a24068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0a4006b960 tx=0x7f0a2c009040 comp rx=0 tx=0).stop 2023-10-30T13:47:30.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 msgr2=0x7f0a40150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a40150a50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c530 tx=0x7f0a3400c9f0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 shutdown_connections 2023-10-30T13:47:30.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0a240664a0 0x7f0a24068960 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0a4015d840 0x7f0a40156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a40153aa0 0x7f0a401514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 --2- 172.21.15.31:0/864988333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a4006b3b0 0x7f0a40150a50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 >> 172.21.15.31:0/864988333 conn(0x7f0a40071fb0 msgr2=0x7f0a40152a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:30.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 shutdown_connections 2023-10-30T13:47:30.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:30.812+0000 7f0a46e54640 1 -- 172.21.15.31:0/864988333 wait complete. 2023-10-30T13:47:30.834 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:32.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.988+0000 7fd16a31d640 1 Processor -- start 2023-10-30T13:47:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.988+0000 7fd16a31d640 1 -- start start 2023-10-30T13:47:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.988+0000 7fd16a31d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd164153bb0 0x7fd1640c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.988+0000 7fd16a31d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd16406c420 0x7fd1640c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd16a31d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640c5f80 con 0x7fd1640c7170 2023-10-30T13:47:32.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd16a31d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640c5690 con 0x7fd16406c420 2023-10-30T13:47:32.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd16a31d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16415a580 con 0x7fd164153bb0 2023-10-30T13:47:32.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd163fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd164153bb0 0x7fd1640c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd1637fe640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd16406c420 0x7fd1640c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd163fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd164153bb0 0x7fd1640c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:47204/0 (socket says 172.21.15.31:47204) 2023-10-30T13:47:32.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd163fff640 1 -- 172.21.15.31:0/3187425460 learned_addr learned my addr 172.21.15.31:0/3187425460 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:32.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd1688ba640 1 --1- 172.21.15.31:0/3187425460 >> v1:172.21.15.33:6789/0 conn(0x7fd1640c7170 0x7fd164159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:41744/0 (socket says 172.21.15.31:41744) 2023-10-30T13:47:32.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd1637fe640 1 -- 172.21.15.31:0/3187425460 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd164153bb0 msgr2=0x7fd1640c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:32.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd1637fe640 1 --2- 172.21.15.31:0/3187425460 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd164153bb0 0x7fd1640c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd1637fe640 1 -- 172.21.15.31:0/3187425460 >> v1:172.21.15.33:6789/0 conn(0x7fd1640c7170 legacy=0x7fd164159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:32.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.992+0000 7fd1637fe640 1 -- 172.21.15.31:0/3187425460 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640c5930 con 0x7fd16406c420 2023-10-30T13:47:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd1637fe640 1 --2- 172.21.15.31:0/3187425460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd16406c420 0x7fd1640c6c30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd1580097c0 tx=0x7fd15801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac2ad5632f961568 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:32.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd162ffd640 1 -- 172.21.15.31:0/3187425460 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580a8970 con 0x7fd16406c420 2023-10-30T13:47:32.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd162ffd640 1 -- 172.21.15.31:0/3187425460 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1580a8ad0 con 0x7fd16406c420 2023-10-30T13:47:32.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd162ffd640 1 -- 172.21.15.31:0/3187425460 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580a8e60 con 0x7fd16406c420 2023-10-30T13:47:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 -- 172.21.15.31:0/3187425460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd16406c420 msgr2=0x7fd1640c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 --2- 172.21.15.31:0/3187425460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd16406c420 0x7fd1640c6c30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd1580097c0 tx=0x7fd15801ec70 comp rx=0 tx=0).stop 2023-10-30T13:47:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 -- 172.21.15.31:0/3187425460 shutdown_connections 2023-10-30T13:47:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 --2- 172.21.15.31:0/3187425460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd16406c420 0x7fd1640c6c30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 --2- 172.21.15.31:0/3187425460 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd164153bb0 0x7fd1640c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 -- 172.21.15.31:0/3187425460 >> 172.21.15.31:0/3187425460 conn(0x7fd1640cac70 msgr2=0x7fd1640cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 -- 172.21.15.31:0/3187425460 shutdown_connections 2023-10-30T13:47:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 -- 172.21.15.31:0/3187425460 wait complete. 2023-10-30T13:47:33.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 Processor -- start 2023-10-30T13:47:33.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:32.996+0000 7fd16a31d640 1 -- start start 2023-10-30T13:47:33.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd16406c420 0x7fd1640c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:33.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1640c7170 0x7fd1640bf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:33.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd164153bb0 0x7fd1640bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:33.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640bd4e0 con 0x7fd1640c7170 2023-10-30T13:47:33.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640bd7c0 con 0x7fd164153bb0 2023-10-30T13:47:33.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640bdaa0 con 0x7fd16406c420 2023-10-30T13:47:33.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1640c7170 0x7fd1640bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:33.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1640c7170 0x7fd1640bf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49104/0 (socket says 172.21.15.31:49104) 2023-10-30T13:47:33.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 -- 172.21.15.31:0/2356814955 learned_addr learned my addr 172.21.15.31:0/2356814955 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:33.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1688ba640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd164153bb0 0x7fd1640bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:33.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd163fff640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd16406c420 0x7fd1640c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:33.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 -- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd16406c420 msgr2=0x7fd1640c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:33.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd16406c420 0x7fd1640c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 -- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd164153bb0 msgr2=0x7fd1640bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:33.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd164153bb0 0x7fd1640bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 -- 172.21.15.31:0/2356814955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd158009420 con 0x7fd1640c7170 2023-10-30T13:47:33.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd1637fe640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1640c7170 0x7fd1640bf330 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd1580b3040 tx=0x7fd15801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:33.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd160ff9640 1 -- 172.21.15.31:0/2356814955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580b9630 con 0x7fd1640c7170 2023-10-30T13:47:33.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd160ff9640 1 -- 172.21.15.31:0/2356814955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1580b97b0 con 0x7fd1640c7170 2023-10-30T13:47:33.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd160ff9640 1 -- 172.21.15.31:0/2356814955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580b9b60 con 0x7fd1640c7170 2023-10-30T13:47:33.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1640bdd80 con 0x7fd1640c7170 2023-10-30T13:47:33.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1640be340 con 0x7fd1640c7170 2023-10-30T13:47:33.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.000+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd164154250 con 0x7fd1640c7170 2023-10-30T13:47:33.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.004+0000 7fd160ff9640 1 -- 172.21.15.31:0/2356814955 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1580b7040 con 0x7fd1640c7170 2023-10-30T13:47:33.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.004+0000 7fd160ff9640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:33.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.004+0000 7fd160ff9640 1 -- 172.21.15.31:0/2356814955 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fd158122150 con 0x7fd1640c7170 2023-10-30T13:47:33.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.004+0000 7fd163fff640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:33.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.004+0000 7fd163fff640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1500664a0 0x7fd150068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd154009ae0 tx=0x7fd154008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:33.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.008+0000 7fd160ff9640 1 -- 172.21.15.31:0/2356814955 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1580ef200 con 0x7fd1640c7170 2023-10-30T13:47:33.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.196+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd164066510 con 0x7fd1640c7170 2023-10-30T13:47:33.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.196+0000 7fd160ff9640 1 -- 172.21.15.31:0/2356814955 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fd1580b5020 con 0x7fd1640c7170 2023-10-30T13:47:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:33.199 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:30.446421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":42,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:33.199 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 63 2023-10-30T13:47:33.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1500664a0 msgr2=0x7fd150068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:33.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1500664a0 0x7fd150068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd154009ae0 tx=0x7fd154008040 comp rx=0 tx=0).stop 2023-10-30T13:47:33.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1640c7170 msgr2=0x7fd1640bf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:33.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1640c7170 0x7fd1640bf330 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd1580b3040 tx=0x7fd15801ba10 comp rx=0 tx=0).stop 2023-10-30T13:47:33.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 shutdown_connections 2023-10-30T13:47:33.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd164153bb0 0x7fd1640bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1640c7170 0x7fd1640bf330 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 --2- 172.21.15.31:0/2356814955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd16406c420 0x7fd1640c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 >> 172.21.15.31:0/2356814955 conn(0x7fd1640cac70 msgr2=0x7fd16406b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:33.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 shutdown_connections 2023-10-30T13:47:33.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:33.200+0000 7fd16a31d640 1 -- 172.21.15.31:0/2356814955 wait complete. 2023-10-30T13:47:33.220 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:35.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:35.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.372+0000 7f46bba91640 1 Processor -- start 2023-10-30T13:47:35.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.372+0000 7f46bba91640 1 -- start start 2023-10-30T13:47:35.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46bba91640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f46b40ccf60 0x7f46b40cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46bba91640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b40c2500 0x7f46b40c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46bba91640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b4151aa0 con 0x7f46b41516c0 2023-10-30T13:47:35.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46bba91640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b40c74a0 con 0x7f46b40ccf60 2023-10-30T13:47:35.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46bba91640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40c2e70 con 0x7f46b40c2500 2023-10-30T13:47:35.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46ba02e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b40c2500 0x7f46b40c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46b902c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f46b40ccf60 0x7f46b40cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46b982d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f46b41516c0 0x7f46b40c7d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:38064/0 (socket says 172.21.15.31:38064) 2023-10-30T13:47:35.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46ba02e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b40c2500 0x7f46b40c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:33234/0 (socket says 172.21.15.31:33234) 2023-10-30T13:47:35.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46b982d640 1 -- 172.21.15.31:0/582747867 learned_addr learned my addr 172.21.15.31:0/582747867 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46ba02e640 1 -- 172.21.15.31:0/582747867 >> v2:172.21.15.33:3300/0 conn(0x7f46b40ccf60 msgr2=0x7f46b40cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46ba02e640 1 --2- 172.21.15.31:0/582747867 >> v2:172.21.15.33:3300/0 conn(0x7f46b40ccf60 0x7f46b40cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46ba02e640 1 -- 172.21.15.31:0/582747867 >> v1:172.21.15.33:6789/0 conn(0x7f46b41516c0 legacy=0x7f46b40c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.376+0000 7f46ba02e640 1 -- 172.21.15.31:0/582747867 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4158af0 con 0x7f46b40c2500 2023-10-30T13:47:35.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46ba02e640 1 --2- 172.21.15.31:0/582747867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b40c2500 0x7f46b40c2930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009d90 tx=0x7f46ac01c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=858970e1ee809bd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:35.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46b882b640 1 -- 172.21.15.31:0/582747867 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac0a5810 con 0x7f46b40c2500 2023-10-30T13:47:35.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46b882b640 1 -- 172.21.15.31:0/582747867 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac0a5970 con 0x7f46b40c2500 2023-10-30T13:47:35.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46b882b640 1 -- 172.21.15.31:0/582747867 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac0a5d00 con 0x7f46b40c2500 2023-10-30T13:47:35.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 -- 172.21.15.31:0/582747867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b40c2500 msgr2=0x7f46b40c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 --2- 172.21.15.31:0/582747867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b40c2500 0x7f46b40c2930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009d90 tx=0x7f46ac01c950 comp rx=0 tx=0).stop 2023-10-30T13:47:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 -- 172.21.15.31:0/582747867 shutdown_connections 2023-10-30T13:47:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 --2- 172.21.15.31:0/582747867 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b40c2500 0x7f46b40c2930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 --2- 172.21.15.31:0/582747867 >> v2:172.21.15.33:3300/0 conn(0x7f46b40ccf60 0x7f46b40cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 -- 172.21.15.31:0/582747867 >> 172.21.15.31:0/582747867 conn(0x7f46b4071fb0 msgr2=0x7f46b40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 -- 172.21.15.31:0/582747867 shutdown_connections 2023-10-30T13:47:35.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.380+0000 7f46bba91640 1 -- 172.21.15.31:0/582747867 wait complete. 2023-10-30T13:47:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 Processor -- start 2023-10-30T13:47:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 -- start start 2023-10-30T13:47:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b40c2500 0x7f46b4160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b40ccf60 0x7f46b41f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b41516c0 0x7f46b40c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40c8c70 con 0x7f46b40c2500 2023-10-30T13:47:35.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40c9270 con 0x7f46b40ccf60 2023-10-30T13:47:35.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b415b620 con 0x7f46b41516c0 2023-10-30T13:47:35.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b40c2500 0x7f46b4160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b40c2500 0x7f46b4160bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43500/0 (socket says 172.21.15.31:43500) 2023-10-30T13:47:35.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 -- 172.21.15.31:0/2249270185 learned_addr learned my addr 172.21.15.31:0/2249270185 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:35.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46ba02e640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b41516c0 0x7f46b40c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b902c640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b40ccf60 0x7f46b41f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 -- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b41516c0 msgr2=0x7f46b40c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b41516c0 0x7f46b40c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 -- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b40ccf60 msgr2=0x7f46b41f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b40ccf60 0x7f46b41f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 -- 172.21.15.31:0/2249270185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac009a70 con 0x7f46b40c2500 2023-10-30T13:47:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46b982d640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b40c2500 0x7f46b4160bb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c420 tx=0x7f46b000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:35.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46aa7fc640 1 -- 172.21.15.31:0/2249270185 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0005cf0 con 0x7f46b40c2500 2023-10-30T13:47:35.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46aa7fc640 1 -- 172.21.15.31:0/2249270185 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b0005e50 con 0x7f46b40c2500 2023-10-30T13:47:35.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46aa7fc640 1 -- 172.21.15.31:0/2249270185 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b00072f0 con 0x7f46b40c2500 2023-10-30T13:47:35.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b415b000 con 0x7f46b40c2500 2023-10-30T13:47:35.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b41f5260 con 0x7f46b40c2500 2023-10-30T13:47:35.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.384+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b40c2c60 con 0x7f46b40c2500 2023-10-30T13:47:35.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.388+0000 7f46aa7fc640 1 -- 172.21.15.31:0/2249270185 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46b0007450 con 0x7f46b40c2500 2023-10-30T13:47:35.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.388+0000 7f46aa7fc640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f46980664a0 0x7f4698068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.388+0000 7f46aa7fc640 1 -- 172.21.15.31:0/2249270185 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f46b0083770 con 0x7f46b40c2500 2023-10-30T13:47:35.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.388+0000 7f46b902c640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f46980664a0 0x7f4698068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.388+0000 7f46b902c640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f46980664a0 0x7f4698068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f46b41610f0 tx=0x7f46a4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:35.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.392+0000 7f46aa7fc640 1 -- 172.21.15.31:0/2249270185 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f46b00508b0 con 0x7f46b40c2500 2023-10-30T13:47:35.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.580+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46b404e170 con 0x7f46b40c2500 2023-10-30T13:47:35.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46aa7fc640 1 -- 172.21.15.31:0/2249270185 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f46b0017090 con 0x7f46b40c2500 2023-10-30T13:47:35.583 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:35.584 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:30.446421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":42,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:35.584 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 63 2023-10-30T13:47:35.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f46980664a0 msgr2=0x7f4698068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f46980664a0 0x7f4698068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f46b41610f0 tx=0x7f46a4008040 comp rx=0 tx=0).stop 2023-10-30T13:47:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b40c2500 msgr2=0x7f46b4160bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b40c2500 0x7f46b4160bb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c420 tx=0x7f46b000c8e0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 shutdown_connections 2023-10-30T13:47:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f46980664a0 0x7f4698068960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b41516c0 0x7f46b40c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b40ccf60 0x7f46b41f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 --2- 172.21.15.31:0/2249270185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b40c2500 0x7f46b4160bb0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 >> 172.21.15.31:0/2249270185 conn(0x7f46b4071fb0 msgr2=0x7f46b4155a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 shutdown_connections 2023-10-30T13:47:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:35.584+0000 7f46bba91640 1 -- 172.21.15.31:0/2249270185 wait complete. 2023-10-30T13:47:35.604 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:36.842 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 2% (249/12071) Updating files: 3% (363/12071) Updating files: 4% (483/12071) Updating files: 4% (512/12071) Updating files: 5% (604/12071) Updating files: 6% (725/12071) Updating files: 6% (835/12071) Updating files: 7% (845/12071) Updating files: 8% (966/12071) Updating files: 8% (1056/12071) Updating files: 9% (1087/12071) Updating files: 9% (1151/12071) Updating files: 9% (1176/12071) Updating files: 10% (1208/12071) Updating files: 10% (1271/12071) Updating files: 10% (1293/12071) Updating files: 11% (1328/12071) Updating files: 11% (1354/12071) Updating files: 11% (1428/12071) Updating files: 12% (1449/12071) Updating files: 12% (1486/12071) Updating files: 13% (1570/12071) Updating files: 14% (1690/12071) Updating files: 14% (1731/12071) Updating files: 15% (1811/12071) Updating files: 15% (1909/12071) Updating files: 16% (1932/12071) Updating files: 16% (2040/12071) Updating files: 17% (2053/12071) Updating files: 17% (2157/12071) Updating files: 18% (2173/12071) Updating files: 18% (2184/12071) Updating files: 19% (2294/12071) Updating files: 19% (2309/12071) Updating files: 20% (2415/12071) Updating files: 20% (2440/12071) Updating files: 21% (2535/12071) Updating files: 21% (2611/12071) Updating files: 22% (2656/12071) Updating files: 23% (2777/12071) Updating files: 23% (2814/12071) Updating files: 23% (2894/12071) Updating files: 24% (2898/12071) Updating files: 25% (3018/12071) Updating files: 25% (3033/12071) Updating files: 26% (3139/12071) Updating files: 26% (3205/12071) Updating files: 26% (3244/12071) Updating files: 27% (3260/12071) Updating files: 27% (3343/12071) Updating files: 28% (3380/12071) Updating files: 28% (3436/12071) Updating files: 29% (3501/12071) Updating files: 30% (3622/12071) Updating files: 30% (3652/12071) Updating files: 31% (3743/12071) Updating files: 31% (3803/12071) Updating files: 31% (3853/12071) Updating files: 32% (3863/12071) Updating files: 32% (3873/12071) Updating files: 32% (3919/12071) Updating files: 33% (3984/12071) Updating files: 34% (4105/12071) Updating files: 34% (4211/12071) Updating files: 35% (4225/12071) Updating files: 36% (4346/12071) error: unable to create file qa/suites/upgrade/telemetry-upgrade/pacific-x/%: File exists 2023-10-30T13:47:37.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.748+0000 7f697c417640 1 Processor -- start 2023-10-30T13:47:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.748+0000 7f697c417640 1 -- start start 2023-10-30T13:47:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.748+0000 7f697c417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f69740c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.748+0000 7f697c417640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f69740c3e40 0x7f6974070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.748+0000 7f697c417640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69740c2310 con 0x7f6974150830 2023-10-30T13:47:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.748+0000 7f697c417640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697406bc10 con 0x7f69740c3e40 2023-10-30T13:47:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.748+0000 7f697c417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6974156710 con 0x7f69740cb7d0 2023-10-30T13:47:37.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f69740c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f697a9b4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f69740c3e40 0x7f6974070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f69740c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:33266/0 (socket says 172.21.15.31:33266) 2023-10-30T13:47:37.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 -- 172.21.15.31:0/781471568 learned_addr learned my addr 172.21.15.31:0/781471568 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:37.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 -- 172.21.15.31:0/781471568 >> v2:172.21.15.33:3300/0 conn(0x7f69740c3e40 msgr2=0x7f6974070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:37.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 --2- 172.21.15.31:0/781471568 >> v2:172.21.15.33:3300/0 conn(0x7f69740c3e40 0x7f6974070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 -- 172.21.15.31:0/781471568 >> v1:172.21.15.33:6789/0 conn(0x7f6974150830 legacy=0x7f697406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 -- 172.21.15.31:0/781471568 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69740c4360 con 0x7f69740cb7d0 2023-10-30T13:47:37.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f697a9b4640 1 --2- 172.21.15.31:0/781471568 >> v2:172.21.15.33:3300/0 conn(0x7f69740c3e40 0x7f6974070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:47:37.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.752+0000 7f69799b2640 1 --2- 172.21.15.31:0/781471568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f69740c3900 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f696400bb10 tx=0x7f6964021400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=822025f790baf110 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:37.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f69791b1640 1 -- 172.21.15.31:0/781471568 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640b1600 con 0x7f69740cb7d0 2023-10-30T13:47:37.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f69791b1640 1 -- 172.21.15.31:0/781471568 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69640ad050 con 0x7f69740cb7d0 2023-10-30T13:47:37.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f69791b1640 1 -- 172.21.15.31:0/781471568 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640b1980 con 0x7f69740cb7d0 2023-10-30T13:47:37.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- 172.21.15.31:0/781471568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 msgr2=0x7f69740c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 --2- 172.21.15.31:0/781471568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f69740c3900 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f696400bb10 tx=0x7f6964021400 comp rx=0 tx=0).stop 2023-10-30T13:47:37.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- 172.21.15.31:0/781471568 shutdown_connections 2023-10-30T13:47:37.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 --2- 172.21.15.31:0/781471568 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f69740c3900 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 --2- 172.21.15.31:0/781471568 >> v2:172.21.15.33:3300/0 conn(0x7f69740c3e40 0x7f6974070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- 172.21.15.31:0/781471568 >> 172.21.15.31:0/781471568 conn(0x7f69740cac70 msgr2=0x7f69740cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:37.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- 172.21.15.31:0/781471568 shutdown_connections 2023-10-30T13:47:37.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- 172.21.15.31:0/781471568 wait complete. 2023-10-30T13:47:37.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 Processor -- start 2023-10-30T13:47:37.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- start start 2023-10-30T13:47:37.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69740c3e40 0x7f6974147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f697414d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6974150830 0x7f697414dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69741eda30 con 0x7f69740c3e40 2023-10-30T13:47:37.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6974147f80 con 0x7f6974150830 2023-10-30T13:47:37.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697c417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697414a6a0 con 0x7f69740cb7d0 2023-10-30T13:47:37.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697a1b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69740c3e40 0x7f6974147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697a1b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69740c3e40 0x7f6974147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43508/0 (socket says 172.21.15.31:43508) 2023-10-30T13:47:37.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697a1b3640 1 -- 172.21.15.31:0/2511036524 learned_addr learned my addr 172.21.15.31:0/2511036524 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:37.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f69799b2640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f697414d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.756+0000 7f697a9b4640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6974150830 0x7f697414dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697a1b3640 1 -- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 msgr2=0x7f697414d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697a1b3640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f697414d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697a1b3640 1 -- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6974150830 msgr2=0x7f697414dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697a1b3640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6974150830 0x7f697414dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697a1b3640 1 -- 172.21.15.31:0/2511036524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c009a70 con 0x7f69740c3e40 2023-10-30T13:47:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697a9b4640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6974150830 0x7f697414dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:37.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697a1b3640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69740c3e40 0x7f6974147a40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f697000c3c0 tx=0x7f697000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:37.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f696affd640 1 -- 172.21.15.31:0/2511036524 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69700052d0 con 0x7f69740c3e40 2023-10-30T13:47:37.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f696affd640 1 -- 172.21.15.31:0/2511036524 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6970005430 con 0x7f69740c3e40 2023-10-30T13:47:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f696affd640 1 -- 172.21.15.31:0/2511036524 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6970003840 con 0x7f69740c3e40 2023-10-30T13:47:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696400b770 con 0x7f69740c3e40 2023-10-30T13:47:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697414ac60 con 0x7f69740c3e40 2023-10-30T13:47:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697404e170 con 0x7f69740c3e40 2023-10-30T13:47:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f696affd640 1 -- 172.21.15.31:0/2511036524 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69700039a0 con 0x7f69740c3e40 2023-10-30T13:47:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f696affd640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69580664a0 0x7f6958068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f696affd640 1 -- 172.21.15.31:0/2511036524 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f6970083660 con 0x7f69740c3e40 2023-10-30T13:47:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.760+0000 7f69799b2640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69580664a0 0x7f6958068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.764+0000 7f69799b2640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69580664a0 0x7f6958068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f69640b3000 tx=0x7f69640ac000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.764+0000 7f696affd640 1 -- 172.21.15.31:0/2511036524 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6970050850 con 0x7f69740c3e40 2023-10-30T13:47:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.952+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6974153fd0 con 0x7f69740c3e40 2023-10-30T13:47:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f696affd640 1 -- 172.21.15.31:0/2511036524 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f6970053e70 con 0x7f69740c3e40 2023-10-30T13:47:37.956 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:37.956 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:30.446421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":42,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:37.956 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 63 2023-10-30T13:47:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69580664a0 msgr2=0x7f6958068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69580664a0 0x7f6958068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f69640b3000 tx=0x7f69640ac000 comp rx=0 tx=0).stop 2023-10-30T13:47:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69740c3e40 msgr2=0x7f6974147a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69740c3e40 0x7f6974147a40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f697000c3c0 tx=0x7f697000c880 comp rx=0 tx=0).stop 2023-10-30T13:47:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 shutdown_connections 2023-10-30T13:47:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69580664a0 0x7f6958068960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6974150830 0x7f697414dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69740cb7d0 0x7f697414d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 --2- 172.21.15.31:0/2511036524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69740c3e40 0x7f6974147a40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 >> 172.21.15.31:0/2511036524 conn(0x7f69740cac70 msgr2=0x7f69740c65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 shutdown_connections 2023-10-30T13:47:37.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:37.956+0000 7f697c417640 1 -- 172.21.15.31:0/2511036524 wait complete. 2023-10-30T13:47:37.979 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5842": { "addr": "172.21.15.33:6836/15857146", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 15857146, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 15857146, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5842, "incarnation": 60, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:47:30.446421+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5842 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:47:39.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:40.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.132+0000 7f84dfc46640 1 Processor -- start 2023-10-30T13:47:40.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.132+0000 7f84dfc46640 1 -- start start 2023-10-30T13:47:40.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dfc46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dfc46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f84d80c4a10 0x7f84d80c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dfc46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d8159f20 con 0x7f84d815a3b0 2023-10-30T13:47:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dfc46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d815dec0 con 0x7f84d80c4a10 2023-10-30T13:47:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dfc46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81599a0 con 0x7f84d8153c80 2023-10-30T13:47:40.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dd1e1640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f84d80c4a10 0x7f84d80c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dd9e2640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84de1e3640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f84d815a3b0 0x7f84d80ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:38072/0 (socket says 172.21.15.31:38072) 2023-10-30T13:47:40.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84de1e3640 1 -- 172.21.15.31:0/3446245177 learned_addr learned my addr 172.21.15.31:0/3446245177 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:40.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dd9e2640 1 -- 172.21.15.31:0/3446245177 >> v2:172.21.15.33:3300/0 conn(0x7f84d80c4a10 msgr2=0x7f84d80c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:40.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dd9e2640 1 --2- 172.21.15.31:0/3446245177 >> v2:172.21.15.33:3300/0 conn(0x7f84d80c4a10 0x7f84d80c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dd9e2640 1 -- 172.21.15.31:0/3446245177 >> v1:172.21.15.33:6789/0 conn(0x7f84d815a3b0 legacy=0x7f84d80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.136+0000 7f84dd9e2640 1 -- 172.21.15.31:0/3446245177 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80c47e0 con 0x7f84d8153c80 2023-10-30T13:47:40.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dd9e2640 1 --2- 172.21.15.31:0/3446245177 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d815cf90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84d40097c0 tx=0x7f84d401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9474df28b826ad6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dc9e0640 1 -- 172.21.15.31:0/3446245177 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40a8970 con 0x7f84d8153c80 2023-10-30T13:47:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dc9e0640 1 -- 172.21.15.31:0/3446245177 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d40a8ad0 con 0x7f84d8153c80 2023-10-30T13:47:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dc9e0640 1 -- 172.21.15.31:0/3446245177 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40a8e60 con 0x7f84d8153c80 2023-10-30T13:47:40.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 -- 172.21.15.31:0/3446245177 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 msgr2=0x7f84d815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 --2- 172.21.15.31:0/3446245177 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d815cf90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84d40097c0 tx=0x7f84d401ec70 comp rx=0 tx=0).stop 2023-10-30T13:47:40.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 -- 172.21.15.31:0/3446245177 shutdown_connections 2023-10-30T13:47:40.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 --2- 172.21.15.31:0/3446245177 >> v2:172.21.15.33:3300/0 conn(0x7f84d80c4a10 0x7f84d80c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 --2- 172.21.15.31:0/3446245177 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d815cf90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 -- 172.21.15.31:0/3446245177 >> 172.21.15.31:0/3446245177 conn(0x7f84d80715c0 msgr2=0x7f84d80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:40.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 -- 172.21.15.31:0/3446245177 shutdown_connections 2023-10-30T13:47:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.140+0000 7f84dfc46640 1 -- 172.21.15.31:0/3446245177 wait complete. 2023-10-30T13:47:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 Processor -- start 2023-10-30T13:47:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 -- start start 2023-10-30T13:47:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84d80c4a10 0x7f84d814f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d80c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84d815a3b0 0x7f84d80c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81f2260 con 0x7f84d80c4a10 2023-10-30T13:47:40.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81f2540 con 0x7f84d815a3b0 2023-10-30T13:47:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81f2820 con 0x7f84d8153c80 2023-10-30T13:47:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84d80c4a10 0x7f84d814f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84d80c4a10 0x7f84d814f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43524/0 (socket says 172.21.15.31:43524) 2023-10-30T13:47:40.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 -- 172.21.15.31:0/2235373843 learned_addr learned my addr 172.21.15.31:0/2235373843 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:40.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd1e1640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d80c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84de1e3640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84d815a3b0 0x7f84d80c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 -- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 msgr2=0x7f84d80c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d80c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 -- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84d815a3b0 msgr2=0x7f84d80c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84d815a3b0 0x7f84d80c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 -- 172.21.15.31:0/2235373843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d4009420 con 0x7f84d80c4a10 2023-10-30T13:47:40.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd1e1640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d80c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:40.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dd9e2640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84d80c4a10 0x7f84d814f800 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84d40b3040 tx=0x7f84d401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:40.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84ce7fc640 1 -- 172.21.15.31:0/2235373843 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40b9630 con 0x7f84d80c4a10 2023-10-30T13:47:40.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84ce7fc640 1 -- 172.21.15.31:0/2235373843 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d40b97b0 con 0x7f84d80c4a10 2023-10-30T13:47:40.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84ce7fc640 1 -- 172.21.15.31:0/2235373843 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40b9b60 con 0x7f84d80c4a10 2023-10-30T13:47:40.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d81f2b00 con 0x7f84d80c4a10 2023-10-30T13:47:40.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d815d150 con 0x7f84d80c4a10 2023-10-30T13:47:40.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.144+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d81f2fb0 con 0x7f84d80c4a10 2023-10-30T13:47:40.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.148+0000 7f84ce7fc640 1 -- 172.21.15.31:0/2235373843 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84d40b7040 con 0x7f84d80c4a10 2023-10-30T13:47:40.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.148+0000 7f84ce7fc640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84bc0664a0 0x7f84bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.148+0000 7f84ce7fc640 1 -- 172.21.15.31:0/2235373843 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f84d4122020 con 0x7f84d80c4a10 2023-10-30T13:47:40.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.148+0000 7f84dd1e1640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84bc0664a0 0x7f84bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.148+0000 7f84dd1e1640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84bc0664a0 0x7f84bc068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f84d8148db0 tx=0x7f84c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:40.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.152+0000 7f84ce7fc640 1 -- 172.21.15.31:0/2235373843 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84d81f2fb0 con 0x7f84d80c4a10 2023-10-30T13:47:40.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.344+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84d81f2fb0 con 0x7f84d80c4a10 2023-10-30T13:47:40.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.344+0000 7f84ce7fc640 1 -- 172.21.15.31:0/2235373843 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5696 (secure 0 0 0) 0x7f84d81f2fb0 con 0x7f84d80c4a10 2023-10-30T13:47:40.347 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:40.347 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 64 2023-10-30T13:47:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84bc0664a0 msgr2=0x7f84bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84bc0664a0 0x7f84bc068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f84d8148db0 tx=0x7f84c8008040 comp rx=0 tx=0).stop 2023-10-30T13:47:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84d80c4a10 msgr2=0x7f84d814f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84d80c4a10 0x7f84d814f800 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84d40b3040 tx=0x7f84d401ba10 comp rx=0 tx=0).stop 2023-10-30T13:47:40.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 shutdown_connections 2023-10-30T13:47:40.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84bc0664a0 0x7f84bc068960 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84d815a3b0 0x7f84d80c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f84d8153c80 0x7f84d80c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 --2- 172.21.15.31:0/2235373843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84d80c4a10 0x7f84d814f800 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 >> 172.21.15.31:0/2235373843 conn(0x7f84d80715c0 msgr2=0x7f84d8157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:40.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 shutdown_connections 2023-10-30T13:47:40.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:40.348+0000 7f84dfc46640 1 -- 172.21.15.31:0/2235373843 wait complete. 2023-10-30T13:47:40.368 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5448 2023-10-30T13:47:40.368 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2023-10-30T13:47:49.369 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:47:49.369 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:47:49.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:47:49.374 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:47:49.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:49.472 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2023-10-30T13:47:49.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.528+0000 7fb751446640 1 Processor -- start 2023-10-30T13:47:49.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.528+0000 7fb751446640 1 -- start start 2023-10-30T13:47:49.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb751446640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb74c070520 0x7fb74c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb751446640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c0c2310 con 0x7fb74c0cb7d0 2023-10-30T13:47:49.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb751446640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c156710 con 0x7fb74c150830 2023-10-30T13:47:49.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb751446640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c06bc10 con 0x7fb74c070520 2023-10-30T13:47:49.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb74b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb74c070520 0x7fb74c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb74b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb74c070520 0x7fb74c070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43974/0 (socket says 172.21.15.31:43974) 2023-10-30T13:47:49.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb74b7fe640 1 -- 172.21.15.31:0/2877865102 learned_addr learned my addr 172.21.15.31:0/2877865102 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:49.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb74b7fe640 1 -- 172.21.15.31:0/2877865102 >> v1:172.21.15.33:6789/0 conn(0x7fb74c150830 legacy=0x7fb74c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb74b7fe640 1 -- 172.21.15.31:0/2877865102 >> v1:172.21.15.31:6789/0 conn(0x7fb74c0cb7d0 legacy=0x7fb74c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.532+0000 7fb74b7fe640 1 -- 172.21.15.31:0/2877865102 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0c4360 con 0x7fb74c070520 2023-10-30T13:47:49.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb74b7fe640 1 --2- 172.21.15.31:0/2877865102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c070520 0x7fb74c070990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb744009e50 tx=0x7fb74401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2222540deeefdab6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:49.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb74a7fc640 1 -- 172.21.15.31:0/2877865102 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440af400 con 0x7fb74c070520 2023-10-30T13:47:49.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb74a7fc640 1 -- 172.21.15.31:0/2877865102 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7440ab050 con 0x7fb74c070520 2023-10-30T13:47:49.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb74a7fc640 1 -- 172.21.15.31:0/2877865102 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440af780 con 0x7fb74c070520 2023-10-30T13:47:49.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- 172.21.15.31:0/2877865102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c070520 msgr2=0x7fb74c070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 --2- 172.21.15.31:0/2877865102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c070520 0x7fb74c070990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb744009e50 tx=0x7fb74401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- 172.21.15.31:0/2877865102 shutdown_connections 2023-10-30T13:47:49.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 --2- 172.21.15.31:0/2877865102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c070520 0x7fb74c070990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- 172.21.15.31:0/2877865102 >> 172.21.15.31:0/2877865102 conn(0x7fb74c0cac70 msgr2=0x7fb74c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:49.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- 172.21.15.31:0/2877865102 shutdown_connections 2023-10-30T13:47:49.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- 172.21.15.31:0/2877865102 wait complete. 2023-10-30T13:47:49.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 Processor -- start 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- start start 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb74c070520 0x7fb74c15a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb74c0cb7d0 0x7fb74c1f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c150830 0x7fb74c1f1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c153990 con 0x7fb74c150830 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1f0950 con 0x7fb74c0cb7d0 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.536+0000 7fb751446640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1f0320 con 0x7fb74c070520 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c150830 0x7fb74c1f1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c150830 0x7fb74c1f1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43980/0 (socket says 172.21.15.31:43980) 2023-10-30T13:47:49.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 -- 172.21.15.31:0/3700462636 learned_addr learned my addr 172.21.15.31:0/3700462636 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:49.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74affd640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb74c070520 0x7fb74c15a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb743fff640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb74c0cb7d0 0x7fb74c1f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 -- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb74c070520 msgr2=0x7fb74c15a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb74c070520 0x7fb74c15a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 -- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb74c0cb7d0 msgr2=0x7fb74c1f0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb74c0cb7d0 0x7fb74c1f0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 -- 172.21.15.31:0/3700462636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744009b30 con 0x7fb74c150830 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74affd640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb74c070520 0x7fb74c15a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb743fff640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb74c0cb7d0 0x7fb74c1f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb74b7fe640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c150830 0x7fb74c1f1320 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb74401be00 tx=0x7fb74401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb7437fe640 1 -- 172.21.15.31:0/3700462636 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440afde0 con 0x7fb74c150830 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb7437fe640 1 -- 172.21.15.31:0/3700462636 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb744009510 con 0x7fb74c150830 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c153be0 con 0x7fb74c150830 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb7437fe640 1 -- 172.21.15.31:0/3700462636 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440ba660 con 0x7fb74c150830 2023-10-30T13:47:49.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c0c4640 con 0x7fb74c150830 2023-10-30T13:47:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c1f0ac0 con 0x7fb74c150830 2023-10-30T13:47:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb7437fe640 1 -- 172.21.15.31:0/3700462636 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7440b6040 con 0x7fb74c150830 2023-10-30T13:47:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb7437fe640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb72c0664a0 0x7fb72c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.540+0000 7fb7437fe640 1 -- 172.21.15.31:0/3700462636 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fb744122530 con 0x7fb74c150830 2023-10-30T13:47:49.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.544+0000 7fb74affd640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb72c0664a0 0x7fb72c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.544+0000 7fb74affd640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb72c0664a0 0x7fb72c068960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000f70 tx=0x7fb73c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:49.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.544+0000 7fb7437fe640 1 -- 172.21.15.31:0/3700462636 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7440ef6a0 con 0x7fb74c150830 2023-10-30T13:47:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb74c066510 con 0x7fb74c150830 2023-10-30T13:47:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb7437fe640 1 -- 172.21.15.31:0/3700462636 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5696 (secure 0 0 0) 0x7fb7440f2cc0 con 0x7fb74c150830 2023-10-30T13:47:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:49.744 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 64 2023-10-30T13:47:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb72c0664a0 msgr2=0x7fb72c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb72c0664a0 0x7fb72c068960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000f70 tx=0x7fb73c009040 comp rx=0 tx=0).stop 2023-10-30T13:47:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c150830 msgr2=0x7fb74c1f1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c150830 0x7fb74c1f1320 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb74401be00 tx=0x7fb74401bf90 comp rx=0 tx=0).stop 2023-10-30T13:47:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 shutdown_connections 2023-10-30T13:47:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb72c0664a0 0x7fb72c068960 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb74c150830 0x7fb74c1f1320 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb74c0cb7d0 0x7fb74c1f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 --2- 172.21.15.31:0/3700462636 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb74c070520 0x7fb74c15a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 >> 172.21.15.31:0/3700462636 conn(0x7fb74c0cac70 msgr2=0x7fb74c06bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 shutdown_connections 2023-10-30T13:47:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:49.744+0000 7fb751446640 1 -- 172.21.15.31:0/3700462636 wait complete. 2023-10-30T13:47:49.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:47:51.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.924+0000 7f8376f56640 1 Processor -- start 2023-10-30T13:47:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.924+0000 7f8376f56640 1 -- start start 2023-10-30T13:47:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376f56640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f837015cd70 0x7f8370069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376f56640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837015dcc0 con 0x7f83700c8eb0 2023-10-30T13:47:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376f56640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8370159d80 con 0x7f837015ac70 2023-10-30T13:47:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376f56640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8370159800 con 0x7f837015cd70 2023-10-30T13:47:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376755640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f837015cd70 0x7f8370069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8375f54640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f83700c8eb0 0x7f837015a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46154/0 (socket says 172.21.15.31:46154) 2023-10-30T13:47:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376755640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f837015cd70 0x7f8370069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43994/0 (socket says 172.21.15.31:43994) 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8375f54640 1 -- 172.21.15.31:0/4173312045 learned_addr learned my addr 172.21.15.31:0/4173312045 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376755640 1 -- 172.21.15.31:0/4173312045 >> v1:172.21.15.33:6789/0 conn(0x7f837015ac70 legacy=0x7f837006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376755640 1 -- 172.21.15.31:0/4173312045 >> v1:172.21.15.31:6789/0 conn(0x7f83700c8eb0 legacy=0x7f837015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8376755640 1 -- 172.21.15.31:0/4173312045 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837015d1c0 con 0x7f837015cd70 2023-10-30T13:47:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.928+0000 7f8374f52640 1 -- 172.21.15.31:0/4173312045 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482380222 0 0) 0x7f836c001120 con 0x7f83700c8eb0 2023-10-30T13:47:51.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376755640 1 --2- 172.21.15.31:0/4173312045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f8370069e90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8368009d90 tx=0x7f836801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f6e06c5635c7be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:51.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8374f52640 1 -- 172.21.15.31:0/4173312045 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83680af600 con 0x7f837015cd70 2023-10-30T13:47:51.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8374f52640 1 -- 172.21.15.31:0/4173312045 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83680ab050 con 0x7f837015cd70 2023-10-30T13:47:51.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8374f52640 1 -- 172.21.15.31:0/4173312045 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83680af980 con 0x7f837015cd70 2023-10-30T13:47:51.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 -- 172.21.15.31:0/4173312045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 msgr2=0x7f8370069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 --2- 172.21.15.31:0/4173312045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f8370069e90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8368009d90 tx=0x7f836801f130 comp rx=0 tx=0).stop 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 -- 172.21.15.31:0/4173312045 shutdown_connections 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 --2- 172.21.15.31:0/4173312045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f8370069e90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 -- 172.21.15.31:0/4173312045 >> 172.21.15.31:0/4173312045 conn(0x7f83700cef00 msgr2=0x7f83700cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 -- 172.21.15.31:0/4173312045 shutdown_connections 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 -- 172.21.15.31:0/4173312045 wait complete. 2023-10-30T13:47:51.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 Processor -- start 2023-10-30T13:47:51.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.932+0000 7f8376f56640 1 -- start start 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83700c8eb0 0x7f837014e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f837015ac70 0x7f837014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f837015b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83700cdd30 con 0x7f837015cd70 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83700ce8b0 con 0x7f837015ac70 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837014e840 con 0x7f83700c8eb0 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f837015b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f837015b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44002/0 (socket says 172.21.15.31:44002) 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 -- 172.21.15.31:0/3890043240 learned_addr learned my addr 172.21.15.31:0/3890043240 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8375753640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f837015ac70 0x7f837014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8375f54640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83700c8eb0 0x7f837014e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 -- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83700c8eb0 msgr2=0x7f837014e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83700c8eb0 0x7f837014e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 -- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f837015ac70 msgr2=0x7f837014f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f837015ac70 0x7f837014f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 -- 172.21.15.31:0/3890043240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8368009a70 con 0x7f837015cd70 2023-10-30T13:47:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376755640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f837015b9d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83680b2040 tx=0x7f836801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f83667fc640 1 -- 172.21.15.31:0/3890043240 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83680b9610 con 0x7f837015cd70 2023-10-30T13:47:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f83667fc640 1 -- 172.21.15.31:0/3890043240 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83680b9770 con 0x7f837015cd70 2023-10-30T13:47:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83700cdf80 con 0x7f837015cd70 2023-10-30T13:47:51.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f83667fc640 1 -- 172.21.15.31:0/3890043240 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83680b9b20 con 0x7f837015cd70 2023-10-30T13:47:51.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83700ce540 con 0x7f837015cd70 2023-10-30T13:47:51.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83700cea20 con 0x7f837015cd70 2023-10-30T13:47:51.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.936+0000 7f83667fc640 1 -- 172.21.15.31:0/3890043240 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83680b6050 con 0x7f837015cd70 2023-10-30T13:47:51.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.940+0000 7f83667fc640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83540664a0 0x7f8354068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.940+0000 7f83667fc640 1 -- 172.21.15.31:0/3890043240 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f83681227a0 con 0x7f837015cd70 2023-10-30T13:47:51.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.940+0000 7f8375f54640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83540664a0 0x7f8354068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.940+0000 7f8375f54640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83540664a0 0x7f8354068960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f836c007fd0 tx=0x7f836c007a20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:51.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:51.944+0000 7f83667fc640 1 -- 172.21.15.31:0/3890043240 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83680efac0 con 0x7f837015cd70 2023-10-30T13:47:52.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.148+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f837004e170 con 0x7f837015cd70 2023-10-30T13:47:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.148+0000 7f83667fc640 1 -- 172.21.15.31:0/3890043240 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f83680f30e0 con 0x7f837015cd70 2023-10-30T13:47:52.149 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:47:52.150 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:47:52.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.148+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83540664a0 msgr2=0x7f8354068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.148+0000 7f8376f56640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83540664a0 0x7f8354068960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f836c007fd0 tx=0x7f836c007a20 comp rx=0 tx=0).stop 2023-10-30T13:47:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.148+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 msgr2=0x7f837015b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.148+0000 7f8376f56640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f837015b9d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83680b2040 tx=0x7f836801bed0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 shutdown_connections 2023-10-30T13:47:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83540664a0 0x7f8354068960 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f837015cd70 0x7f837015b9d0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f837015ac70 0x7f837014f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 --2- 172.21.15.31:0/3890043240 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83700c8eb0 0x7f837014e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 >> 172.21.15.31:0/3890043240 conn(0x7f83700cef00 msgr2=0x7f83700733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 shutdown_connections 2023-10-30T13:47:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:47:52.152+0000 7f8376f56640 1 -- 172.21.15.31:0/3890043240 wait complete. 2023-10-30T13:47:52.171 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:47:52.171 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-10-30T13:47:56.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:56.833 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:27.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:27.029 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:57.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:57.078 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:59.786 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 36% (4400/12071) Updating files: 36% (4401/12071) Updating files: 36% (4441/12071) Updating files: 37% (4467/12071) Updating files: 38% (4587/12071) Updating files: 38% (4617/12071) Updating files: 39% (4708/12071) Updating files: 39% (4761/12071) Updating files: 40% (4829/12071) Updating files: 41% (4950/12071) Updating files: 41% (5007/12071) Updating files: 42% (5070/12071) Updating files: 43% (5191/12071) Updating files: 43% (5208/12071) Updating files: 43% (5217/12071) Updating files: 44% (5312/12071) Updating files: 44% (5382/12071) Updating files: 44% (5395/12071) Updating files: 45% (5432/12071) Updating files: 46% (5553/12071) Updating files: 46% (5594/12071) Updating files: 47% (5674/12071) Updating files: 47% (5743/12071) Updating files: 48% (5795/12071) Updating files: 48% (5832/12071) Updating files: 48% (5866/12071) Updating files: 49% (5915/12071) Updating files: 49% (5956/12071) Updating files: 50% (6036/12071) Updating files: 51% (6157/12071) Updating files: 51% (6189/12071) Updating files: 51% (6234/12071) Updating files: 52% (6277/12071) Updating files: 53% (6398/12071) Updating files: 53% (6454/12071) Updating files: 54% (6519/12071) Updating files: 54% (6558/12071) Updating files: 55% (6640/12071) Updating files: 55% (6732/12071) Updating files: 56% (6760/12071) Updating files: 56% (6805/12071) Updating files: 56% (6845/12071) Updating files: 57% (6881/12071) Updating files: 57% (6894/12071) Updating files: 57% (6933/12071) Updating files: 58% (7002/12071) Updating files: 59% (7122/12071) Updating files: 59% (7198/12071) Updating files: 60% (7243/12071) Updating files: 61% (7364/12071) Updating files: 61% (7403/12071) Updating files: 61% (7470/12071) Updating files: 62% (7485/12071) Updating files: 62% (7532/12071) Updating files: 63% (7605/12071) Updating files: 63% (7701/12071) Updating files: 64% (7726/12071) Updating files: 64% (7829/12071) Updating files: 65% (7847/12071) Updating files: 65% (7891/12071) Updating files: 65% (7930/12071) Updating files: 66% (7967/12071) Updating files: 66% (8071/12071) Updating files: 67% (8088/12071) Updating files: 68% (8209/12071) Updating files: 69% (8329/12071) Updating files: 69% (8389/12071) Updating files: 70% (8450/12071) Updating files: 70% (8521/12071) Updating files: 71% (8571/12071) Updating files: 71% (8589/12071) Updating files: 71% (8647/12071) Updating files: 72% (8692/12071) Updating files: 72% (8748/12071) Updating files: 72% (8800/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 74% (9031/12071) Updating files: 74% (9043/12071) Updating files: 75% (9054/12071) Updating files: 75% (9136/12071) Updating files: 75% (9146/12071) Updating files: 75% (9154/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 77% (9406/12071) Updating files: 78% (9416/12071) Updating files: 78% (9422/12071) Updating files: 79% (9537/12071) Updating files: 79% (9623/12071) Updating files: 79% (9625/12071) Updating files: 80% (9657/12071) Updating files: 80% (9660/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 82% (9900/12071) Updating files: 83% (10019/12071) Updating files: 83% (10047/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 85% (10347/12071) Updating files: 86% (10382/12071) Updating files: 86% (10485/12071) Updating files: 87% (10502/12071) Updating files: 87% (10535/12071) Updating files: 87% (10620/12071) Updating files: 88% (10623/12071) Updating files: 88% (10662/12071) Updating files: 88% (10731/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 91% (10985/12071) Updating files: 91% (11008/12071) Updating files: 91% (11066/12071) Updating files: 92% (11106/12071) Updating files: 92% (11151/12071) Updating files: 92% (11195/12071) Updating files: 92% (11225/12071) Updating files: 93% (11227/12071) Updating files: 93% (11286/12071) Updating files: 94% (11347/12071) Updating files: 94% (11377/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 96% (11603/12071) Updating files: 96% (11659/12071) Updating files: 97% (11709/12071) Updating files: 97% (11766/12071) Updating files: 98% (11830/12071) Updating files: 98% (11845/12071) Updating files: 99% (11951/12071) Updating files: 99% (12005/12071) Updating files: 99% (12039/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T13:48:59.787 INFO:tasks.workunit.client.0.smithi031.stderr:fatal: unable to checkout working tree 2023-10-30T13:48:59.791 INFO:tasks.workunit.client.0.smithi031.stderr:warning: Clone succeeded, but checkout failed. 2023-10-30T13:48:59.791 INFO:tasks.workunit.client.0.smithi031.stderr:You can inspect what was checked out with 'git status' 2023-10-30T13:48:59.791 INFO:tasks.workunit.client.0.smithi031.stderr:and retry with 'git restore --source=HEAD :/' 2023-10-30T13:48:59.791 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-30T13:48:59.811 INFO:tasks.workunit.client.0.smithi031.stderr:+ retry 2023-10-30T13:48:59.811 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm -rf ceph 2023-10-30T13:49:21.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:21.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.314+0000 7f0438609640 1 Processor -- start 2023-10-30T13:49:21.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.314+0000 7f0438609640 1 -- start start 2023-10-30T13:49:21.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.314+0000 7f0438609640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0430150830 0x7f043006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.314+0000 7f0438609640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f0438609640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04300c2310 con 0x7f04300c3e40 2023-10-30T13:49:21.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f0438609640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0430156710 con 0x7f0430150830 2023-10-30T13:49:21.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f0438609640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04300c4360 con 0x7f04300cb7d0 2023-10-30T13:49:21.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0430150830 0x7f043006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0430150830 0x7f043006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43518/0 (socket says 172.21.15.31:43518) 2023-10-30T13:49:21.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 -- 172.21.15.31:0/1282803787 learned_addr learned my addr 172.21.15.31:0/1282803787 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:21.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f0435ba4640 1 --2- 172.21.15.31:0/1282803787 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 -- 172.21.15.31:0/1282803787 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 msgr2=0x7f04300c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:21.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 --2- 172.21.15.31:0/1282803787 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 -- 172.21.15.31:0/1282803787 >> v1:172.21.15.33:6789/0 conn(0x7f04300c3e40 legacy=0x7f0430070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 -- 172.21.15.31:0/1282803787 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04301569b0 con 0x7f0430150830 2023-10-30T13:49:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f0435ba4640 1 --2- 172.21.15.31:0/1282803787 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.318+0000 7f04363a5640 1 --2- 172.21.15.31:0/1282803787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0430150830 0x7f043006b500 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f042c009870 tx=0x7f042c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cc148fa5840b13 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f04353a3640 1 -- 172.21.15.31:0/1282803787 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0a8a30 con 0x7f0430150830 2023-10-30T13:49:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f04353a3640 1 -- 172.21.15.31:0/1282803787 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c0a8b90 con 0x7f0430150830 2023-10-30T13:49:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f04353a3640 1 -- 172.21.15.31:0/1282803787 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0b0440 con 0x7f0430150830 2023-10-30T13:49:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 -- 172.21.15.31:0/1282803787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0430150830 msgr2=0x7f043006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 --2- 172.21.15.31:0/1282803787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0430150830 0x7f043006b500 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f042c009870 tx=0x7f042c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 -- 172.21.15.31:0/1282803787 shutdown_connections 2023-10-30T13:49:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 --2- 172.21.15.31:0/1282803787 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 --2- 172.21.15.31:0/1282803787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0430150830 0x7f043006b500 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 -- 172.21.15.31:0/1282803787 >> 172.21.15.31:0/1282803787 conn(0x7f04300cac70 msgr2=0x7f04300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 -- 172.21.15.31:0/1282803787 shutdown_connections 2023-10-30T13:49:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.322+0000 7f0438609640 1 -- 172.21.15.31:0/1282803787 wait complete. 2023-10-30T13:49:21.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 Processor -- start 2023-10-30T13:49:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 -- start start 2023-10-30T13:49:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04300c3e40 0x7f043015a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0430150830 0x7f04301efff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043015a3b0 con 0x7f04300c3e40 2023-10-30T13:49:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0430158030 con 0x7f0430150830 2023-10-30T13:49:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0438609640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04301582f0 con 0x7f04300cb7d0 2023-10-30T13:49:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f04363a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04300c3e40 0x7f043015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f04363a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04300c3e40 0x7f043015a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43522/0 (socket says 172.21.15.31:43522) 2023-10-30T13:49:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f04363a5640 1 -- 172.21.15.31:0/2648041816 learned_addr learned my addr 172.21.15.31:0/2648041816 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0435ba4640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.326+0000 7f0436ba6640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0430150830 0x7f04301efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f04363a5640 1 -- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 msgr2=0x7f04300ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f04363a5640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f04363a5640 1 -- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0430150830 msgr2=0x7f04301efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f04363a5640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0430150830 0x7f04301efff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f04363a5640 1 -- 172.21.15.31:0/2648041816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c0094d0 con 0x7f04300c3e40 2023-10-30T13:49:21.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f04363a5640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04300c3e40 0x7f043015a670 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f042c0b3040 tx=0x7f042c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f0426ffd640 1 -- 172.21.15.31:0/2648041816 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0be070 con 0x7f04300c3e40 2023-10-30T13:49:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f0426ffd640 1 -- 172.21.15.31:0/2648041816 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c0ba400 con 0x7f04300c3e40 2023-10-30T13:49:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f0426ffd640 1 -- 172.21.15.31:0/2648041816 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0ba7d0 con 0x7f04300c3e40 2023-10-30T13:49:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04301585d0 con 0x7f04300c3e40 2023-10-30T13:49:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0430158b90 con 0x7f04300c3e40 2023-10-30T13:49:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0430156c70 con 0x7f04300c3e40 2023-10-30T13:49:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.330+0000 7f0426ffd640 1 -- 172.21.15.31:0/2648041816 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f042c0c3410 con 0x7f04300c3e40 2023-10-30T13:49:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.334+0000 7f0426ffd640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f04140664a0 0x7f0414068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.334+0000 7f0426ffd640 1 -- 172.21.15.31:0/2648041816 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f042c121c30 con 0x7f04300c3e40 2023-10-30T13:49:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.334+0000 7f0435ba4640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f04140664a0 0x7f0414068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.334+0000 7f0435ba4640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f04140664a0 0x7f0414068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f043015abb0 tx=0x7f0420008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.338+0000 7f0426ffd640 1 -- 172.21.15.31:0/2648041816 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0430156c70 con 0x7f04300c3e40 2023-10-30T13:49:21.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.530+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0430156c70 con 0x7f04300c3e40 2023-10-30T13:49:21.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.530+0000 7f0426ffd640 1 -- 172.21.15.31:0/2648041816 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f0430156c70 con 0x7f04300c3e40 2023-10-30T13:49:21.535 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:21.535 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:21.536 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:21.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f04140664a0 msgr2=0x7f0414068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f04140664a0 0x7f0414068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f043015abb0 tx=0x7f0420008040 comp rx=0 tx=0).stop 2023-10-30T13:49:21.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04300c3e40 msgr2=0x7f043015a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04300c3e40 0x7f043015a670 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f042c0b3040 tx=0x7f042c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 shutdown_connections 2023-10-30T13:49:21.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f04140664a0 0x7f0414068960 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0430150830 0x7f04301efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f04300cb7d0 0x7f04300ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 --2- 172.21.15.31:0/2648041816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04300c3e40 0x7f043015a670 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 >> 172.21.15.31:0/2648041816 conn(0x7f04300cac70 msgr2=0x7f04300733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:21.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 shutdown_connections 2023-10-30T13:49:21.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.534+0000 7f0438609640 1 -- 172.21.15.31:0/2648041816 wait complete. 2023-10-30T13:49:21.556 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:49:21.556 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:49:21.557 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:49:21.579 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:49:21.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:21.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.718+0000 7fb8cc71d640 1 Processor -- start 2023-10-30T13:49:21.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.718+0000 7fb8cc71d640 1 -- start start 2023-10-30T13:49:21.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8cc71d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb8c4151840 0x7fb8c40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8cc71d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c40cb8b0 0x7fb8c406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8cc71d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c40c5030 con 0x7fb8c4151c20 2023-10-30T13:49:21.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8cc71d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c4157830 con 0x7fb8c4151840 2023-10-30T13:49:21.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8cc71d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4157b10 con 0x7fb8c40cb8b0 2023-10-30T13:49:21.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8ca4b9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb8c4151840 0x7fb8c40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8cacba640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c40cb8b0 0x7fb8c406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8ca4b9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb8c4151840 0x7fb8c40c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:49998/0 (socket says 172.21.15.31:49998) 2023-10-30T13:49:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8ca4b9640 1 -- 172.21.15.31:0/3873101710 learned_addr learned my addr 172.21.15.31:0/3873101710 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8ca4b9640 1 -- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c40cb8b0 msgr2=0x7fb8c406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:21.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8ca4b9640 1 --2- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c40cb8b0 0x7fb8c406b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8ca4b9640 1 -- 172.21.15.31:0/3873101710 >> v1:172.21.15.33:6789/0 conn(0x7fb8c4151c20 legacy=0x7fb8c4072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8ca4b9640 1 -- 172.21.15.31:0/3873101710 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4157df0 con 0x7fb8c4151840 2023-10-30T13:49:21.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.722+0000 7fb8cacba640 1 --2- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c40cb8b0 0x7fb8c406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:21.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8ca4b9640 1 --2- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c4151840 0x7fb8c40c49a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009870 tx=0x7fb8c001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5a0251f734ca119 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8c94b7640 1 -- 172.21.15.31:0/3873101710 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00a8a30 con 0x7fb8c4151840 2023-10-30T13:49:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8c94b7640 1 -- 172.21.15.31:0/3873101710 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c00a8b90 con 0x7fb8c4151840 2023-10-30T13:49:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8c94b7640 1 -- 172.21.15.31:0/3873101710 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00b0440 con 0x7fb8c4151840 2023-10-30T13:49:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 -- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c4151840 msgr2=0x7fb8c40c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c4151840 0x7fb8c40c49a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009870 tx=0x7fb8c001ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:21.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 -- 172.21.15.31:0/3873101710 shutdown_connections 2023-10-30T13:49:21.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c40cb8b0 0x7fb8c406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/3873101710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c4151840 0x7fb8c40c49a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 -- 172.21.15.31:0/3873101710 >> 172.21.15.31:0/3873101710 conn(0x7fb8c40cac70 msgr2=0x7fb8c40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 -- 172.21.15.31:0/3873101710 shutdown_connections 2023-10-30T13:49:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.726+0000 7fb8cc71d640 1 -- 172.21.15.31:0/3873101710 wait complete. 2023-10-30T13:49:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 Processor -- start 2023-10-30T13:49:21.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 -- start start 2023-10-30T13:49:21.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c40cb8b0 0x7fb8c414cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8c4151840 0x7fb8c414d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c4151c20 0x7fb8c4153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4149150 con 0x7fb8c4151840 2023-10-30T13:49:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4152e90 con 0x7fb8c40cb8b0 2023-10-30T13:49:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4154d10 con 0x7fb8c4151c20 2023-10-30T13:49:21.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8c4151840 0x7fb8c414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8c4151840 0x7fb8c414d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43536/0 (socket says 172.21.15.31:43536) 2023-10-30T13:49:21.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 -- 172.21.15.31:0/2602467104 learned_addr learned my addr 172.21.15.31:0/2602467104 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:21.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cacba640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c4151c20 0x7fb8c4153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8ca4b9640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c40cb8b0 0x7fb8c414cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 -- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c4151c20 msgr2=0x7fb8c4153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c4151c20 0x7fb8c4153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 -- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c40cb8b0 msgr2=0x7fb8c414cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c40cb8b0 0x7fb8c414cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 -- 172.21.15.31:0/2602467104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc009a70 con 0x7fb8c4151840 2023-10-30T13:49:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8c9cb8640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8c4151840 0x7fb8c414d450 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400c370 tx=0x7fb8b400c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8baffd640 1 -- 172.21.15.31:0/2602467104 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40052d0 con 0x7fb8c4151840 2023-10-30T13:49:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8baffd640 1 -- 172.21.15.31:0/2602467104 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b4005430 con 0x7fb8c4151840 2023-10-30T13:49:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8baffd640 1 -- 172.21.15.31:0/2602467104 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4007260 con 0x7fb8c4151840 2023-10-30T13:49:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c00094d0 con 0x7fb8c4151840 2023-10-30T13:49:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c4153130 con 0x7fb8c4151840 2023-10-30T13:49:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.730+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c404e170 con 0x7fb8c4151840 2023-10-30T13:49:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.734+0000 7fb8baffd640 1 -- 172.21.15.31:0/2602467104 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8b40073c0 con 0x7fb8c4151840 2023-10-30T13:49:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.734+0000 7fb8baffd640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8a80664a0 0x7fb8a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.734+0000 7fb8baffd640 1 -- 172.21.15.31:0/2602467104 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fb8b40833f0 con 0x7fb8c4151840 2023-10-30T13:49:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.734+0000 7fb8ca4b9640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8a80664a0 0x7fb8a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.734+0000 7fb8ca4b9640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8a80664a0 0x7fb8a8068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00b2000 tx=0x7fb8c00aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.738+0000 7fb8baffd640 1 -- 172.21.15.31:0/2602467104 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8b40505e0 con 0x7fb8c4151840 2023-10-30T13:49:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.926+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8c4146ed0 con 0x7fb8c4151840 2023-10-30T13:49:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8baffd640 1 -- 172.21.15.31:0/2602467104 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7fb8b4053c00 con 0x7fb8c4151840 2023-10-30T13:49:21.932 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:21.932 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8a80664a0 msgr2=0x7fb8a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8a80664a0 0x7fb8a8068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00b2000 tx=0x7fb8c00aa070 comp rx=0 tx=0).stop 2023-10-30T13:49:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8c4151840 msgr2=0x7fb8c414d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8c4151840 0x7fb8c414d450 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400c370 tx=0x7fb8b400c770 comp rx=0 tx=0).stop 2023-10-30T13:49:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 shutdown_connections 2023-10-30T13:49:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb8a80664a0 0x7fb8a8068960 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb8c4151c20 0x7fb8c4153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8c4151840 0x7fb8c414d450 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 --2- 172.21.15.31:0/2602467104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8c40cb8b0 0x7fb8c414cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 >> 172.21.15.31:0/2602467104 conn(0x7fb8c40cac70 msgr2=0x7fb8c4072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 shutdown_connections 2023-10-30T13:49:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:21.930+0000 7fb8cc71d640 1 -- 172.21.15.31:0/2602467104 wait complete. 2023-10-30T13:49:21.954 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:49:23.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:24.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.106+0000 7ff2e099d640 1 Processor -- start 2023-10-30T13:49:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.106+0000 7ff2e099d640 1 -- start start 2023-10-30T13:49:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2e099d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc06fdf0 0x7ff2dc072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2e099d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc06b500 con 0x7ff2dc14e030 2023-10-30T13:49:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2e099d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc14e410 con 0x7ff2dc06b990 2023-10-30T13:49:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2e099d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc1582e0 con 0x7ff2dc06fdf0 2023-10-30T13:49:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff2dc14e030 0x7ff2dc0707e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52866/0 (socket says 172.21.15.31:52866) 2023-10-30T13:49:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 -- 172.21.15.31:0/2597314607 learned_addr learned my addr 172.21.15.31:0/2597314607 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d3fff640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc06fdf0 0x7ff2dc072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434403904 0 0) 0x7ff2dc06b500 con 0x7ff2dc14e030 2023-10-30T13:49:24.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c8003530 con 0x7ff2dc14e030 2023-10-30T13:49:24.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283737015 0 0) 0x7ff2c8003530 con 0x7ff2dc14e030 2023-10-30T13:49:24.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc06fdf0 msgr2=0x7ff2dc072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:24.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc06fdf0 0x7ff2dc072890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 >> v1:172.21.15.33:6789/0 conn(0x7ff2dc06b990 legacy=0x7ff2dc06be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc0c5820 con 0x7ff2dc14e030 2023-10-30T13:49:24.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7ff2cc001680 con 0x7ff2dc14e030 2023-10-30T13:49:24.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 >> v1:172.21.15.31:6789/0 conn(0x7ff2dc14e030 legacy=0x7ff2dc0707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2c80039b0 0x7ff2c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2c8004360 0x7ff2c8004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c8004fe0 0x7ff2c800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc1582e0 con 0x7ff2c80039b0 2023-10-30T13:49:24.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c800bf00 con 0x7ff2c8004fe0 2023-10-30T13:49:24.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c800c0c0 con 0x7ff2c8004360 2023-10-30T13:49:24.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d3fff640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc06fdf0 0x7ff2dc072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:24.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2c80039b0 0x7ff2c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2dad9d640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c8004fe0 0x7ff2c800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2d3fff640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2c8004360 0x7ff2c8004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 -- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2c8004360 msgr2=0x7ff2c8004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:24.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2c8004360 0x7ff2c8004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 -- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c8004fe0 msgr2=0x7ff2c800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:24.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c8004fe0 0x7ff2c800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.110+0000 7ff2da59c640 1 -- 172.21.15.31:0/2597314607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8003530 con 0x7ff2c80039b0 2023-10-30T13:49:24.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2da59c640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2c80039b0 0x7ff2c8003d90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00c190 tx=0x7ff2cc01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444e1f2d388a56ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc005ae0 con 0x7ff2c80039b0 2023-10-30T13:49:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc005c60 con 0x7ff2c80039b0 2023-10-30T13:49:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2d9d9b640 1 -- 172.21.15.31:0/2597314607 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc007070 con 0x7ff2c80039b0 2023-10-30T13:49:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2e099d640 1 -- 172.21.15.31:0/2597314607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2c80039b0 msgr2=0x7ff2c8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2e099d640 1 --2- 172.21.15.31:0/2597314607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2c80039b0 0x7ff2c8003d90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00c190 tx=0x7ff2cc01cbe0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2dad9d640 1 -- 172.21.15.31:0/2597314607 reap_dead start 2023-10-30T13:49:24.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2e099d640 1 -- 172.21.15.31:0/2597314607 shutdown_connections 2023-10-30T13:49:24.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2e099d640 1 -- 172.21.15.31:0/2597314607 >> 172.21.15.31:0/2597314607 conn(0x7ff2dc069e90 msgr2=0x7ff2dc06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:24.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2e099d640 1 -- 172.21.15.31:0/2597314607 shutdown_connections 2023-10-30T13:49:24.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2e099d640 1 -- 172.21.15.31:0/2597314607 wait complete. 2023-10-30T13:49:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.114+0000 7ff2e099d640 1 Processor -- start 2023-10-30T13:49:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 -- start start 2023-10-30T13:49:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c80039b0 0x7ff2dc0bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2dc0c0330 0x7ff2dc1446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc1f2120 0x7ff2dc142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0c0100 con 0x7ff2dc0c0330 2023-10-30T13:49:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc161900 con 0x7ff2c80039b0 2023-10-30T13:49:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc161be0 con 0x7ff2dc1f2120 2023-10-30T13:49:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2dc0c0330 0x7ff2dc1446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2dc0c0330 0x7ff2dc1446c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43562/0 (socket says 172.21.15.31:43562) 2023-10-30T13:49:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 -- 172.21.15.31:0/865269851 learned_addr learned my addr 172.21.15.31:0/865269851 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2da59c640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c80039b0 0x7ff2dc0bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2dad9d640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc1f2120 0x7ff2dc142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 -- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc1f2120 msgr2=0x7ff2dc142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc1f2120 0x7ff2dc142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 -- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c80039b0 msgr2=0x7ff2dc0bfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c80039b0 0x7ff2dc0bfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 -- 172.21.15.31:0/865269851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc161ec0 con 0x7ff2dc0c0330 2023-10-30T13:49:24.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d3fff640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2dc0c0330 0x7ff2dc1446c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4011800 tx=0x7ff2c4011c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:24.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d2ffd640 1 -- 172.21.15.31:0/865269851 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c400f940 con 0x7ff2dc0c0330 2023-10-30T13:49:24.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d2ffd640 1 -- 172.21.15.31:0/865269851 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c400faa0 con 0x7ff2dc0c0330 2023-10-30T13:49:24.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2d2ffd640 1 -- 172.21.15.31:0/865269851 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c400fe50 con 0x7ff2dc0c0330 2023-10-30T13:49:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc1605a0 con 0x7ff2dc0c0330 2023-10-30T13:49:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2dc160b60 con 0x7ff2dc0c0330 2023-10-30T13:49:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.118+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc160d20 con 0x7ff2dc0c0330 2023-10-30T13:49:24.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.122+0000 7ff2d2ffd640 1 -- 172.21.15.31:0/865269851 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff2c4015460 con 0x7ff2dc0c0330 2023-10-30T13:49:24.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.122+0000 7ff2d2ffd640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2bc0664d0 0x7ff2bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.122+0000 7ff2d2ffd640 1 -- 172.21.15.31:0/865269851 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7ff2c40844e0 con 0x7ff2dc0c0330 2023-10-30T13:49:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.122+0000 7ff2da59c640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2bc0664d0 0x7ff2bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.122+0000 7ff2da59c640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2bc0664d0 0x7ff2bc068990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0ab000 tx=0x7ff2cc0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.126+0000 7ff2d2ffd640 1 -- 172.21.15.31:0/865269851 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2c4051620 con 0x7ff2dc0c0330 2023-10-30T13:49:24.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.314+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2dc151bb0 con 0x7ff2dc0c0330 2023-10-30T13:49:24.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.314+0000 7ff2d2ffd640 1 -- 172.21.15.31:0/865269851 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7ff2dc151bb0 con 0x7ff2dc0c0330 2023-10-30T13:49:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:24.319 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:24.319 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:24.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2bc0664d0 msgr2=0x7ff2bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2bc0664d0 0x7ff2bc068990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0ab000 tx=0x7ff2cc0a7070 comp rx=0 tx=0).stop 2023-10-30T13:49:24.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2dc0c0330 msgr2=0x7ff2dc1446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2dc0c0330 0x7ff2dc1446c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4011800 tx=0x7ff2c4011c00 comp rx=0 tx=0).stop 2023-10-30T13:49:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 shutdown_connections 2023-10-30T13:49:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2bc0664d0 0x7ff2bc068990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2dc1f2120 0x7ff2dc142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2dc0c0330 0x7ff2dc1446c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 --2- 172.21.15.31:0/865269851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2c80039b0 0x7ff2dc0bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 >> 172.21.15.31:0/865269851 conn(0x7ff2dc069e90 msgr2=0x7ff2dc148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 shutdown_connections 2023-10-30T13:49:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:24.318+0000 7ff2e099d640 1 -- 172.21.15.31:0/865269851 wait complete. 2023-10-30T13:49:24.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:49:26.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:26.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f48616d6640 1 Processor -- start 2023-10-30T13:49:26.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f48616d6640 1 -- start start 2023-10-30T13:49:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f48616d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f485c153c80 0x7f485c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f48616d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f485c0c42a0 0x7f485c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f48616d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c159f20 con 0x7f485c0c4a10 2023-10-30T13:49:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f48616d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c15dec0 con 0x7f485c153c80 2023-10-30T13:49:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f48616d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c1599a0 con 0x7f485c0c42a0 2023-10-30T13:49:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485affd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f485c153c80 0x7f485c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f4853fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f485c0c4a10 0x7f485c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:32976/0 (socket says 172.21.15.31:32976) 2023-10-30T13:49:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f4853fff640 1 -- 172.21.15.31:0/1948768534 learned_addr learned my addr 172.21.15.31:0/1948768534 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485b7fe640 1 --2- 172.21.15.31:0/1948768534 >> v2:172.21.15.33:3300/0 conn(0x7f485c0c42a0 0x7f485c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485a7fc640 1 -- 172.21.15.31:0/1948768534 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447337696 0 0) 0x7f485c159f20 con 0x7f485c0c4a10 2023-10-30T13:49:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485a7fc640 1 -- 172.21.15.31:0/1948768534 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848003530 con 0x7f485c0c4a10 2023-10-30T13:49:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485affd640 1 -- 172.21.15.31:0/1948768534 >> v2:172.21.15.33:3300/0 conn(0x7f485c0c42a0 msgr2=0x7f485c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485affd640 1 --2- 172.21.15.31:0/1948768534 >> v2:172.21.15.33:3300/0 conn(0x7f485c0c42a0 0x7f485c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485affd640 1 -- 172.21.15.31:0/1948768534 >> v1:172.21.15.31:6789/0 conn(0x7f485c0c4a10 legacy=0x7f485c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485affd640 1 -- 172.21.15.31:0/1948768534 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c15d620 con 0x7f485c153c80 2023-10-30T13:49:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.494+0000 7f485b7fe640 1 --2- 172.21.15.31:0/1948768534 >> v2:172.21.15.33:3300/0 conn(0x7f485c0c42a0 0x7f485c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:26.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f485affd640 1 --2- 172.21.15.31:0/1948768534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c15cf90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f485c0b1d40 tx=0x7f484c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e278fc856034c7e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:26.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f485a7fc640 1 -- 172.21.15.31:0/1948768534 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0a8950 con 0x7f485c153c80 2023-10-30T13:49:26.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f485a7fc640 1 -- 172.21.15.31:0/1948768534 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c0a8ab0 con 0x7f485c153c80 2023-10-30T13:49:26.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f485a7fc640 1 -- 172.21.15.31:0/1948768534 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0a8e40 con 0x7f485c153c80 2023-10-30T13:49:26.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 -- 172.21.15.31:0/1948768534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 msgr2=0x7f485c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 --2- 172.21.15.31:0/1948768534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c15cf90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f485c0b1d40 tx=0x7f484c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:49:26.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 -- 172.21.15.31:0/1948768534 shutdown_connections 2023-10-30T13:49:26.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 --2- 172.21.15.31:0/1948768534 >> v2:172.21.15.33:3300/0 conn(0x7f485c0c42a0 0x7f485c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 --2- 172.21.15.31:0/1948768534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c15cf90 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 -- 172.21.15.31:0/1948768534 >> 172.21.15.31:0/1948768534 conn(0x7f485c0715c0 msgr2=0x7f485c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:26.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 -- 172.21.15.31:0/1948768534 shutdown_connections 2023-10-30T13:49:26.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.498+0000 7f48616d6640 1 -- 172.21.15.31:0/1948768534 wait complete. 2023-10-30T13:49:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 Processor -- start 2023-10-30T13:49:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 -- start start 2023-10-30T13:49:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f485c0c42a0 0x7f485c14fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f485c0c4a10 0x7f485c154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c14df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c157340 con 0x7f485c153c80 2023-10-30T13:49:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c1efb20 con 0x7f485c0c4a10 2023-10-30T13:49:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0b5530 con 0x7f485c0c42a0 2023-10-30T13:49:26.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c14df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c14df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47028/0 (socket says 172.21.15.31:47028) 2023-10-30T13:49:26.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 -- 172.21.15.31:0/560508798 learned_addr learned my addr 172.21.15.31:0/560508798 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:26.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f4853fff640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f485c0c4a10 0x7f485c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485affd640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f485c0c42a0 0x7f485c14fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 -- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f485c0c42a0 msgr2=0x7f485c14fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f485c0c42a0 0x7f485c14fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 -- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f485c0c4a10 msgr2=0x7f485c154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f485c0c4a10 0x7f485c154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 -- 172.21.15.31:0/560508798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c009420 con 0x7f485c153c80 2023-10-30T13:49:26.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f485b7fe640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c14df30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4854007ef0 tx=0x7f485400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:26.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48537fe640 1 -- 172.21.15.31:0/560508798 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854003ba0 con 0x7f485c153c80 2023-10-30T13:49:26.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48537fe640 1 -- 172.21.15.31:0/560508798 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485400f040 con 0x7f485c153c80 2023-10-30T13:49:26.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48537fe640 1 -- 172.21.15.31:0/560508798 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485400de80 con 0x7f485c153c80 2023-10-30T13:49:26.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c1effb0 con 0x7f485c153c80 2023-10-30T13:49:26.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.502+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c1ef1f0 con 0x7f485c153c80 2023-10-30T13:49:26.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.506+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c1efc90 con 0x7f485c153c80 2023-10-30T13:49:26.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.506+0000 7f48537fe640 1 -- 172.21.15.31:0/560508798 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4854004030 con 0x7f485c153c80 2023-10-30T13:49:26.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.506+0000 7f48537fe640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f483c0664d0 0x7f483c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.506+0000 7f48537fe640 1 -- 172.21.15.31:0/560508798 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f48540843e0 con 0x7f485c153c80 2023-10-30T13:49:26.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.506+0000 7f485affd640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f483c0664d0 0x7f483c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.506+0000 7f485affd640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f483c0664d0 0x7f483c068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f484c0b2000 tx=0x7f484c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:26.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.510+0000 7f48537fe640 1 -- 172.21.15.31:0/560508798 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48540515e0 con 0x7f485c153c80 2023-10-30T13:49:26.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.702+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f485c04e170 con 0x7f485c153c80 2023-10-30T13:49:26.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.702+0000 7f48537fe640 1 -- 172.21.15.31:0/560508798 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f4854054c00 con 0x7f485c153c80 2023-10-30T13:49:26.708 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:26.708 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:26.708 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:26.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f483c0664d0 msgr2=0x7f483c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f483c0664d0 0x7f483c068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f484c0b2000 tx=0x7f484c008040 comp rx=0 tx=0).stop 2023-10-30T13:49:26.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 msgr2=0x7f485c14df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c14df30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4854007ef0 tx=0x7f485400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 shutdown_connections 2023-10-30T13:49:26.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f483c0664d0 0x7f483c068990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f485c153c80 0x7f485c14df30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f485c0c4a10 0x7f485c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 --2- 172.21.15.31:0/560508798 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f485c0c42a0 0x7f485c14fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 >> 172.21.15.31:0/560508798 conn(0x7f485c0715c0 msgr2=0x7f485c0c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:26.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 shutdown_connections 2023-10-30T13:49:26.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:26.706+0000 7f48616d6640 1 -- 172.21.15.31:0/560508798 wait complete. 2023-10-30T13:49:26.729 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:49:27.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:27.130 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:28.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b5613e640 1 Processor -- start 2023-10-30T13:49:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b5613e640 1 -- start start 2023-10-30T13:49:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b5613e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8b500c4a10 0x7f8b5015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b5613e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b50159f20 con 0x7f8b500c42a0 2023-10-30T13:49:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b5613e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5015d620 con 0x7f8b50153c80 2023-10-30T13:49:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b5613e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5015dec0 con 0x7f8b500c4a10 2023-10-30T13:49:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b4effd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8b500c4a10 0x7f8b5015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b4effd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8b500c4a10 0x7f8b5015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47036/0 (socket says 172.21.15.31:47036) 2023-10-30T13:49:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b4effd640 1 -- 172.21.15.31:0/103504622 learned_addr learned my addr 172.21.15.31:0/103504622 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:28.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b4effd640 1 -- 172.21.15.31:0/103504622 >> v1:172.21.15.33:6789/0 conn(0x7f8b50153c80 legacy=0x7f8b5015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:28.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b4effd640 1 -- 172.21.15.31:0/103504622 >> v1:172.21.15.31:6789/0 conn(0x7f8b500c42a0 legacy=0x7f8b500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:28.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b4effd640 1 -- 172.21.15.31:0/103504622 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5015d900 con 0x7f8b500c4a10 2023-10-30T13:49:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.882+0000 7f8b4e7fc640 1 -- 172.21.15.31:0/103504622 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696142894 0 0) 0x7f8b50159f20 con 0x7f8b500c42a0 2023-10-30T13:49:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b4effd640 1 --2- 172.21.15.31:0/103504622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b500c4a10 0x7f8b5015cf90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009870 tx=0x7f8b4401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf745af3e2361fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:28.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b4e7fc640 1 -- 172.21.15.31:0/103504622 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b440a8a30 con 0x7f8b500c4a10 2023-10-30T13:49:28.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b4e7fc640 1 -- 172.21.15.31:0/103504622 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b440a8b90 con 0x7f8b500c4a10 2023-10-30T13:49:28.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b4e7fc640 1 -- 172.21.15.31:0/103504622 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b440b0440 con 0x7f8b500c4a10 2023-10-30T13:49:28.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b5613e640 1 -- 172.21.15.31:0/103504622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b500c4a10 msgr2=0x7f8b5015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:28.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b5613e640 1 --2- 172.21.15.31:0/103504622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b500c4a10 0x7f8b5015cf90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009870 tx=0x7f8b4401ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:28.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b5613e640 1 -- 172.21.15.31:0/103504622 shutdown_connections 2023-10-30T13:49:28.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b5613e640 1 --2- 172.21.15.31:0/103504622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b500c4a10 0x7f8b5015cf90 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:28.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b5613e640 1 -- 172.21.15.31:0/103504622 >> 172.21.15.31:0/103504622 conn(0x7f8b500715c0 msgr2=0x7f8b500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:28.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b5613e640 1 -- 172.21.15.31:0/103504622 shutdown_connections 2023-10-30T13:49:28.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.886+0000 7f8b5613e640 1 -- 172.21.15.31:0/103504622 wait complete. 2023-10-30T13:49:28.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 Processor -- start 2023-10-30T13:49:28.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 -- start start 2023-10-30T13:49:28.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b500c42a0 0x7f8b50149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:28.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b500c4a10 0x7f8b5006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b50153c80 0x7f8b5006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5015cd70 con 0x7f8b50153c80 2023-10-30T13:49:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500b4fb0 con 0x7f8b500c42a0 2023-10-30T13:49:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5015d2a0 con 0x7f8b500c4a10 2023-10-30T13:49:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b50153c80 0x7f8b5006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b50153c80 0x7f8b5006d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47046/0 (socket says 172.21.15.31:47046) 2023-10-30T13:49:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 -- 172.21.15.31:0/1452795487 learned_addr learned my addr 172.21.15.31:0/1452795487 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4effd640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b500c4a10 0x7f8b5006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4f7fe640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b500c42a0 0x7f8b50149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 -- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b500c4a10 msgr2=0x7f8b5006d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b500c4a10 0x7f8b5006d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 -- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b500c42a0 msgr2=0x7f8b50149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b500c42a0 0x7f8b50149de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 -- 172.21.15.31:0/1452795487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440094d0 con 0x7f8b50153c80 2023-10-30T13:49:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b4ffff640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b50153c80 0x7f8b5006d6f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800c780 tx=0x7f8b4800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b37fff640 1 -- 172.21.15.31:0/1452795487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48003db0 con 0x7f8b50153c80 2023-10-30T13:49:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b37fff640 1 -- 172.21.15.31:0/1452795487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4800ce90 con 0x7f8b50153c80 2023-10-30T13:49:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b37fff640 1 -- 172.21.15.31:0/1452795487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b480052b0 con 0x7f8b50153c80 2023-10-30T13:49:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5014fcb0 con 0x7f8b50153c80 2023-10-30T13:49:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.890+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5015cfc0 con 0x7f8b50153c80 2023-10-30T13:49:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.894+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5006f210 con 0x7f8b50153c80 2023-10-30T13:49:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.894+0000 7f8b37fff640 1 -- 172.21.15.31:0/1452795487 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b48005410 con 0x7f8b50153c80 2023-10-30T13:49:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.894+0000 7f8b37fff640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8b3c0664a0 0x7f8b3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.894+0000 7f8b37fff640 1 -- 172.21.15.31:0/1452795487 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f8b48083ed0 con 0x7f8b50153c80 2023-10-30T13:49:28.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.894+0000 7f8b4f7fe640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8b3c0664a0 0x7f8b3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:28.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.898+0000 7f8b4f7fe640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8b3c0664a0 0x7f8b3c068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000f70 tx=0x7f8b40009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:28.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:28.898+0000 7f8b37fff640 1 -- 172.21.15.31:0/1452795487 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b480510a0 con 0x7f8b50153c80 2023-10-30T13:49:29.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b5004e170 con 0x7f8b50153c80 2023-10-30T13:49:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b37fff640 1 -- 172.21.15.31:0/1452795487 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f8b480546c0 con 0x7f8b50153c80 2023-10-30T13:49:29.093 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:29.093 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:29.093 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:29.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8b3c0664a0 msgr2=0x7f8b3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8b3c0664a0 0x7f8b3c068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000f70 tx=0x7f8b40009040 comp rx=0 tx=0).stop 2023-10-30T13:49:29.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b50153c80 msgr2=0x7f8b5006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b50153c80 0x7f8b5006d6f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800c780 tx=0x7f8b4800cc40 comp rx=0 tx=0).stop 2023-10-30T13:49:29.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 shutdown_connections 2023-10-30T13:49:29.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8b3c0664a0 0x7f8b3c068960 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b50153c80 0x7f8b5006d6f0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8b500c4a10 0x7f8b5006d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 --2- 172.21.15.31:0/1452795487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b500c42a0 0x7f8b50149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 >> 172.21.15.31:0/1452795487 conn(0x7f8b500715c0 msgr2=0x7f8b50151770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 shutdown_connections 2023-10-30T13:49:29.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:29.090+0000 7f8b5613e640 1 -- 172.21.15.31:0/1452795487 wait complete. 2023-10-30T13:49:29.114 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:49:31.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:31.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.266+0000 7f926a7f8640 1 Processor -- start 2023-10-30T13:49:31.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.266+0000 7f926a7f8640 1 -- start start 2023-10-30T13:49:31.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f926a7f8640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f926415d840 0x7f92640ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f926a7f8640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92640c7b00 con 0x7f92640cd420 2023-10-30T13:49:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f926a7f8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92640c2050 con 0x7f9264153aa0 2023-10-30T13:49:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f926a7f8640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9264070a50 con 0x7f926415d840 2023-10-30T13:49:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9268d95640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f92640cd420 0x7f926406b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:33008/0 (socket says 172.21.15.31:33008) 2023-10-30T13:49:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9268d95640 1 -- 172.21.15.31:0/3985269850 learned_addr learned my addr 172.21.15.31:0/3985269850 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:31.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f92637fe640 1 --2- 172.21.15.31:0/3985269850 >> v2:172.21.15.33:3300/0 conn(0x7f926415d840 0x7f92640ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356157986 0 0) 0x7f92640c7b00 con 0x7f92640cd420 2023-10-30T13:49:31.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f924c003530 con 0x7f92640cd420 2023-10-30T13:49:31.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229067407 0 0) 0x7f924c003530 con 0x7f92640cd420 2023-10-30T13:49:31.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 >> v2:172.21.15.33:3300/0 conn(0x7f926415d840 msgr2=0x7f92640ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:31.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 --2- 172.21.15.31:0/3985269850 >> v2:172.21.15.33:3300/0 conn(0x7f926415d840 0x7f92640ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 >> v1:172.21.15.33:6789/0 conn(0x7f9264153aa0 legacy=0x7f92640701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92640c2330 con 0x7f92640cd420 2023-10-30T13:49:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f925c001a00 con 0x7f92640cd420 2023-10-30T13:49:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 >> v1:172.21.15.31:6789/0 conn(0x7f92640cd420 legacy=0x7f926406b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f924c003980 0x7f924c003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c004380 0x7f924c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f924c005050 0x7f924c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9264070a50 con 0x7f924c003980 2023-10-30T13:49:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c00bf00 con 0x7f924c004380 2023-10-30T13:49:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c00c0c0 con 0x7f924c005050 2023-10-30T13:49:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9263fff640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f924c003980 0x7f924c003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f92637fe640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c004380 0x7f924c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9268d95640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f924c005050 0x7f924c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9263fff640 1 -- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f924c005050 msgr2=0x7f924c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9263fff640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f924c005050 0x7f924c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9263fff640 1 -- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c004380 msgr2=0x7f924c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9263fff640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c004380 0x7f924c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9263fff640 1 -- 172.21.15.31:0/3985269850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c003530 con 0x7f924c003980 2023-10-30T13:49:31.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.270+0000 7f9268d95640 1 -- 172.21.15.31:0/3985269850 reap_dead start 2023-10-30T13:49:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f9263fff640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f924c003980 0x7f924c003d60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9254001860 tx=0x7f925401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e2e070786f1a21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:31.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92540a8870 con 0x7f924c003980 2023-10-30T13:49:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92540a89d0 con 0x7f924c003980 2023-10-30T13:49:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f9262ffd640 1 -- 172.21.15.31:0/3985269850 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92540a8dc0 con 0x7f924c003980 2023-10-30T13:49:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f926a7f8640 1 -- 172.21.15.31:0/3985269850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f924c003980 msgr2=0x7f924c003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f926a7f8640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f924c003980 0x7f924c003d60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9254001860 tx=0x7f925401ec50 comp rx=0 tx=0).stop 2023-10-30T13:49:31.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f926a7f8640 1 -- 172.21.15.31:0/3985269850 shutdown_connections 2023-10-30T13:49:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f926a7f8640 1 --2- 172.21.15.31:0/3985269850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f924c003980 0x7f924c003d60 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f926a7f8640 1 -- 172.21.15.31:0/3985269850 >> 172.21.15.31:0/3985269850 conn(0x7f9264071fb0 msgr2=0x7f92640723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f926a7f8640 1 -- 172.21.15.31:0/3985269850 shutdown_connections 2023-10-30T13:49:31.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.274+0000 7f926a7f8640 1 -- 172.21.15.31:0/3985269850 wait complete. 2023-10-30T13:49:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 Processor -- start 2023-10-30T13:49:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 -- start start 2023-10-30T13:49:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c003980 0x7f92641ef5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92640c43b0 0x7f9264152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92640cca60 0x7f9264158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92640cd010 con 0x7f92640c43b0 2023-10-30T13:49:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92641efb30 con 0x7f924c003980 2023-10-30T13:49:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92640cbb60 con 0x7f92640cca60 2023-10-30T13:49:31.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92640c43b0 0x7f9264152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92640c43b0 0x7f9264152530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47060/0 (socket says 172.21.15.31:47060) 2023-10-30T13:49:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 -- 172.21.15.31:0/4291873130 learned_addr learned my addr 172.21.15.31:0/4291873130 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f9268d95640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92640cca60 0x7f9264158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f9263fff640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c003980 0x7f92641ef5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 -- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92640cca60 msgr2=0x7f9264158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92640cca60 0x7f9264158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 -- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c003980 msgr2=0x7f92641ef5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c003980 0x7f92641ef5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 -- 172.21.15.31:0/4291873130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92640caf50 con 0x7f92640c43b0 2023-10-30T13:49:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f9268d95640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92640cca60 0x7f9264158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f9263fff640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c003980 0x7f92641ef5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f92637fe640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92640c43b0 0x7f9264152530 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9258011aa0 tx=0x7f9258011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f9260ff9640 1 -- 172.21.15.31:0/4291873130 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925800fb80 con 0x7f92640c43b0 2023-10-30T13:49:31.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f9260ff9640 1 -- 172.21.15.31:0/4291873130 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925800fce0 con 0x7f92640c43b0 2023-10-30T13:49:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92640cb230 con 0x7f92640c43b0 2023-10-30T13:49:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f9260ff9640 1 -- 172.21.15.31:0/4291873130 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258014680 con 0x7f92640c43b0 2023-10-30T13:49:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.278+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92640cb7f0 con 0x7f92640c43b0 2023-10-30T13:49:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.282+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92640cbd10 con 0x7f92640c43b0 2023-10-30T13:49:31.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.282+0000 7f9260ff9640 1 -- 172.21.15.31:0/4291873130 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f925801d400 con 0x7f92640c43b0 2023-10-30T13:49:31.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.282+0000 7f9260ff9640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f92500664a0 0x7f9250068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.282+0000 7f9260ff9640 1 -- 172.21.15.31:0/4291873130 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f9258084a60 con 0x7f92640c43b0 2023-10-30T13:49:31.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.282+0000 7f9263fff640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f92500664a0 0x7f9250068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.282+0000 7f9263fff640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f92500664a0 0x7f9250068960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f92540b2000 tx=0x7f92540aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.286+0000 7f9260ff9640 1 -- 172.21.15.31:0/4291873130 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92640cbd10 con 0x7f92640c43b0 2023-10-30T13:49:31.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.478+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92640cbd10 con 0x7f92640c43b0 2023-10-30T13:49:31.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.478+0000 7f9260ff9640 1 -- 172.21.15.31:0/4291873130 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f92640cbd10 con 0x7f92640c43b0 2023-10-30T13:49:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:31.482 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:31.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.478+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f92500664a0 msgr2=0x7f9250068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.478+0000 7f926a7f8640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f92500664a0 0x7f9250068960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f92540b2000 tx=0x7f92540aa070 comp rx=0 tx=0).stop 2023-10-30T13:49:31.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.478+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92640c43b0 msgr2=0x7f9264152530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.478+0000 7f926a7f8640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92640c43b0 0x7f9264152530 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9258011aa0 tx=0x7f9258011f60 comp rx=0 tx=0).stop 2023-10-30T13:49:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 shutdown_connections 2023-10-30T13:49:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f92500664a0 0x7f9250068960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f92640cca60 0x7f9264158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92640c43b0 0x7f9264152530 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 --2- 172.21.15.31:0/4291873130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f924c003980 0x7f92641ef5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 >> 172.21.15.31:0/4291873130 conn(0x7f9264071fb0 msgr2=0x7f926406c680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:31.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 shutdown_connections 2023-10-30T13:49:31.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:31.482+0000 7f926a7f8640 1 -- 172.21.15.31:0/4291873130 wait complete. 2023-10-30T13:49:31.503 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:49:33.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:33.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b1e13e640 1 Processor -- start 2023-10-30T13:49:33.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b1e13e640 1 -- start start 2023-10-30T13:49:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b1e13e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2b1815cf90 0x7f2b180c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b1e13e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b18159f20 con 0x7f2b18153c80 2023-10-30T13:49:33.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b1e13e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b180c47e0 con 0x7f2b180c4a10 2023-10-30T13:49:33.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b1e13e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1815dec0 con 0x7f2b1815cf90 2023-10-30T13:49:33.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b177fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2b18153c80 0x7f2b1815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:33016/0 (socket says 172.21.15.31:33016) 2023-10-30T13:49:33.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b177fe640 1 -- 172.21.15.31:0/3478610876 learned_addr learned my addr 172.21.15.31:0/3478610876 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:33.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b17fff640 1 --2- 172.21.15.31:0/3478610876 >> v2:172.21.15.33:3300/0 conn(0x7f2b1815cf90 0x7f2b180c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311650152 0 0) 0x7f2b18159f20 con 0x7f2b18153c80 2023-10-30T13:49:33.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b00003530 con 0x7f2b18153c80 2023-10-30T13:49:33.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3904881584 0 0) 0x7f2b00003530 con 0x7f2b18153c80 2023-10-30T13:49:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 >> v2:172.21.15.33:3300/0 conn(0x7f2b1815cf90 msgr2=0x7f2b180c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 --2- 172.21.15.31:0/3478610876 >> v2:172.21.15.33:3300/0 conn(0x7f2b1815cf90 0x7f2b180c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 >> v1:172.21.15.33:6789/0 conn(0x7f2b180c4a10 legacy=0x7f2b180ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1815d620 con 0x7f2b18153c80 2023-10-30T13:49:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f2b08001680 con 0x7f2b18153c80 2023-10-30T13:49:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 >> v1:172.21.15.31:6789/0 conn(0x7f2b18153c80 legacy=0x7f2b1815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b00003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b000047d0 0x7f2b00004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b00005190 0x7f2b0000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1815dec0 con 0x7f2b00003980 2023-10-30T13:49:33.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0000c040 con 0x7f2b000047d0 2023-10-30T13:49:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0000c200 con 0x7f2b00005190 2023-10-30T13:49:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b177fe640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b00003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b16ffd640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b000047d0 0x7f2b00004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.658+0000 7f2b17fff640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b00005190 0x7f2b0000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b177fe640 1 -- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b00005190 msgr2=0x7f2b0000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:33.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b177fe640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b00005190 0x7f2b0000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b177fe640 1 -- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b000047d0 msgr2=0x7f2b00004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:33.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b177fe640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b000047d0 0x7f2b00004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b177fe640 1 -- 172.21.15.31:0/3478610876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b00003530 con 0x7f2b00003980 2023-10-30T13:49:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b17fff640 1 -- 172.21.15.31:0/3478610876 reap_dead start 2023-10-30T13:49:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b16ffd640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b000047d0 0x7f2b00004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b177fe640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b00003da0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800c210 tx=0x7f2b0801f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5bac11058f2eed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08005ae0 con 0x7f2b00003980 2023-10-30T13:49:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b080ab070 con 0x7f2b00003980 2023-10-30T13:49:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b167fc640 1 -- 172.21.15.31:0/3478610876 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08005e80 con 0x7f2b00003980 2023-10-30T13:49:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b1e13e640 1 -- 172.21.15.31:0/3478610876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 msgr2=0x7f2b00003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b00003da0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800c210 tx=0x7f2b0801f310 comp rx=0 tx=0).stop 2023-10-30T13:49:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b1e13e640 1 -- 172.21.15.31:0/3478610876 shutdown_connections 2023-10-30T13:49:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/3478610876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b00003da0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b1e13e640 1 -- 172.21.15.31:0/3478610876 >> 172.21.15.31:0/3478610876 conn(0x7f2b180715c0 msgr2=0x7f2b180719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b1e13e640 1 -- 172.21.15.31:0/3478610876 shutdown_connections 2023-10-30T13:49:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.662+0000 7f2b1e13e640 1 -- 172.21.15.31:0/3478610876 wait complete. 2023-10-30T13:49:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 Processor -- start 2023-10-30T13:49:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 -- start start 2023-10-30T13:49:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b1815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b181557a0 0x7f2b181f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b181f0840 0x7f2b181f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b18157490 con 0x7f2b00003980 2023-10-30T13:49:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b181554e0 con 0x7f2b181f0840 2023-10-30T13:49:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b181efe70 con 0x7f2b181557a0 2023-10-30T13:49:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b1815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b1815c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47072/0 (socket says 172.21.15.31:47072) 2023-10-30T13:49:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 -- 172.21.15.31:0/991900210 learned_addr learned my addr 172.21.15.31:0/991900210 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b16ffd640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b181557a0 0x7f2b181f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b17fff640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b181f0840 0x7f2b181f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 -- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b181557a0 msgr2=0x7f2b181f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b181557a0 0x7f2b181f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 -- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b181f0840 msgr2=0x7f2b181f1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b181f0840 0x7f2b181f1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 -- 172.21.15.31:0/991900210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18151530 con 0x7f2b00003980 2023-10-30T13:49:33.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b177fe640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b1815c020 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2b0809ba60 tx=0x7f2b00005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:33.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2afffff640 1 -- 172.21.15.31:0/991900210 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b080b6070 con 0x7f2b00003980 2023-10-30T13:49:33.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2afffff640 1 -- 172.21.15.31:0/991900210 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b080048b0 con 0x7f2b00003980 2023-10-30T13:49:33.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2afffff640 1 -- 172.21.15.31:0/991900210 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08004c80 con 0x7f2b00003980 2023-10-30T13:49:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b181566b0 con 0x7f2b00003980 2023-10-30T13:49:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.666+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b18152150 con 0x7f2b00003980 2023-10-30T13:49:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.670+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1804e170 con 0x7f2b00003980 2023-10-30T13:49:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.670+0000 7f2afffff640 1 -- 172.21.15.31:0/991900210 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b08007070 con 0x7f2b00003980 2023-10-30T13:49:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.670+0000 7f2afffff640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2b040664a0 0x7f2b04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:33.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.670+0000 7f2afffff640 1 -- 172.21.15.31:0/991900210 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f2b08120d40 con 0x7f2b00003980 2023-10-30T13:49:33.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.670+0000 7f2b16ffd640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2b040664a0 0x7f2b04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.674+0000 7f2b16ffd640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2b040664a0 0x7f2b04068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000c10 tx=0x7f2b0c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.674+0000 7f2afffff640 1 -- 172.21.15.31:0/991900210 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b080eddf0 con 0x7f2b00003980 2023-10-30T13:49:33.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.862+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b181efc40 con 0x7f2b00003980 2023-10-30T13:49:33.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2afffff640 1 -- 172.21.15.31:0/991900210 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f2b080c0070 con 0x7f2b00003980 2023-10-30T13:49:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:33.868 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:33.868 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:33.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2b040664a0 msgr2=0x7f2b04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:33.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2b040664a0 0x7f2b04068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000c10 tx=0x7f2b0c009040 comp rx=0 tx=0).stop 2023-10-30T13:49:33.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 msgr2=0x7f2b1815c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:33.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b1815c020 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2b0809ba60 tx=0x7f2b00005800 comp rx=0 tx=0).stop 2023-10-30T13:49:33.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 shutdown_connections 2023-10-30T13:49:33.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2b040664a0 0x7f2b04068960 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b181f0840 0x7f2b181f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b181557a0 0x7f2b181f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 --2- 172.21.15.31:0/991900210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b00003980 0x7f2b1815c020 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:33.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 >> 172.21.15.31:0/991900210 conn(0x7f2b180715c0 msgr2=0x7f2b180c5290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:33.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 shutdown_connections 2023-10-30T13:49:33.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:33.866+0000 7f2b1e13e640 1 -- 172.21.15.31:0/991900210 wait complete. 2023-10-30T13:49:33.889 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:49:35.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:36.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2b04a6b640 1 Processor -- start 2023-10-30T13:49:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2b04a6b640 1 -- start start 2023-10-30T13:49:36.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2b04a6b640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2b0006b3b0 0x7f2b0006b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2b04a6b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b000c7b00 con 0x7f2b00153aa0 2023-10-30T13:49:36.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2b04a6b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0006bd20 con 0x7f2b0015d840 2023-10-30T13:49:36.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2b04a6b640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b00070a50 con 0x7f2b0006b3b0 2023-10-30T13:49:36.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2afe59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2b00153aa0 0x7f2b000701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58688/0 (socket says 172.21.15.31:58688) 2023-10-30T13:49:36.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.042+0000 7f2afe59c640 1 -- 172.21.15.31:0/222926410 learned_addr learned my addr 172.21.15.31:0/222926410 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:36.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 --2- 172.21.15.31:0/222926410 >> v2:172.21.15.33:3300/0 conn(0x7f2b0006b3b0 0x7f2b0006b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736268786 0 0) 0x7f2b000c7b00 con 0x7f2b00153aa0 2023-10-30T13:49:36.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec003530 con 0x7f2b00153aa0 2023-10-30T13:49:36.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2624048095 0 0) 0x7f2aec003530 con 0x7f2b00153aa0 2023-10-30T13:49:36.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 >> v2:172.21.15.33:3300/0 conn(0x7f2b0006b3b0 msgr2=0x7f2b0006b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:36.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 --2- 172.21.15.31:0/222926410 >> v2:172.21.15.33:3300/0 conn(0x7f2b0006b3b0 0x7f2b0006b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 >> v1:172.21.15.33:6789/0 conn(0x7f2b0015d840 legacy=0x7f2b000ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b000c2050 con 0x7f2b00153aa0 2023-10-30T13:49:36.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f2af0001680 con 0x7f2b00153aa0 2023-10-30T13:49:36.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 >> v1:172.21.15.31:6789/0 conn(0x7f2b00153aa0 legacy=0x7f2b000701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec003980 0x7f2aec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2aec0047d0 0x7f2aec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aec005190 0x7f2aec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00070a50 con 0x7f2aec005190 2023-10-30T13:49:36.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec00c040 con 0x7f2aec003980 2023-10-30T13:49:36.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec00c200 con 0x7f2aec0047d0 2023-10-30T13:49:36.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 --2- 172.21.15.31:0/222926410 >> v2:172.21.15.33:3300/0 conn(0x7f2b0006b3b0 0x7f2b0006b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:36.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aec005190 0x7f2aec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afe59c640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec003980 0x7f2aec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2af7fff640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2aec0047d0 0x7f2aec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 -- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2aec0047d0 msgr2=0x7f2aec004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:36.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2aec0047d0 0x7f2aec004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 -- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec003980 msgr2=0x7f2aec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:36.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec003980 0x7f2aec003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.046+0000 7f2afed9d640 1 -- 172.21.15.31:0/222926410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec003530 con 0x7f2aec005190 2023-10-30T13:49:36.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2afed9d640 1 -- 172.21.15.31:0/222926410 reap_dead start 2023-10-30T13:49:36.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2afed9d640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aec005190 0x7f2aec00b990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2af800e350 tx=0x7f2af801ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b9179ede9c998 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:36.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af80a5e50 con 0x7f2aec005190 2023-10-30T13:49:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af800f040 con 0x7f2aec005190 2023-10-30T13:49:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2afdd9b640 1 -- 172.21.15.31:0/222926410 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8003be0 con 0x7f2aec005190 2023-10-30T13:49:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 -- 172.21.15.31:0/222926410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aec005190 msgr2=0x7f2aec00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aec005190 0x7f2aec00b990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2af800e350 tx=0x7f2af801ce40 comp rx=0 tx=0).stop 2023-10-30T13:49:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 -- 172.21.15.31:0/222926410 shutdown_connections 2023-10-30T13:49:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/222926410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2aec005190 0x7f2aec00b990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 -- 172.21.15.31:0/222926410 >> 172.21.15.31:0/222926410 conn(0x7f2b00071fb0 msgr2=0x7f2b000723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 -- 172.21.15.31:0/222926410 shutdown_connections 2023-10-30T13:49:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 -- 172.21.15.31:0/222926410 wait complete. 2023-10-30T13:49:36.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 Processor -- start 2023-10-30T13:49:36.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.050+0000 7f2b04a6b640 1 -- start start 2023-10-30T13:49:36.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec005190 0x7f2b00156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b001581f0 0x7f2b001576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b001587c0 0x7f2b00157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00159500 con 0x7f2b001581f0 2023-10-30T13:49:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b000ca080 con 0x7f2aec005190 2023-10-30T13:49:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b001ef580 con 0x7f2b001587c0 2023-10-30T13:49:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b001581f0 0x7f2b001576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b001581f0 0x7f2b001576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36244/0 (socket says 172.21.15.31:36244) 2023-10-30T13:49:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 -- 172.21.15.31:0/591325872 learned_addr learned my addr 172.21.15.31:0/591325872 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2afe59c640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec005190 0x7f2b00156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2afed9d640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b001587c0 0x7f2b00157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 -- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b001587c0 msgr2=0x7f2b00157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b001587c0 0x7f2b00157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 -- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec005190 msgr2=0x7f2b00156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec005190 0x7f2b00156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 -- 172.21.15.31:0/591325872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b000ca2d0 con 0x7f2b001581f0 2023-10-30T13:49:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af7fff640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b001581f0 0x7f2b001576d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800f5f0 tx=0x7f2ae800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af6ffd640 1 -- 172.21.15.31:0/591325872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae80052e0 con 0x7f2b001581f0 2023-10-30T13:49:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af6ffd640 1 -- 172.21.15.31:0/591325872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae8005440 con 0x7f2b001581f0 2023-10-30T13:49:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af6ffd640 1 -- 172.21.15.31:0/591325872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8003820 con 0x7f2b001581f0 2023-10-30T13:49:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b000ca5b0 con 0x7f2b001581f0 2023-10-30T13:49:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b001ef1f0 con 0x7f2b001581f0 2023-10-30T13:49:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b00159640 con 0x7f2b001581f0 2023-10-30T13:49:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.054+0000 7f2af6ffd640 1 -- 172.21.15.31:0/591325872 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ae8003980 con 0x7f2b001581f0 2023-10-30T13:49:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.058+0000 7f2af6ffd640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2ae00664d0 0x7f2ae0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.058+0000 7f2af6ffd640 1 -- 172.21.15.31:0/591325872 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f2ae8083fe0 con 0x7f2b001581f0 2023-10-30T13:49:36.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.058+0000 7f2afe59c640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2ae00664d0 0x7f2ae0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.058+0000 7f2afe59c640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2ae00664d0 0x7f2ae0068990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2af0001210 tx=0x7f2af0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:36.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.062+0000 7f2af6ffd640 1 -- 172.21.15.31:0/591325872 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b00159640 con 0x7f2b001581f0 2023-10-30T13:49:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.250+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b00159640 con 0x7f2b001581f0 2023-10-30T13:49:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.250+0000 7f2af6ffd640 1 -- 172.21.15.31:0/591325872 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6484 (secure 0 0 0) 0x7f2b00159640 con 0x7f2b001581f0 2023-10-30T13:49:36.254 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:36.255 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:47:37.115829+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5842},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5842":{"gid":5842,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":44,"addr":"172.21.15.33:6836/15857146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":15857146},{"type":"v1","addr":"172.21.15.33:6836","nonce":15857146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2023-10-30T13:49:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2ae00664d0 msgr2=0x7f2ae0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2ae00664d0 0x7f2ae0068990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2af0001210 tx=0x7f2af0009040 comp rx=0 tx=0).stop 2023-10-30T13:49:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b001581f0 msgr2=0x7f2b001576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b001581f0 0x7f2b001576d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800f5f0 tx=0x7f2ae800fab0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 shutdown_connections 2023-10-30T13:49:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2ae00664d0 0x7f2ae0068990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2b001587c0 0x7f2b00157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b001581f0 0x7f2b001576d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 --2- 172.21.15.31:0/591325872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2aec005190 0x7f2b00156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 >> 172.21.15.31:0/591325872 conn(0x7f2b00071fb0 msgr2=0x7f2b0006ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:36.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 shutdown_connections 2023-10-30T13:49:36.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:36.254+0000 7f2b04a6b640 1 -- 172.21.15.31:0/591325872 wait complete. 2023-10-30T13:49:36.276 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:49:38.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:38.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 Processor -- start 2023-10-30T13:49:38.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 -- start start 2023-10-30T13:49:38.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e84151840 0x7f9e840c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f9e84151c20 0x7f9e84072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9e840cb8b0 0x7f9e84072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e840c5030 con 0x7f9e840cb8b0 2023-10-30T13:49:38.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e84157830 con 0x7f9e84151c20 2023-10-30T13:49:38.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e8933e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e84157b10 con 0x7f9e84151840 2023-10-30T13:49:38.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9e840cb8b0 0x7f9e84072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f9e840cb8b0 0x7f9e84072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36258/0 (socket says 172.21.15.31:36258) 2023-10-30T13:49:38.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 -- 172.21.15.31:0/2384159452 learned_addr learned my addr 172.21.15.31:0/2384159452 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:38.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e82ffd640 1 --2- 172.21.15.31:0/2384159452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e84151840 0x7f9e840c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e827fc640 1 --2- 172.21.15.31:0/2384159452 >> v2:172.21.15.33:3300/0 conn(0x7f9e84151c20 0x7f9e84072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 -- 172.21.15.31:0/2384159452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e84151840 msgr2=0x7f9e840c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 --2- 172.21.15.31:0/2384159452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e84151840 0x7f9e840c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 -- 172.21.15.31:0/2384159452 >> v2:172.21.15.33:3300/0 conn(0x7f9e84151c20 msgr2=0x7f9e84072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 --2- 172.21.15.31:0/2384159452 >> v2:172.21.15.33:3300/0 conn(0x7f9e84151c20 0x7f9e84072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.426+0000 7f9e837fe640 1 -- 172.21.15.31:0/2384159452 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e84157df0 con 0x7f9e840cb8b0 2023-10-30T13:49:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e837fe640 1 --2- 172.21.15.31:0/2384159452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e840cb8b0 0x7f9e84072540 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9e8406fa70 tx=0x7f9e7c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4c1f5e200fe594 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e81ffb640 1 -- 172.21.15.31:0/2384159452 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c0af400 con 0x7f9e840cb8b0 2023-10-30T13:49:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e81ffb640 1 -- 172.21.15.31:0/2384159452 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c0ab050 con 0x7f9e840cb8b0 2023-10-30T13:49:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e81ffb640 1 -- 172.21.15.31:0/2384159452 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c0af780 con 0x7f9e840cb8b0 2023-10-30T13:49:38.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 -- 172.21.15.31:0/2384159452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e840cb8b0 msgr2=0x7f9e84072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 --2- 172.21.15.31:0/2384159452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e840cb8b0 0x7f9e84072540 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9e8406fa70 tx=0x7f9e7c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 -- 172.21.15.31:0/2384159452 shutdown_connections 2023-10-30T13:49:38.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 --2- 172.21.15.31:0/2384159452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e840cb8b0 0x7f9e84072540 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 --2- 172.21.15.31:0/2384159452 >> v2:172.21.15.33:3300/0 conn(0x7f9e84151c20 0x7f9e84072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 --2- 172.21.15.31:0/2384159452 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e84151840 0x7f9e840c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 -- 172.21.15.31:0/2384159452 >> 172.21.15.31:0/2384159452 conn(0x7f9e840cac70 msgr2=0x7f9e840cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:38.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 -- 172.21.15.31:0/2384159452 shutdown_connections 2023-10-30T13:49:38.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.430+0000 7f9e8933e640 1 -- 172.21.15.31:0/2384159452 wait complete. 2023-10-30T13:49:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 Processor -- start 2023-10-30T13:49:38.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 -- start start 2023-10-30T13:49:38.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e840cb8b0 0x7f9e84145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e84151840 0x7f9e841f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84151c20 0x7f9e841565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e841e30d0 con 0x7f9e84151840 2023-10-30T13:49:38.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e84150ac0 con 0x7f9e84151c20 2023-10-30T13:49:38.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8414f610 con 0x7f9e840cb8b0 2023-10-30T13:49:38.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e84151840 0x7f9e841f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e84151840 0x7f9e841f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36270/0 (socket says 172.21.15.31:36270) 2023-10-30T13:49:38.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 -- 172.21.15.31:0/1520011088 learned_addr learned my addr 172.21.15.31:0/1520011088 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:38.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e82ffd640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e840cb8b0 0x7f9e84145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e837fe640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84151c20 0x7f9e841565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 -- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e840cb8b0 msgr2=0x7f9e84145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e840cb8b0 0x7f9e84145530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 -- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84151c20 msgr2=0x7f9e841565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84151c20 0x7f9e841565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 -- 172.21.15.31:0/1520011088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c009b30 con 0x7f9e84151840 2023-10-30T13:49:38.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e837fe640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84151c20 0x7f9e841565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:38.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e827fc640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e84151840 0x7f9e841f2200 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800d800 tx=0x7f9e7800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e6b7fe640 1 -- 172.21.15.31:0/1520011088 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800b6c0 con 0x7f9e84151840 2023-10-30T13:49:38.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e6b7fe640 1 -- 172.21.15.31:0/1520011088 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7800b820 con 0x7f9e84151840 2023-10-30T13:49:38.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e841e2120 con 0x7f9e84151840 2023-10-30T13:49:38.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e6b7fe640 1 -- 172.21.15.31:0/1520011088 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800bbb0 con 0x7f9e84151840 2023-10-30T13:49:38.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.434+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e84145da0 con 0x7f9e84151840 2023-10-30T13:49:38.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.438+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e841580b0 con 0x7f9e84151840 2023-10-30T13:49:38.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.438+0000 7f9e6b7fe640 1 -- 172.21.15.31:0/1520011088 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e7800bd10 con 0x7f9e84151840 2023-10-30T13:49:38.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.438+0000 7f9e6b7fe640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e700664a0 0x7f9e70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.438+0000 7f9e6b7fe640 1 -- 172.21.15.31:0/1520011088 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9e780846d0 con 0x7f9e84151840 2023-10-30T13:49:38.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.438+0000 7f9e82ffd640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e700664a0 0x7f9e70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.438+0000 7f9e82ffd640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e700664a0 0x7f9e70068960 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9e740045f0 tx=0x7f9e74008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.442+0000 7f9e6b7fe640 1 -- 172.21.15.31:0/1520011088 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e841580b0 con 0x7f9e84151840 2023-10-30T13:49:38.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.634+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e841580b0 con 0x7f9e84151840 2023-10-30T13:49:38.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.634+0000 7f9e6b7fe640 1 -- 172.21.15.31:0/1520011088 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5698 (secure 0 0 0) 0x7f9e841580b0 con 0x7f9e84151840 2023-10-30T13:49:38.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:38.640 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:36.991547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:38.640 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 66 2023-10-30T13:49:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e700664a0 msgr2=0x7f9e70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e700664a0 0x7f9e70068960 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9e740045f0 tx=0x7f9e74008040 comp rx=0 tx=0).stop 2023-10-30T13:49:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e84151840 msgr2=0x7f9e841f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e84151840 0x7f9e841f2200 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800d800 tx=0x7f9e7800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 shutdown_connections 2023-10-30T13:49:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9e700664a0 0x7f9e70068960 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84151c20 0x7f9e841565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e84151840 0x7f9e841f2200 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 --2- 172.21.15.31:0/1520011088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9e840cb8b0 0x7f9e84145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 >> 172.21.15.31:0/1520011088 conn(0x7f9e840cac70 msgr2=0x7f9e84072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:38.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 shutdown_connections 2023-10-30T13:49:38.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.638+0000 7f9e8933e640 1 -- 172.21.15.31:0/1520011088 wait complete. 2023-10-30T13:49:38.661 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:49:38.661 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:49:38.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:38.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8b372640 1 Processor -- start 2023-10-30T13:49:38.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8b372640 1 -- start start 2023-10-30T13:49:38.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8b372640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8b372640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fae840c4a10 0x7fae840c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8b372640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae84159f20 con 0x7fae8415a3b0 2023-10-30T13:49:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8b372640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8415dec0 con 0x7fae840c4a10 2023-10-30T13:49:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8b372640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae841599a0 con 0x7fae84153c80 2023-10-30T13:49:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8890d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fae840c4a10 0x7fae840c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8890d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fae840c4a10 0x7fae840c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36274/0 (socket says 172.21.15.31:36274) 2023-10-30T13:49:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8890d640 1 -- 172.21.15.31:0/3181894848 learned_addr learned my addr 172.21.15.31:0/3181894848 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8990f640 1 --1- 172.21.15.31:0/3181894848 >> v1:172.21.15.31:6789/0 conn(0x7fae8415a3b0 0x7fae840ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58692/0 (socket says 172.21.15.31:58692) 2023-10-30T13:49:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8910e640 1 --2- 172.21.15.31:0/3181894848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8890d640 1 -- 172.21.15.31:0/3181894848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 msgr2=0x7fae8415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8890d640 1 --2- 172.21.15.31:0/3181894848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8890d640 1 -- 172.21.15.31:0/3181894848 >> v1:172.21.15.31:6789/0 conn(0x7fae8415a3b0 legacy=0x7fae840ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae8890d640 1 -- 172.21.15.31:0/3181894848 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae840c47e0 con 0x7fae840c4a10 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.802+0000 7fae7bfff640 1 -- 172.21.15.31:0/3181894848 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938310512 0 0) 0x7fae84159f20 con 0x7fae8415a3b0 2023-10-30T13:49:38.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8890d640 1 --2- 172.21.15.31:0/3181894848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae840c42a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fae74009870 tx=0x7fae7401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea81915716d74ad4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae7bfff640 1 -- 172.21.15.31:0/3181894848 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae740a8a30 con 0x7fae840c4a10 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae7bfff640 1 -- 172.21.15.31:0/3181894848 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae740a8b90 con 0x7fae840c4a10 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae7bfff640 1 -- 172.21.15.31:0/3181894848 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae740b0440 con 0x7fae840c4a10 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 -- 172.21.15.31:0/3181894848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 msgr2=0x7fae840c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 --2- 172.21.15.31:0/3181894848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae840c42a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fae74009870 tx=0x7fae7401ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:38.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 -- 172.21.15.31:0/3181894848 shutdown_connections 2023-10-30T13:49:38.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 --2- 172.21.15.31:0/3181894848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae840c42a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 --2- 172.21.15.31:0/3181894848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 -- 172.21.15.31:0/3181894848 >> 172.21.15.31:0/3181894848 conn(0x7fae840715c0 msgr2=0x7fae840719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 -- 172.21.15.31:0/3181894848 shutdown_connections 2023-10-30T13:49:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.806+0000 7fae8b372640 1 -- 172.21.15.31:0/3181894848 wait complete. 2023-10-30T13:49:38.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 Processor -- start 2023-10-30T13:49:38.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 -- start start 2023-10-30T13:49:38.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae8415ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae8415a3b0 0x7fae8414f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8415d1f0 con 0x7fae840c4a10 2023-10-30T13:49:38.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8415c300 con 0x7fae8415a3b0 2023-10-30T13:49:38.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8415c5e0 con 0x7fae84153c80 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae8415ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae8415ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36284/0 (socket says 172.21.15.31:36284) 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 -- 172.21.15.31:0/3750563905 learned_addr learned my addr 172.21.15.31:0/3750563905 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8890d640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8990f640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae8415a3b0 0x7fae8414f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 -- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 msgr2=0x7fae8415b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 -- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae8415a3b0 msgr2=0x7fae8414f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae8415a3b0 0x7fae8414f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 -- 172.21.15.31:0/3750563905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae740094d0 con 0x7fae840c4a10 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8910e640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae8415ad10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fae8000e930 tx=0x7fae8000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae79ffb640 1 -- 172.21.15.31:0/3750563905 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80009690 con 0x7fae840c4a10 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae79ffb640 1 -- 172.21.15.31:0/3750563905 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8000ca00 con 0x7fae840c4a10 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae79ffb640 1 -- 172.21.15.31:0/3750563905 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8000cd70 con 0x7fae840c4a10 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8415c8c0 con 0x7fae840c4a10 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.810+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8415ce80 con 0x7fae840c4a10 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.814+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8404e170 con 0x7fae840c4a10 2023-10-30T13:49:38.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.814+0000 7fae79ffb640 1 -- 172.21.15.31:0/3750563905 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae80013470 con 0x7fae840c4a10 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.814+0000 7fae79ffb640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae680664a0 0x7fae68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.814+0000 7fae79ffb640 1 -- 172.21.15.31:0/3750563905 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fae80084570 con 0x7fae840c4a10 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.814+0000 7fae8890d640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae680664a0 0x7fae68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.814+0000 7fae8890d640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae680664a0 0x7fae68068960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae740b2000 tx=0x7fae740aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:38.818+0000 7fae79ffb640 1 -- 172.21.15.31:0/3750563905 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae80051690 con 0x7fae840c4a10 2023-10-30T13:49:39.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.006+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae840cad40 con 0x7fae840c4a10 2023-10-30T13:49:39.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.006+0000 7fae79ffb640 1 -- 172.21.15.31:0/3750563905 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5698 (secure 0 0 0) 0x7fae80054cb0 con 0x7fae840c4a10 2023-10-30T13:49:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:36.991547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 66 2023-10-30T13:49:39.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae680664a0 msgr2=0x7fae68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae680664a0 0x7fae68068960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae740b2000 tx=0x7fae740aa070 comp rx=0 tx=0).stop 2023-10-30T13:49:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 msgr2=0x7fae8415ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae8415ad10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fae8000e930 tx=0x7fae8000edf0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 shutdown_connections 2023-10-30T13:49:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fae680664a0 0x7fae68068960 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae8415a3b0 0x7fae8414f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fae84153c80 0x7fae8415b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 --2- 172.21.15.31:0/3750563905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae840c4a10 0x7fae8415ad10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 >> 172.21.15.31:0/3750563905 conn(0x7fae840715c0 msgr2=0x7fae840719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:39.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 shutdown_connections 2023-10-30T13:49:39.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:39.010+0000 7fae8b372640 1 -- 172.21.15.31:0/3750563905 wait complete. 2023-10-30T13:49:39.033 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 66, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 66, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5889": { "addr": "172.21.15.31:6837/327248983", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 327248983, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 327248983, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5889, "incarnation": 66, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:49:36.991547+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5889 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:49:41.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc543b1d640 1 Processor -- start 2023-10-30T13:49:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc543b1d640 1 -- start start 2023-10-30T13:49:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc543b1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c1537f0 0x7fc53c0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc543b1d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0c2aa0 con 0x7fc53c06b2f0 2023-10-30T13:49:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc543b1d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0708e0 con 0x7fc53c15a2e0 2023-10-30T13:49:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc543b1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0c3ab0 con 0x7fc53c1537f0 2023-10-30T13:49:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc5420ba640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc53c06b2f0 0x7fc53c1641e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58706/0 (socket says 172.21.15.31:58706) 2023-10-30T13:49:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.182+0000 7fc5420ba640 1 -- 172.21.15.31:0/1720792592 learned_addr learned my addr 172.21.15.31:0/1720792592 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:41.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5418b9640 1 --2- 172.21.15.31:0/1720792592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c1537f0 0x7fc53c0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906700139 0 0) 0x7fc53c0c2aa0 con 0x7fc53c06b2f0 2023-10-30T13:49:41.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc528003530 con 0x7fc53c06b2f0 2023-10-30T13:49:41.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598501326 0 0) 0x7fc53c0708e0 con 0x7fc53c15a2e0 2023-10-30T13:49:41.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc53c0c2aa0 con 0x7fc53c15a2e0 2023-10-30T13:49:41.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3861844354 0 0) 0x7fc528003530 con 0x7fc53c06b2f0 2023-10-30T13:49:41.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5418b9640 1 -- 172.21.15.31:0/1720792592 >> v1:172.21.15.33:6789/0 conn(0x7fc53c15a2e0 legacy=0x7fc53c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5418b9640 1 -- 172.21.15.31:0/1720792592 >> v1:172.21.15.31:6789/0 conn(0x7fc53c06b2f0 legacy=0x7fc53c1641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5418b9640 1 -- 172.21.15.31:0/1720792592 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0c1eb0 con 0x7fc53c1537f0 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.186+0000 7fc5418b9640 1 --2- 172.21.15.31:0/1720792592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c1537f0 0x7fc53c0701a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc5380097f0 tx=0x7fc53801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec7edaae46960504 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc534001a00 con 0x7fc53c1537f0 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5380a8650 con 0x7fc53c1537f0 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc5408b7640 1 -- 172.21.15.31:0/1720792592 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5380a89c0 con 0x7fc53c1537f0 2023-10-30T13:49:41.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- 172.21.15.31:0/1720792592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c1537f0 msgr2=0x7fc53c0701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1720792592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c1537f0 0x7fc53c0701a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc5380097f0 tx=0x7fc53801ebd0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- 172.21.15.31:0/1720792592 shutdown_connections 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1720792592 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c1537f0 0x7fc53c0701a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- 172.21.15.31:0/1720792592 >> 172.21.15.31:0/1720792592 conn(0x7fc53c071fb0 msgr2=0x7fc53c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- 172.21.15.31:0/1720792592 shutdown_connections 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- 172.21.15.31:0/1720792592 wait complete. 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 Processor -- start 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- start start 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc53c06b2f0 0x7fc53c1ef260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc53c1537f0 0x7fc53c14ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c15a2e0 0x7fc53c0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1ef7a0 con 0x7fc53c1537f0 2023-10-30T13:49:41.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0c5500 con 0x7fc53c06b2f0 2023-10-30T13:49:41.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.190+0000 7fc543b1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0c57e0 con 0x7fc53c15a2e0 2023-10-30T13:49:41.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc53c1537f0 0x7fc53c14ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc53c1537f0 0x7fc53c14ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36300/0 (socket says 172.21.15.31:36300) 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 -- 172.21.15.31:0/1379050420 learned_addr learned my addr 172.21.15.31:0/1379050420 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5418b9640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc53c06b2f0 0x7fc53c1ef260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5420ba640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c15a2e0 0x7fc53c0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 -- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c15a2e0 msgr2=0x7fc53c0c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c15a2e0 0x7fc53c0c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 -- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc53c06b2f0 msgr2=0x7fc53c1ef260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc53c06b2f0 0x7fc53c1ef260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 -- 172.21.15.31:0/1379050420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc538009420 con 0x7fc53c1537f0 2023-10-30T13:49:41.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5420ba640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c15a2e0 0x7fc53c0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:41.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5410b8640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc53c1537f0 0x7fc53c14ccd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c430 tx=0x7fc52c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5327fc640 1 -- 172.21.15.31:0/1379050420 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0052d0 con 0x7fc53c1537f0 2023-10-30T13:49:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5327fc640 1 -- 172.21.15.31:0/1379050420 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c005450 con 0x7fc53c1537f0 2023-10-30T13:49:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5327fc640 1 -- 172.21.15.31:0/1379050420 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c003840 con 0x7fc53c1537f0 2023-10-30T13:49:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c0c5ac0 con 0x7fc53c1537f0 2023-10-30T13:49:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c0c6080 con 0x7fc53c1537f0 2023-10-30T13:49:41.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c04e170 con 0x7fc53c1537f0 2023-10-30T13:49:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5327fc640 1 -- 172.21.15.31:0/1379050420 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc52c0039a0 con 0x7fc53c1537f0 2023-10-30T13:49:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.194+0000 7fc5327fc640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5200664a0 0x7fc520068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.198+0000 7fc5327fc640 1 -- 172.21.15.31:0/1379050420 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc52c083740 con 0x7fc53c1537f0 2023-10-30T13:49:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.198+0000 7fc5418b9640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5200664a0 0x7fc520068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.198+0000 7fc5418b9640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5200664a0 0x7fc520068960 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc5380b2000 tx=0x7fc5380aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.198+0000 7fc5327fc640 1 -- 172.21.15.31:0/1379050420 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc52c0508f0 con 0x7fc53c1537f0 2023-10-30T13:49:41.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc53c0b3100 con 0x7fc53c1537f0 2023-10-30T13:49:41.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc5327fc640 1 -- 172.21.15.31:0/1379050420 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5698 (secure 0 0 0) 0x7fc52c053f10 con 0x7fc53c1537f0 2023-10-30T13:49:41.392 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:41.393 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:36.991547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:41.393 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 66 2023-10-30T13:49:41.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5200664a0 msgr2=0x7fc520068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5200664a0 0x7fc520068960 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc5380b2000 tx=0x7fc5380aa070 comp rx=0 tx=0).stop 2023-10-30T13:49:41.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc53c1537f0 msgr2=0x7fc53c14ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc53c1537f0 0x7fc53c14ccd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c430 tx=0x7fc52c00c8f0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 shutdown_connections 2023-10-30T13:49:41.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5200664a0 0x7fc520068960 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc53c15a2e0 0x7fc53c0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc53c1537f0 0x7fc53c14ccd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 --2- 172.21.15.31:0/1379050420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc53c06b2f0 0x7fc53c1ef260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 >> 172.21.15.31:0/1379050420 conn(0x7fc53c071fb0 msgr2=0x7fc53c151270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:41.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 shutdown_connections 2023-10-30T13:49:41.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:41.390+0000 7fc543b1d640 1 -- 172.21.15.31:0/1379050420 wait complete. 2023-10-30T13:49:41.413 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:43.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.558+0000 7f06f0965640 1 Processor -- start 2023-10-30T13:49:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.558+0000 7f06f0965640 1 -- start start 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.558+0000 7f06f0965640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f06ec0c4a10 0x7f06ec15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06f0965640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec159f20 con 0x7f06ec153c80 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06f0965640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec15d620 con 0x7f06ec0c42a0 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06f0965640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec15dec0 con 0x7f06ec0c4a10 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06ea59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f06ec153c80 0x7f06ec15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58722/0 (socket says 172.21.15.31:58722) 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06ea59c640 1 -- 172.21.15.31:0/946759658 learned_addr learned my addr 172.21.15.31:0/946759658 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06e9d9b640 1 --2- 172.21.15.31:0/946759658 >> v2:172.21.15.33:3300/0 conn(0x7f06ec0c4a10 0x7f06ec15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06e959a640 1 -- 172.21.15.31:0/946759658 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065044117 0 0) 0x7f06ec159f20 con 0x7f06ec153c80 2023-10-30T13:49:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06e959a640 1 -- 172.21.15.31:0/946759658 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d4003530 con 0x7f06ec153c80 2023-10-30T13:49:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06e9d9b640 1 -- 172.21.15.31:0/946759658 >> v1:172.21.15.33:6789/0 conn(0x7f06ec0c42a0 legacy=0x7f06ec0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06e9d9b640 1 -- 172.21.15.31:0/946759658 >> v1:172.21.15.31:6789/0 conn(0x7f06ec153c80 legacy=0x7f06ec15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06e959a640 1 -- 172.21.15.31:0/946759658 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081878920 0 0) 0x7f06d4003530 con 0x7f06ec153c80 2023-10-30T13:49:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.562+0000 7f06e9d9b640 1 -- 172.21.15.31:0/946759658 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec15d900 con 0x7f06ec0c4a10 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06e9d9b640 1 --2- 172.21.15.31:0/946759658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec15cf90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f06e00097c0 tx=0x7f06e001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b71f020dcd0fc67 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06e959a640 1 -- 172.21.15.31:0/946759658 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc001680 con 0x7f06ec0c4a10 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06e959a640 1 -- 172.21.15.31:0/946759658 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e00a8610 con 0x7f06ec0c4a10 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06e959a640 1 -- 172.21.15.31:0/946759658 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e00a89a0 con 0x7f06ec0c4a10 2023-10-30T13:49:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06f0965640 1 -- 172.21.15.31:0/946759658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 msgr2=0x7f06ec15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06f0965640 1 --2- 172.21.15.31:0/946759658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec15cf90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f06e00097c0 tx=0x7f06e001eb90 comp rx=0 tx=0).stop 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06f0965640 1 -- 172.21.15.31:0/946759658 shutdown_connections 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06f0965640 1 --2- 172.21.15.31:0/946759658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec15cf90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06f0965640 1 -- 172.21.15.31:0/946759658 >> 172.21.15.31:0/946759658 conn(0x7f06ec0715c0 msgr2=0x7f06ec0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06f0965640 1 -- 172.21.15.31:0/946759658 shutdown_connections 2023-10-30T13:49:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.566+0000 7f06f0965640 1 -- 172.21.15.31:0/946759658 wait complete. 2023-10-30T13:49:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 Processor -- start 2023-10-30T13:49:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 -- start start 2023-10-30T13:49:43.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f06ec0c42a0 0x7f06ec1ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec153c80 0x7f06ec149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec154a60 con 0x7f06ec153c80 2023-10-30T13:49:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0b3690 con 0x7f06ec0c4a10 2023-10-30T13:49:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0b3950 con 0x7f06ec0c42a0 2023-10-30T13:49:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec153c80 0x7f06ec149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec153c80 0x7f06ec149ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36312/0 (socket says 172.21.15.31:36312) 2023-10-30T13:49:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 -- 172.21.15.31:0/338300627 learned_addr learned my addr 172.21.15.31:0/338300627 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ea59c640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f06ec0c42a0 0x7f06ec1ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06e9d9b640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 -- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f06ec0c42a0 msgr2=0x7f06ec1ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f06ec0c42a0 0x7f06ec1ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 -- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 msgr2=0x7f06ec154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 -- 172.21.15.31:0/338300627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e0009420 con 0x7f06ec153c80 2023-10-30T13:49:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06e9d9b640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06ead9d640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec153c80 0x7f06ec149ec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c770 tx=0x7f06e400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06daffd640 1 -- 172.21.15.31:0/338300627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4003b80 con 0x7f06ec153c80 2023-10-30T13:49:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06daffd640 1 -- 172.21.15.31:0/338300627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e4003ce0 con 0x7f06ec153c80 2023-10-30T13:49:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec0b3c30 con 0x7f06ec153c80 2023-10-30T13:49:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06daffd640 1 -- 172.21.15.31:0/338300627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e400aa90 con 0x7f06ec153c80 2023-10-30T13:49:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.570+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec0b41f0 con 0x7f06ec153c80 2023-10-30T13:49:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.574+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec04e170 con 0x7f06ec153c80 2023-10-30T13:49:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.574+0000 7f06daffd640 1 -- 172.21.15.31:0/338300627 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06e400abf0 con 0x7f06ec153c80 2023-10-30T13:49:43.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.574+0000 7f06daffd640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f06cc0664a0 0x7f06cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.574+0000 7f06daffd640 1 -- 172.21.15.31:0/338300627 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f06e4084a40 con 0x7f06ec153c80 2023-10-30T13:49:43.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.574+0000 7f06ea59c640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f06cc0664a0 0x7f06cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.578+0000 7f06ea59c640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f06cc0664a0 0x7f06cc068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f06dc000f70 tx=0x7f06dc001d70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.578+0000 7f06daffd640 1 -- 172.21.15.31:0/338300627 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06e4051bb0 con 0x7f06ec153c80 2023-10-30T13:49:43.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.774+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06ec0cad40 con 0x7f06ec153c80 2023-10-30T13:49:43.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.774+0000 7f06daffd640 1 -- 172.21.15.31:0/338300627 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5698 (secure 0 0 0) 0x7f06e40551d0 con 0x7f06ec153c80 2023-10-30T13:49:43.779 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:43.779 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:36.991547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:43.780 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 66 2023-10-30T13:49:43.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f06cc0664a0 msgr2=0x7f06cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f06cc0664a0 0x7f06cc068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f06dc000f70 tx=0x7f06dc001d70 comp rx=0 tx=0).stop 2023-10-30T13:49:43.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec153c80 msgr2=0x7f06ec149ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec153c80 0x7f06ec149ec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c770 tx=0x7f06e400cc30 comp rx=0 tx=0).stop 2023-10-30T13:49:43.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 shutdown_connections 2023-10-30T13:49:43.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f06cc0664a0 0x7f06cc068960 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec153c80 0x7f06ec149ec0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06ec0c4a10 0x7f06ec154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 --2- 172.21.15.31:0/338300627 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f06ec0c42a0 0x7f06ec1ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 >> 172.21.15.31:0/338300627 conn(0x7f06ec0715c0 msgr2=0x7f06ec1524a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 shutdown_connections 2023-10-30T13:49:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:43.778+0000 7f06f0965640 1 -- 172.21.15.31:0/338300627 wait complete. 2023-10-30T13:49:43.801 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:45.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f6964b1d640 1 Processor -- start 2023-10-30T13:49:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f6964b1d640 1 -- start start 2023-10-30T13:49:45.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f6964b1d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6960153c80 0x7f696015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f6964b1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f69600ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f6964b1d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6960159f20 con 0x7f69600c4a10 2023-10-30T13:49:45.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f6964b1d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696015dec0 con 0x7f6960153c80 2023-10-30T13:49:45.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f6964b1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69601599a0 con 0x7f69600c42a0 2023-10-30T13:49:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695e59c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6960153c80 0x7f696015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695e59c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6960153c80 0x7f696015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57690/0 (socket says 172.21.15.31:57690) 2023-10-30T13:49:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695dd9b640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f69600c4a10 0x7f696015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46336/0 (socket says 172.21.15.31:46336) 2023-10-30T13:49:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695e59c640 1 -- 172.21.15.31:0/3572065772 learned_addr learned my addr 172.21.15.31:0/3572065772 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695ed9d640 1 --2- 172.21.15.31:0/3572065772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f69600ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695e59c640 1 -- 172.21.15.31:0/3572065772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 msgr2=0x7f69600ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695e59c640 1 --2- 172.21.15.31:0/3572065772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f69600ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695e59c640 1 -- 172.21.15.31:0/3572065772 >> v1:172.21.15.31:6789/0 conn(0x7f69600c4a10 legacy=0x7f696015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695e59c640 1 -- 172.21.15.31:0/3572065772 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696015d620 con 0x7f6960153c80 2023-10-30T13:49:45.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695ed9d640 1 --2- 172.21.15.31:0/3572065772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f69600ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:45.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.945+0000 7f695d59a640 1 -- 172.21.15.31:0/3572065772 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238508253 0 0) 0x7f6960159f20 con 0x7f69600c4a10 2023-10-30T13:49:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f695e59c640 1 --2- 172.21.15.31:0/3572065772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6960153c80 0x7f696015cf90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6950009870 tx=0x7f695001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeda9be468fac4b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f695d59a640 1 -- 172.21.15.31:0/3572065772 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69500a8a30 con 0x7f6960153c80 2023-10-30T13:49:45.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f695d59a640 1 -- 172.21.15.31:0/3572065772 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69500a8b90 con 0x7f6960153c80 2023-10-30T13:49:45.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f695d59a640 1 -- 172.21.15.31:0/3572065772 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69500b0440 con 0x7f6960153c80 2023-10-30T13:49:45.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 -- 172.21.15.31:0/3572065772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6960153c80 msgr2=0x7f696015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 --2- 172.21.15.31:0/3572065772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6960153c80 0x7f696015cf90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6950009870 tx=0x7f695001ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:45.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 -- 172.21.15.31:0/3572065772 shutdown_connections 2023-10-30T13:49:45.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 --2- 172.21.15.31:0/3572065772 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f69600ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 --2- 172.21.15.31:0/3572065772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6960153c80 0x7f696015cf90 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 -- 172.21.15.31:0/3572065772 >> 172.21.15.31:0/3572065772 conn(0x7f69600715c0 msgr2=0x7f69600719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:45.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 -- 172.21.15.31:0/3572065772 shutdown_connections 2023-10-30T13:49:45.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.949+0000 7f6964b1d640 1 -- 172.21.15.31:0/3572065772 wait complete. 2023-10-30T13:49:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 Processor -- start 2023-10-30T13:49:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 -- start start 2023-10-30T13:49:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f6960148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69600c4a10 0x7f696014f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960153c80 0x7f6960145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6960148490 con 0x7f69600c4a10 2023-10-30T13:49:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696015cc80 con 0x7f6960153c80 2023-10-30T13:49:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696015cf60 con 0x7f69600c42a0 2023-10-30T13:49:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69600c4a10 0x7f696014f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69600c4a10 0x7f696014f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57706/0 (socket says 172.21.15.31:57706) 2023-10-30T13:49:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 -- 172.21.15.31:0/1045064837 learned_addr learned my addr 172.21.15.31:0/1045064837 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695e59c640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f6960148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695ed9d640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960153c80 0x7f6960145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 -- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 msgr2=0x7f6960148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f6960148f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 -- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960153c80 msgr2=0x7f6960145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960153c80 0x7f6960145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 -- 172.21.15.31:0/1045064837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69500094d0 con 0x7f69600c4a10 2023-10-30T13:49:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f695dd9b640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69600c4a10 0x7f696014f780 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f695400c4b0 tx=0x7f695400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:45.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f694effd640 1 -- 172.21.15.31:0/1045064837 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69540074e0 con 0x7f69600c4a10 2023-10-30T13:49:45.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f694effd640 1 -- 172.21.15.31:0/1045064837 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6954007660 con 0x7f69600c4a10 2023-10-30T13:49:45.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6960148730 con 0x7f69600c4a10 2023-10-30T13:49:45.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f694effd640 1 -- 172.21.15.31:0/1045064837 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954004a30 con 0x7f69600c4a10 2023-10-30T13:49:45.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6960147260 con 0x7f69600c4a10 2023-10-30T13:49:45.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.953+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696004e170 con 0x7f69600c4a10 2023-10-30T13:49:45.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.957+0000 7f694effd640 1 -- 172.21.15.31:0/1045064837 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6954004b90 con 0x7f69600c4a10 2023-10-30T13:49:45.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.957+0000 7f694effd640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69400664a0 0x7f6940068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.957+0000 7f694effd640 1 -- 172.21.15.31:0/1045064837 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f69540836d0 con 0x7f69600c4a10 2023-10-30T13:49:45.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.957+0000 7f695e59c640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69400664a0 0x7f6940068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.957+0000 7f695e59c640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69400664a0 0x7f6940068960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f69500b2000 tx=0x7f69500aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:45.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:45.961+0000 7f694effd640 1 -- 172.21.15.31:0/1045064837 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69540508a0 con 0x7f69600c4a10 2023-10-30T13:49:46.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.157+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69600cad40 con 0x7f69600c4a10 2023-10-30T13:49:46.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.157+0000 7f694effd640 1 -- 172.21.15.31:0/1045064837 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f6954018090 con 0x7f69600c4a10 2023-10-30T13:49:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:46.162 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:45.622405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:46.163 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2023-10-30T13:49:46.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69400664a0 msgr2=0x7f6940068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:46.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69400664a0 0x7f6940068960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f69500b2000 tx=0x7f69500aa070 comp rx=0 tx=0).stop 2023-10-30T13:49:46.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69600c4a10 msgr2=0x7f696014f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:46.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69600c4a10 0x7f696014f780 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f695400c4b0 tx=0x7f695400c970 comp rx=0 tx=0).stop 2023-10-30T13:49:46.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 shutdown_connections 2023-10-30T13:49:46.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f69400664a0 0x7f6940068960 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960153c80 0x7f6960145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69600c4a10 0x7f696014f780 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 --2- 172.21.15.31:0/1045064837 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f69600c42a0 0x7f6960148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 >> 172.21.15.31:0/1045064837 conn(0x7f69600715c0 msgr2=0x7f69600cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:46.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 shutdown_connections 2023-10-30T13:49:46.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:46.161+0000 7f6964b1d640 1 -- 172.21.15.31:0/1045064837 wait complete. 2023-10-30T13:49:46.183 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:48.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:48.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.333+0000 7f9cf5609640 1 Processor -- start 2023-10-30T13:49:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.333+0000 7f9cf5609640 1 -- start start 2023-10-30T13:49:48.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cf5609640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 0x7f9cf00c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cf5609640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf00c2310 con 0x7f9cf0150830 2023-10-30T13:49:48.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cf5609640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf0156710 con 0x7f9cf00c3e40 2023-10-30T13:49:48.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cf5609640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf006bc10 con 0x7f9cf00cb7d0 2023-10-30T13:49:48.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9ceeffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f9cf0150830 0x7f9cf006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46342/0 (socket says 172.21.15.31:46342) 2023-10-30T13:49:48.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9ceeffd640 1 -- 172.21.15.31:0/2879622349 learned_addr learned my addr 172.21.15.31:0/2879622349 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:48.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9ce7fff640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 0x7f9cf00c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747514370 0 0) 0x7f9cf00c2310 con 0x7f9cf0150830 2023-10-30T13:49:48.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cdc003530 con 0x7f9cf0150830 2023-10-30T13:49:48.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389328829 0 0) 0x7f9cdc003530 con 0x7f9cf0150830 2023-10-30T13:49:48.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 msgr2=0x7f9cf00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:48.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 0x7f9cf00c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 >> v1:172.21.15.33:6789/0 conn(0x7f9cf00c3e40 legacy=0x7f9cf0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00c4360 con 0x7f9cf0150830 2023-10-30T13:49:48.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f9ce0001680 con 0x7f9cf0150830 2023-10-30T13:49:48.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 >> v1:172.21.15.31:6789/0 conn(0x7f9cf0150830 legacy=0x7f9cf006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cdc0039b0 0x7f9cdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 0x7f9cdc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf006bc10 con 0x7f9cdc004d10 2023-10-30T13:49:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc00dce0 con 0x7f9cdc0039b0 2023-10-30T13:49:48.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc00dea0 con 0x7f9cf00cb7d0 2023-10-30T13:49:48.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9ceeffd640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cdc0039b0 0x7f9cdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9ce7fff640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 0x7f9cdc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cef7fe640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cef7fe640 1 -- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 msgr2=0x7f9cdc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cef7fe640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cf00cb7d0 0x7f9cdc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cef7fe640 1 -- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cdc0039b0 msgr2=0x7f9cdc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cef7fe640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cdc0039b0 0x7f9cdc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.337+0000 7f9cef7fe640 1 -- 172.21.15.31:0/2879622349 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc003530 con 0x7f9cdc004d10 2023-10-30T13:49:48.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cef7fe640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00c4290 tx=0x7f9ce801c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29502f9d798c09b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:48.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8003b80 con 0x7f9cdc004d10 2023-10-30T13:49:48.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce8003ce0 con 0x7f9cdc004d10 2023-10-30T13:49:48.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cee7fc640 1 -- 172.21.15.31:0/2879622349 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8005310 con 0x7f9cdc004d10 2023-10-30T13:49:48.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 -- 172.21.15.31:0/2879622349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cdc004d10 msgr2=0x7f9cdc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 --2- 172.21.15.31:0/2879622349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00c4290 tx=0x7f9ce801c700 comp rx=0 tx=0).stop 2023-10-30T13:49:48.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cef7fe640 1 -- 172.21.15.31:0/2879622349 reap_dead start 2023-10-30T13:49:48.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 -- 172.21.15.31:0/2879622349 shutdown_connections 2023-10-30T13:49:48.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 -- 172.21.15.31:0/2879622349 >> 172.21.15.31:0/2879622349 conn(0x7f9cf00cac70 msgr2=0x7f9cf00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:48.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 -- 172.21.15.31:0/2879622349 shutdown_connections 2023-10-30T13:49:48.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 -- 172.21.15.31:0/2879622349 wait complete. 2023-10-30T13:49:48.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 Processor -- start 2023-10-30T13:49:48.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.341+0000 7f9cf5609640 1 -- start start 2023-10-30T13:49:48.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cdc004d10 0x7f9cf0141a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cf01e0510 0x7f9cf014e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cf014edf0 0x7f9cf01f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf00c8ed0 con 0x7f9cf01e0510 2023-10-30T13:49:48.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf01e0d60 con 0x7f9cf014edf0 2023-10-30T13:49:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf01f2190 con 0x7f9cdc004d10 2023-10-30T13:49:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cf01e0510 0x7f9cf014e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cf01e0510 0x7f9cf014e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57724/0 (socket says 172.21.15.31:57724) 2023-10-30T13:49:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 -- 172.21.15.31:0/1560100199 learned_addr learned my addr 172.21.15.31:0/1560100199 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ceeffd640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cdc004d10 0x7f9cf0141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cef7fe640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cf014edf0 0x7f9cf01f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 -- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cdc004d10 msgr2=0x7f9cf0141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cdc004d10 0x7f9cf0141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 -- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cf014edf0 msgr2=0x7f9cf01f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cf014edf0 0x7f9cf01f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 -- 172.21.15.31:0/1560100199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0141fd0 con 0x7f9cf01e0510 2023-10-30T13:49:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cef7fe640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cf014edf0 0x7f9cf01f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce7fff640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cf01e0510 0x7f9cf014e8b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80119d0 tx=0x7f9cd8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce77fe640 1 -- 172.21.15.31:0/1560100199 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd800faf0 con 0x7f9cf01e0510 2023-10-30T13:49:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce77fe640 1 -- 172.21.15.31:0/1560100199 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd800fc50 con 0x7f9cf01e0510 2023-10-30T13:49:48.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9ce77fe640 1 -- 172.21.15.31:0/1560100199 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8014440 con 0x7f9cf01e0510 2023-10-30T13:49:48.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf0156400 con 0x7f9cf01e0510 2023-10-30T13:49:48.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf01e1290 con 0x7f9cf01e0510 2023-10-30T13:49:48.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.345+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf014f1d0 con 0x7f9cf01e0510 2023-10-30T13:49:48.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.349+0000 7f9ce77fe640 1 -- 172.21.15.31:0/1560100199 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9cd80145a0 con 0x7f9cf01e0510 2023-10-30T13:49:48.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.349+0000 7f9ce77fe640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9cd00664a0 0x7f9cd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.349+0000 7f9ce77fe640 1 -- 172.21.15.31:0/1560100199 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9cd8084a30 con 0x7f9cf01e0510 2023-10-30T13:49:48.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.349+0000 7f9ceeffd640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9cd00664a0 0x7f9cd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.349+0000 7f9ceeffd640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9cd00664a0 0x7f9cd0068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0001060 tx=0x7f9ce0000f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:48.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.353+0000 7f9ce77fe640 1 -- 172.21.15.31:0/1560100199 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9cd8051be0 con 0x7f9cf01e0510 2023-10-30T13:49:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.541+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cf004e170 con 0x7f9cf01e0510 2023-10-30T13:49:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9ce77fe640 1 -- 172.21.15.31:0/1560100199 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f9cd8022020 con 0x7f9cf01e0510 2023-10-30T13:49:48.548 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:45.622405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:48.549 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2023-10-30T13:49:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9cd00664a0 msgr2=0x7f9cd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9cd00664a0 0x7f9cd0068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0001060 tx=0x7f9ce0000f90 comp rx=0 tx=0).stop 2023-10-30T13:49:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cf01e0510 msgr2=0x7f9cf014e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cf01e0510 0x7f9cf014e8b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80119d0 tx=0x7f9cd8011e90 comp rx=0 tx=0).stop 2023-10-30T13:49:48.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 shutdown_connections 2023-10-30T13:49:48.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9cd00664a0 0x7f9cd0068960 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9cf014edf0 0x7f9cf01f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cf01e0510 0x7f9cf014e8b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 --2- 172.21.15.31:0/1560100199 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9cdc004d10 0x7f9cf0141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 >> 172.21.15.31:0/1560100199 conn(0x7f9cf00cac70 msgr2=0x7f9cf006bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 shutdown_connections 2023-10-30T13:49:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:48.545+0000 7f9cf5609640 1 -- 172.21.15.31:0/1560100199 wait complete. 2023-10-30T13:49:48.569 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:50.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:50.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.721+0000 7fec13009640 1 Processor -- start 2023-10-30T13:49:50.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.721+0000 7fec13009640 1 -- start start 2023-10-30T13:49:50.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.721+0000 7fec13009640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fec0c150830 0x7fec0c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:50.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.721+0000 7fec13009640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fec0c070520 0x7fec0c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:50.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.721+0000 7fec13009640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0c0c2310 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.721+0000 7fec13009640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0c156710 con 0x7fec0c150830 2023-10-30T13:49:50.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.721+0000 7fec13009640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0c0c4360 con 0x7fec0c070520 2023-10-30T13:49:50.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fec0c150830 0x7fec0c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:50.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fec0c150830 0x7fec0c06b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57736/0 (socket says 172.21.15.31:57736) 2023-10-30T13:49:50.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 -- 172.21.15.31:0/570852844 learned_addr learned my addr 172.21.15.31:0/570852844 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec115a6640 1 --2- 172.21.15.31:0/570852844 >> v2:172.21.15.33:3300/0 conn(0x7fec0c070520 0x7fec0c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 -- 172.21.15.31:0/570852844 >> v2:172.21.15.33:3300/0 conn(0x7fec0c070520 msgr2=0x7fec0c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 --2- 172.21.15.31:0/570852844 >> v2:172.21.15.33:3300/0 conn(0x7fec0c070520 0x7fec0c070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 -- 172.21.15.31:0/570852844 >> v1:172.21.15.31:6789/0 conn(0x7fec0c0cb7d0 legacy=0x7fec0c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:50.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec0b7fe640 1 -- 172.21.15.31:0/570852844 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512396743 0 0) 0x7fec0c0c2310 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 -- 172.21.15.31:0/570852844 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c1569b0 con 0x7fec0c150830 2023-10-30T13:49:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec10da5640 1 --2- 172.21.15.31:0/570852844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c150830 0x7fec0c06b500 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fec0c066e10 tx=0x7fec0001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356cdbd725f74b3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec0b7fe640 1 -- 172.21.15.31:0/570852844 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000a89f0 con 0x7fec0c150830 2023-10-30T13:49:50.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec0b7fe640 1 -- 172.21.15.31:0/570852844 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec000a8b50 con 0x7fec0c150830 2023-10-30T13:49:50.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.725+0000 7fec0b7fe640 1 -- 172.21.15.31:0/570852844 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000b0440 con 0x7fec0c150830 2023-10-30T13:49:50.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- 172.21.15.31:0/570852844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c150830 msgr2=0x7fec0c06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:50.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 --2- 172.21.15.31:0/570852844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c150830 0x7fec0c06b500 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fec0c066e10 tx=0x7fec0001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- 172.21.15.31:0/570852844 shutdown_connections 2023-10-30T13:49:50.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 --2- 172.21.15.31:0/570852844 >> v2:172.21.15.33:3300/0 conn(0x7fec0c070520 0x7fec0c070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 --2- 172.21.15.31:0/570852844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c150830 0x7fec0c06b500 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- 172.21.15.31:0/570852844 >> 172.21.15.31:0/570852844 conn(0x7fec0c0cac70 msgr2=0x7fec0c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:50.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- 172.21.15.31:0/570852844 shutdown_connections 2023-10-30T13:49:50.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- 172.21.15.31:0/570852844 wait complete. 2023-10-30T13:49:50.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 Processor -- start 2023-10-30T13:49:50.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- start start 2023-10-30T13:49:50.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec0c070520 0x7fec0c1e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:50.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c0cb7d0 0x7fec0c0c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:50.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec0c150830 0x7fec0c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:50.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c1ee750 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c1eefb0 con 0x7fec0c150830 2023-10-30T13:49:50.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec13009640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c143130 con 0x7fec0c070520 2023-10-30T13:49:50.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c0cb7d0 0x7fec0c0c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:50.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c0cb7d0 0x7fec0c0c90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57750/0 (socket says 172.21.15.31:57750) 2023-10-30T13:49:50.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 -- 172.21.15.31:0/2874502284 learned_addr learned my addr 172.21.15.31:0/2874502284 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:50.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec115a6640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec0c150830 0x7fec0c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:50.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec10da5640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec0c070520 0x7fec0c1e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:50.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 -- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec0c070520 msgr2=0x7fec0c1e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:50.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec0c070520 0x7fec0c1e7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 -- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec0c150830 msgr2=0x7fec0c0c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:50.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec0c150830 0x7fec0c0c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec0bfff640 1 -- 172.21.15.31:0/2874502284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec000094d0 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec10da5640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec0c070520 0x7fec0c1e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:50.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.729+0000 7fec115a6640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec0c150830 0x7fec0c0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:50.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec0bfff640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c0cb7d0 0x7fec0c0c90e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7febf800c520 tx=0x7febf800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:50.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec097fa640 1 -- 172.21.15.31:0/2874502284 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80052d0 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec097fa640 1 -- 172.21.15.31:0/2874502284 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8005430 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec097fa640 1 -- 172.21.15.31:0/2874502284 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8012870 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c156060 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c1ee960 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0c14e1c0 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec097fa640 1 -- 172.21.15.31:0/2874502284 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febf8003620 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec097fa640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7febf00664a0 0x7febf0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:50.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec097fa640 1 -- 172.21.15.31:0/2874502284 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7febf8083790 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.733+0000 7fec10da5640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7febf00664a0 0x7febf0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:50.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.737+0000 7fec10da5640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7febf00664a0 0x7febf0068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fec000b2000 tx=0x7fec00008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:50.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.737+0000 7fec097fa640 1 -- 172.21.15.31:0/2874502284 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febf8050890 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec0c1ef120 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec097fa640 1 -- 172.21.15.31:0/2874502284 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5702 (secure 0 0 0) 0x7febf8053eb0 con 0x7fec0c0cb7d0 2023-10-30T13:49:50.937 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:50.937 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:45.622405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:50.937 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2023-10-30T13:49:50.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7febf00664a0 msgr2=0x7febf0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:50.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7febf00664a0 0x7febf0068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fec000b2000 tx=0x7fec00008040 comp rx=0 tx=0).stop 2023-10-30T13:49:50.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c0cb7d0 msgr2=0x7fec0c0c90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:50.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c0cb7d0 0x7fec0c0c90e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7febf800c520 tx=0x7febf800c9e0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 shutdown_connections 2023-10-30T13:49:50.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7febf00664a0 0x7febf0068960 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec0c150830 0x7fec0c0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0c0cb7d0 0x7fec0c0c90e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 --2- 172.21.15.31:0/2874502284 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec0c070520 0x7fec0c1e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:50.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 >> 172.21.15.31:0/2874502284 conn(0x7fec0c0cac70 msgr2=0x7fec0c14a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:50.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 shutdown_connections 2023-10-30T13:49:50.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:50.933+0000 7fec13009640 1 -- 172.21.15.31:0/2874502284 wait complete. 2023-10-30T13:49:50.959 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 67, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5889": { "addr": "172.21.15.31:6837/327248983", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 327248983, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 327248983, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5889, "incarnation": 66, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:49:45.622405+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5889 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:49:52.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:53.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 Processor -- start 2023-10-30T13:49:53.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 -- start start 2023-10-30T13:49:53.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e9815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:53.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2e980c4a10 0x7f2e980c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:53.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2e9815a3b0 0x7f2e980ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:53.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9815dec0 con 0x7f2e980c4a10 2023-10-30T13:49:53.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e981599a0 con 0x7f2e9815a3b0 2023-10-30T13:49:53.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9c980640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e98159f20 con 0x7f2e98153c80 2023-10-30T13:49:53.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2e980c4a10 0x7f2e980c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:53.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2e980c4a10 0x7f2e980c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57762/0 (socket says 172.21.15.31:57762) 2023-10-30T13:49:53.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 -- 172.21.15.31:0/1907047929 learned_addr learned my addr 172.21.15.31:0/1907047929 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:53.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e9659c640 1 --2- 172.21.15.31:0/1907047929 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e9815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:53.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e96d9d640 1 --2- 172.21.15.31:0/1907047929 >> v2:172.21.15.33:3300/0 conn(0x7f2e9815a3b0 0x7f2e980ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:53.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 -- 172.21.15.31:0/1907047929 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 msgr2=0x7f2e9815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:53.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 --2- 172.21.15.31:0/1907047929 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e9815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 -- 172.21.15.31:0/1907047929 >> v2:172.21.15.33:3300/0 conn(0x7f2e9815a3b0 msgr2=0x7f2e980ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:53.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 --2- 172.21.15.31:0/1907047929 >> v2:172.21.15.33:3300/0 conn(0x7f2e9815a3b0 0x7f2e980ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.117+0000 7f2e95d9b640 1 -- 172.21.15.31:0/1907047929 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980c47e0 con 0x7f2e980c4a10 2023-10-30T13:49:53.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e95d9b640 1 --2- 172.21.15.31:0/1907047929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e980c4a10 0x7f2e980c42a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2e980cbbc0 tx=0x7f2e8c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c769efcdda58d6f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:53.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9559a640 1 -- 172.21.15.31:0/1907047929 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c0a89b0 con 0x7f2e980c4a10 2023-10-30T13:49:53.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9559a640 1 -- 172.21.15.31:0/1907047929 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c0a8b10 con 0x7f2e980c4a10 2023-10-30T13:49:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9559a640 1 -- 172.21.15.31:0/1907047929 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c0a8ea0 con 0x7f2e980c4a10 2023-10-30T13:49:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 -- 172.21.15.31:0/1907047929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e980c4a10 msgr2=0x7f2e980c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1907047929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e980c4a10 0x7f2e980c42a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2e980cbbc0 tx=0x7f2e8c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 -- 172.21.15.31:0/1907047929 shutdown_connections 2023-10-30T13:49:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1907047929 >> v2:172.21.15.33:3300/0 conn(0x7f2e9815a3b0 0x7f2e980ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1907047929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e980c4a10 0x7f2e980c42a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1907047929 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e9815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 -- 172.21.15.31:0/1907047929 >> 172.21.15.31:0/1907047929 conn(0x7f2e980715c0 msgr2=0x7f2e980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 -- 172.21.15.31:0/1907047929 shutdown_connections 2023-10-30T13:49:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.121+0000 7f2e9c980640 1 -- 172.21.15.31:0/1907047929 wait complete. 2023-10-30T13:49:53.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 Processor -- start 2023-10-30T13:49:53.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 -- start start 2023-10-30T13:49:53.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e980c4a10 0x7f2e9806c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:53.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e980b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e9815a3b0 0x7f2e980b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b2020 con 0x7f2e9815a3b0 2023-10-30T13:49:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e981f2f30 con 0x7f2e980c4a10 2023-10-30T13:49:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9815cd00 con 0x7f2e98153c80 2023-10-30T13:49:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e9815a3b0 0x7f2e980b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:53.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e9815a3b0 0x7f2e980b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57774/0 (socket says 172.21.15.31:57774) 2023-10-30T13:49:53.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 -- 172.21.15.31:0/1072315933 learned_addr learned my addr 172.21.15.31:0/1072315933 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:53.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e95d9b640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e980b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:53.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9659c640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e980c4a10 0x7f2e9806c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 -- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 msgr2=0x7f2e980b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e980b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 -- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e980c4a10 msgr2=0x7f2e9806c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e980c4a10 0x7f2e9806c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 -- 172.21.15.31:0/1072315933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c0094d0 con 0x7f2e9815a3b0 2023-10-30T13:49:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e96d9d640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e9815a3b0 0x7f2e980b2fb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2e90007e60 tx=0x7f2e9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e86ffd640 1 -- 172.21.15.31:0/1072315933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e90014420 con 0x7f2e9815a3b0 2023-10-30T13:49:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e86ffd640 1 -- 172.21.15.31:0/1072315933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e900145a0 con 0x7f2e9815a3b0 2023-10-30T13:49:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e86ffd640 1 -- 172.21.15.31:0/1072315933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e90014950 con 0x7f2e9815a3b0 2023-10-30T13:49:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9815cfe0 con 0x7f2e9815a3b0 2023-10-30T13:49:53.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.125+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e980b2550 con 0x7f2e9815a3b0 2023-10-30T13:49:53.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.129+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980b2740 con 0x7f2e9815a3b0 2023-10-30T13:49:53.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.129+0000 7f2e86ffd640 1 -- 172.21.15.31:0/1072315933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e90014ab0 con 0x7f2e9815a3b0 2023-10-30T13:49:53.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.129+0000 7f2e86ffd640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e780664a0 0x7f2e78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:53.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.129+0000 7f2e86ffd640 1 -- 172.21.15.31:0/1072315933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2e90084970 con 0x7f2e9815a3b0 2023-10-30T13:49:53.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.129+0000 7f2e9659c640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e780664a0 0x7f2e78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:53.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.129+0000 7f2e9659c640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e780664a0 0x7f2e78068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e880045f0 tx=0x7f2e88008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:53.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.133+0000 7f2e86ffd640 1 -- 172.21.15.31:0/1072315933 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e90051ba0 con 0x7f2e9815a3b0 2023-10-30T13:49:53.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.321+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e9804e170 con 0x7f2e9815a3b0 2023-10-30T13:49:53.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e86ffd640 1 -- 172.21.15.31:0/1072315933 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f2e900551c0 con 0x7f2e9815a3b0 2023-10-30T13:49:53.328 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:53.328 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:51.067462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:53.328 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 68 2023-10-30T13:49:53.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e780664a0 msgr2=0x7f2e78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:53.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e780664a0 0x7f2e78068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e880045f0 tx=0x7f2e88008040 comp rx=0 tx=0).stop 2023-10-30T13:49:53.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e9815a3b0 msgr2=0x7f2e980b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:53.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e9815a3b0 0x7f2e980b2fb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2e90007e60 tx=0x7f2e9000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 shutdown_connections 2023-10-30T13:49:53.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e780664a0 0x7f2e78068960 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e9815a3b0 0x7f2e980b2fb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e98153c80 0x7f2e980b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 --2- 172.21.15.31:0/1072315933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e980c4a10 0x7f2e9806c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:53.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 >> 172.21.15.31:0/1072315933 conn(0x7f2e980715c0 msgr2=0x7f2e98158550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:53.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 shutdown_connections 2023-10-30T13:49:53.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:53.325+0000 7f2e9c980640 1 -- 172.21.15.31:0/1072315933 wait complete. 2023-10-30T13:49:53.349 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:55.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:55.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9e88640 1 Processor -- start 2023-10-30T13:49:55.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9e88640 1 -- start start 2023-10-30T13:49:55.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9e88640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f83f40c8eb0 0x7f83f415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:55.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9e88640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f83f406aeb0 0x7f83f406b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:55.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9e88640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f415dcc0 con 0x7f83f415ac70 2023-10-30T13:49:55.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9e88640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f4159800 con 0x7f83f406aeb0 2023-10-30T13:49:55.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9e88640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f4159d80 con 0x7f83f40c8eb0 2023-10-30T13:49:55.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f3fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f83f415ac70 0x7f83f415a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:47056/0 (socket says 172.21.15.31:47056) 2023-10-30T13:49:55.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f3fff640 1 -- 172.21.15.31:0/2071703329 learned_addr learned my addr 172.21.15.31:0/2071703329 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:49:55.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9687640 1 --2- 172.21.15.31:0/2071703329 >> v2:172.21.15.31:3300/0 conn(0x7f83f406aeb0 0x7f83f406b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:55.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f8e86640 1 --2- 172.21.15.31:0/2071703329 >> v2:172.21.15.33:3300/0 conn(0x7f83f40c8eb0 0x7f83f415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:55.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f37fe640 1 -- 172.21.15.31:0/2071703329 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981964172 0 0) 0x7f83e0001120 con 0x7f83f415ac70 2023-10-30T13:49:55.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f37fe640 1 -- 172.21.15.31:0/2071703329 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e4003530 con 0x7f83f415ac70 2023-10-30T13:49:55.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9687640 1 -- 172.21.15.31:0/2071703329 >> v2:172.21.15.33:3300/0 conn(0x7f83f40c8eb0 msgr2=0x7f83f415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:55.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9687640 1 --2- 172.21.15.31:0/2071703329 >> v2:172.21.15.33:3300/0 conn(0x7f83f40c8eb0 0x7f83f415cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9687640 1 -- 172.21.15.31:0/2071703329 >> v1:172.21.15.31:6789/0 conn(0x7f83f415ac70 legacy=0x7f83f415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:55.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.501+0000 7f83f9687640 1 -- 172.21.15.31:0/2071703329 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f415d420 con 0x7f83f406aeb0 2023-10-30T13:49:55.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9687640 1 --2- 172.21.15.31:0/2071703329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f406aeb0 0x7f83f406b300 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f83f406aa40 tx=0x7f83ec01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb099bb93bb28a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:55.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f37fe640 1 -- 172.21.15.31:0/2071703329 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0b0440 con 0x7f83f406aeb0 2023-10-30T13:49:55.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f37fe640 1 -- 172.21.15.31:0/2071703329 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ec0b05a0 con 0x7f83f406aeb0 2023-10-30T13:49:55.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f37fe640 1 -- 172.21.15.31:0/2071703329 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0b0930 con 0x7f83f406aeb0 2023-10-30T13:49:55.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 -- 172.21.15.31:0/2071703329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f406aeb0 msgr2=0x7f83f406b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:55.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 --2- 172.21.15.31:0/2071703329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f406aeb0 0x7f83f406b300 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f83f406aa40 tx=0x7f83ec01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 -- 172.21.15.31:0/2071703329 shutdown_connections 2023-10-30T13:49:55.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 --2- 172.21.15.31:0/2071703329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f406aeb0 0x7f83f406b300 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 --2- 172.21.15.31:0/2071703329 >> v2:172.21.15.33:3300/0 conn(0x7f83f40c8eb0 0x7f83f415cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 -- 172.21.15.31:0/2071703329 >> 172.21.15.31:0/2071703329 conn(0x7f83f40cef00 msgr2=0x7f83f40cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:55.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 -- 172.21.15.31:0/2071703329 shutdown_connections 2023-10-30T13:49:55.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.505+0000 7f83f9e88640 1 -- 172.21.15.31:0/2071703329 wait complete. 2023-10-30T13:49:55.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 Processor -- start 2023-10-30T13:49:55.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 -- start start 2023-10-30T13:49:55.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83f406aeb0 0x7f83f40c3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:55.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f40c8eb0 0x7f83f40c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:55.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f415ac70 0x7f83f41ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:55.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f41524b0 con 0x7f83f415ac70 2023-10-30T13:49:55.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f4153030 con 0x7f83f40c8eb0 2023-10-30T13:49:55.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f40c7980 con 0x7f83f406aeb0 2023-10-30T13:49:55.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f415ac70 0x7f83f41ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:55.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f415ac70 0x7f83f41ee840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41338/0 (socket says 172.21.15.31:41338) 2023-10-30T13:49:55.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 -- 172.21.15.31:0/241228279 learned_addr learned my addr 172.21.15.31:0/241228279 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:55.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f8e86640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83f406aeb0 0x7f83f40c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:55.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f3fff640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f40c8eb0 0x7f83f40c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:55.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 -- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83f406aeb0 msgr2=0x7f83f40c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:55.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83f406aeb0 0x7f83f40c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 -- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f40c8eb0 msgr2=0x7f83f40c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:55.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f40c8eb0 0x7f83f40c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 -- 172.21.15.31:0/241228279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec009a70 con 0x7f83f415ac70 2023-10-30T13:49:55.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9687640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f415ac70 0x7f83f41ee840 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0b3040 tx=0x7f83ec01be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:55.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f17fa640 1 -- 172.21.15.31:0/241228279 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0b0440 con 0x7f83f415ac70 2023-10-30T13:49:55.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f17fa640 1 -- 172.21.15.31:0/241228279 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ec0bc650 con 0x7f83f415ac70 2023-10-30T13:49:55.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f4152700 con 0x7f83f415ac70 2023-10-30T13:49:55.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f17fa640 1 -- 172.21.15.31:0/241228279 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0bca00 con 0x7f83f415ac70 2023-10-30T13:49:55.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f4152cc0 con 0x7f83f415ac70 2023-10-30T13:49:55.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.509+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f404e170 con 0x7f83f415ac70 2023-10-30T13:49:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.513+0000 7f83f17fa640 1 -- 172.21.15.31:0/241228279 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83ec0bcb60 con 0x7f83f415ac70 2023-10-30T13:49:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.513+0000 7f83f17fa640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83d80664f0 0x7f83d80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.513+0000 7f83f17fa640 1 -- 172.21.15.31:0/241228279 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f83ec122cf0 con 0x7f83f415ac70 2023-10-30T13:49:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.513+0000 7f83f8e86640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83d80664f0 0x7f83d80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.513+0000 7f83f8e86640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83d80664f0 0x7f83d80689b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009af0 tx=0x7f83e8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:55.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.517+0000 7f83f17fa640 1 -- 172.21.15.31:0/241228279 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83ec0f0010 con 0x7f83f415ac70 2023-10-30T13:49:55.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.705+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83f4153170 con 0x7f83f415ac70 2023-10-30T13:49:55.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f17fa640 1 -- 172.21.15.31:0/241228279 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f83ec009450 con 0x7f83f415ac70 2023-10-30T13:49:55.712 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:55.712 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:51.067462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:55.712 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 68 2023-10-30T13:49:55.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83d80664f0 msgr2=0x7f83d80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:55.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83d80664f0 0x7f83d80689b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009af0 tx=0x7f83e8008040 comp rx=0 tx=0).stop 2023-10-30T13:49:55.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f415ac70 msgr2=0x7f83f41ee840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:55.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f415ac70 0x7f83f41ee840 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0b3040 tx=0x7f83ec01be90 comp rx=0 tx=0).stop 2023-10-30T13:49:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 shutdown_connections 2023-10-30T13:49:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f83d80664f0 0x7f83d80689b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f415ac70 0x7f83f41ee840 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f40c8eb0 0x7f83f40c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 --2- 172.21.15.31:0/241228279 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f83f406aeb0 0x7f83f40c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:55.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 >> 172.21.15.31:0/241228279 conn(0x7f83f40cef00 msgr2=0x7f83f406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:55.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 shutdown_connections 2023-10-30T13:49:55.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:55.709+0000 7f83f9e88640 1 -- 172.21.15.31:0/241228279 wait complete. 2023-10-30T13:49:55.733 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:57.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:57.181 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:57.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:57.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.885+0000 7f447b66b640 1 Processor -- start 2023-10-30T13:49:57.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.885+0000 7f447b66b640 1 -- start start 2023-10-30T13:49:57.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.885+0000 7f447b66b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4474153c80 0x7f447415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:57.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f447b66b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4a10 0x7f44740c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:57.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f447b66b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4474159f20 con 0x7f447415a3b0 2023-10-30T13:49:57.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f447b66b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447415dec0 con 0x7f4474153c80 2023-10-30T13:49:57.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f447b66b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44741599a0 con 0x7f44740c4a10 2023-10-30T13:49:57.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4474153c80 0x7f447415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:57.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4474153c80 0x7f447415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41348/0 (socket says 172.21.15.31:41348) 2023-10-30T13:49:57.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 -- 172.21.15.31:0/988477940 learned_addr learned my addr 172.21.15.31:0/988477940 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:57.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4478c06640 1 --2- 172.21.15.31:0/988477940 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4a10 0x7f44740c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:57.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 -- 172.21.15.31:0/988477940 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4a10 msgr2=0x7f44740c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:57.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 --2- 172.21.15.31:0/988477940 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4a10 0x7f44740c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:57.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 -- 172.21.15.31:0/988477940 >> v1:172.21.15.33:6789/0 conn(0x7f447415a3b0 legacy=0x7f44740ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:57.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 -- 172.21.15.31:0/988477940 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44740c47e0 con 0x7f4474153c80 2023-10-30T13:49:57.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f4479407640 1 --2- 172.21.15.31:0/988477940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4474153c80 0x7f447415cf90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4470009870 tx=0x7f447001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2781fa4b6ea7a826 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f446bfff640 1 -- 172.21.15.31:0/988477940 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44700a8a30 con 0x7f4474153c80 2023-10-30T13:49:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f446bfff640 1 -- 172.21.15.31:0/988477940 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44700a8b90 con 0x7f4474153c80 2023-10-30T13:49:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.889+0000 7f446bfff640 1 -- 172.21.15.31:0/988477940 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44700b0440 con 0x7f4474153c80 2023-10-30T13:49:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- 172.21.15.31:0/988477940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4474153c80 msgr2=0x7f447415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:57.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 --2- 172.21.15.31:0/988477940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4474153c80 0x7f447415cf90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4470009870 tx=0x7f447001ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:57.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- 172.21.15.31:0/988477940 shutdown_connections 2023-10-30T13:49:57.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 --2- 172.21.15.31:0/988477940 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4a10 0x7f44740c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:57.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 --2- 172.21.15.31:0/988477940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4474153c80 0x7f447415cf90 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:57.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- 172.21.15.31:0/988477940 >> 172.21.15.31:0/988477940 conn(0x7f44740715c0 msgr2=0x7f44740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:57.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- 172.21.15.31:0/988477940 shutdown_connections 2023-10-30T13:49:57.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- 172.21.15.31:0/988477940 wait complete. 2023-10-30T13:49:57.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 Processor -- start 2023-10-30T13:49:57.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- start start 2023-10-30T13:49:57.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4a10 0x7f44740cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:57.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4474153c80 0x7f44740cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:57.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f447415a3b0 0x7f44740cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:57.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4474155390 con 0x7f447415a3b0 2023-10-30T13:49:57.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740cea70 con 0x7f44740c4a10 2023-10-30T13:49:57.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f447b66b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44741ef910 con 0x7f4474153c80 2023-10-30T13:49:57.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f4479c08640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f447415a3b0 0x7f44740cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:57.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f4479407640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4a10 0x7f44740cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:57.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f4479c08640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f447415a3b0 0x7f44740cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41362/0 (socket says 172.21.15.31:41362) 2023-10-30T13:49:57.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f4479c08640 1 -- 172.21.15.31:0/3871377850 learned_addr learned my addr 172.21.15.31:0/3871377850 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:49:57.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.893+0000 7f4478c06640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4474153c80 0x7f44740cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:57.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4479c08640 1 -- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4474153c80 msgr2=0x7f44740cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:57.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4479c08640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4474153c80 0x7f44740cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:57.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4479c08640 1 -- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4a10 msgr2=0x7f44740cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:57.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4479c08640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4a10 0x7f44740cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:57.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4479c08640 1 -- 172.21.15.31:0/3871377850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44700094d0 con 0x7f447415a3b0 2023-10-30T13:49:57.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4479c08640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f447415a3b0 0x7f44740cf740 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f446c00c760 tx=0x7f446c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:57.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4469ffb640 1 -- 172.21.15.31:0/3871377850 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c003b80 con 0x7f447415a3b0 2023-10-30T13:49:57.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4469ffb640 1 -- 172.21.15.31:0/3871377850 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c003ce0 con 0x7f447415a3b0 2023-10-30T13:49:57.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44740cd250 con 0x7f447415a3b0 2023-10-30T13:49:57.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4469ffb640 1 -- 172.21.15.31:0/3871377850 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c00aa30 con 0x7f447415a3b0 2023-10-30T13:49:57.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44740cece0 con 0x7f447415a3b0 2023-10-30T13:49:57.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44740cef00 con 0x7f447415a3b0 2023-10-30T13:49:57.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4469ffb640 1 -- 172.21.15.31:0/3871377850 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f446c00ab90 con 0x7f447415a3b0 2023-10-30T13:49:57.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4469ffb640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44580664d0 0x7f4458068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:57.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.897+0000 7f4469ffb640 1 -- 172.21.15.31:0/3871377850 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f446c083e40 con 0x7f447415a3b0 2023-10-30T13:49:57.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.901+0000 7f4479407640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44580664d0 0x7f4458068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:57.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.901+0000 7f4479407640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44580664d0 0x7f4458068990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f44700b2000 tx=0x7f4470009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:57.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:57.901+0000 7f4469ffb640 1 -- 172.21.15.31:0/3871377850 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f446c050fe0 con 0x7f447415a3b0 2023-10-30T13:49:58.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.097+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f447404e170 con 0x7f447415a3b0 2023-10-30T13:49:58.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.097+0000 7f4469ffb640 1 -- 172.21.15.31:0/3871377850 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f446c054600 con 0x7f447415a3b0 2023-10-30T13:49:58.103 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:49:58.104 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:56.226057+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:58.104 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 69 2023-10-30T13:49:58.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44580664d0 msgr2=0x7f4458068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:58.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44580664d0 0x7f4458068990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f44700b2000 tx=0x7f4470009040 comp rx=0 tx=0).stop 2023-10-30T13:49:58.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f447415a3b0 msgr2=0x7f44740cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:58.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f447415a3b0 0x7f44740cf740 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f446c00c760 tx=0x7f446c00cc20 comp rx=0 tx=0).stop 2023-10-30T13:49:58.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 shutdown_connections 2023-10-30T13:49:58.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44580664d0 0x7f4458068990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f447415a3b0 0x7f44740cf740 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4474153c80 0x7f44740cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 --2- 172.21.15.31:0/3871377850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4a10 0x7f44740cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 >> 172.21.15.31:0/3871377850 conn(0x7f44740715c0 msgr2=0x7f44740cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 shutdown_connections 2023-10-30T13:49:58.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:49:58.101+0000 7f447b66b640 1 -- 172.21.15.31:0/3871377850 wait complete. 2023-10-30T13:49:58.125 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:00.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:00.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe3a30f640 1 Processor -- start 2023-10-30T13:50:00.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe3a30f640 1 -- start start 2023-10-30T13:50:00.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe3a30f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbe34151840 0x7fbe340c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:00.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe3a30f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fbe34151c20 0x7fbe34072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:00.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe3a30f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe340c5030 con 0x7fbe340cb8b0 2023-10-30T13:50:00.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe3a30f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe34157830 con 0x7fbe34151840 2023-10-30T13:50:00.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe3a30f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe34157b10 con 0x7fbe34151c20 2023-10-30T13:50:00.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe33fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbe34151840 0x7fbe340c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:00.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe33fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbe34151840 0x7fbe340c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41378/0 (socket says 172.21.15.31:41378) 2023-10-30T13:50:00.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe33fff640 1 -- 172.21.15.31:0/1547877981 learned_addr learned my addr 172.21.15.31:0/1547877981 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:00.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.273+0000 7fbe337fe640 1 --2- 172.21.15.31:0/1547877981 >> v2:172.21.15.33:3300/0 conn(0x7fbe34151c20 0x7fbe34072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:00.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe33fff640 1 -- 172.21.15.31:0/1547877981 >> v2:172.21.15.33:3300/0 conn(0x7fbe34151c20 msgr2=0x7fbe34072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:00.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe33fff640 1 --2- 172.21.15.31:0/1547877981 >> v2:172.21.15.33:3300/0 conn(0x7fbe34151c20 0x7fbe34072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe33fff640 1 -- 172.21.15.31:0/1547877981 >> v1:172.21.15.33:6789/0 conn(0x7fbe340cb8b0 legacy=0x7fbe3406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:00.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe33fff640 1 -- 172.21.15.31:0/1547877981 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe34157df0 con 0x7fbe34151840 2023-10-30T13:50:00.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe33fff640 1 --2- 172.21.15.31:0/1547877981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe340c49a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fbe3406d4d0 tx=0x7fbe2401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6e97480563fdca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:00.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe32ffd640 1 -- 172.21.15.31:0/1547877981 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe240a89b0 con 0x7fbe34151840 2023-10-30T13:50:00.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe32ffd640 1 -- 172.21.15.31:0/1547877981 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe240a8b10 con 0x7fbe34151840 2023-10-30T13:50:00.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe32ffd640 1 -- 172.21.15.31:0/1547877981 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe240a8ea0 con 0x7fbe34151840 2023-10-30T13:50:00.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 -- 172.21.15.31:0/1547877981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 msgr2=0x7fbe340c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:00.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/1547877981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe340c49a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fbe3406d4d0 tx=0x7fbe2401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 -- 172.21.15.31:0/1547877981 shutdown_connections 2023-10-30T13:50:00.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/1547877981 >> v2:172.21.15.33:3300/0 conn(0x7fbe34151c20 0x7fbe34072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/1547877981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe340c49a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 -- 172.21.15.31:0/1547877981 >> 172.21.15.31:0/1547877981 conn(0x7fbe340cac70 msgr2=0x7fbe340cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:00.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 -- 172.21.15.31:0/1547877981 shutdown_connections 2023-10-30T13:50:00.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.277+0000 7fbe3a30f640 1 -- 172.21.15.31:0/1547877981 wait complete. 2023-10-30T13:50:00.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 Processor -- start 2023-10-30T13:50:00.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 -- start start 2023-10-30T13:50:00.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbe340cb8b0 0x7fbe341eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:00.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe341f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:00.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe34151c20 0x7fbe341e4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:00.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe340caa20 con 0x7fbe34151840 2023-10-30T13:50:00.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe341e00a0 con 0x7fbe34151c20 2023-10-30T13:50:00.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe341f29d0 con 0x7fbe340cb8b0 2023-10-30T13:50:00.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe341f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:00.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe341f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41394/0 (socket says 172.21.15.31:41394) 2023-10-30T13:50:00.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 -- 172.21.15.31:0/3809799006 learned_addr learned my addr 172.21.15.31:0/3809799006 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:00.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe388ac640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe34151c20 0x7fbe341e4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:00.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe33fff640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbe340cb8b0 0x7fbe341eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:00.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 -- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbe340cb8b0 msgr2=0x7fbe341eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:00.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbe340cb8b0 0x7fbe341eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 -- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe34151c20 msgr2=0x7fbe341e4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:00.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe34151c20 0x7fbe341e4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 -- 172.21.15.31:0/3809799006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe240094d0 con 0x7fbe34151840 2023-10-30T13:50:00.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe388ac640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe34151c20 0x7fbe341e4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:00.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe337fe640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe341f2e60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800d620 tx=0x7fbe2800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:00.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe30ff9640 1 -- 172.21.15.31:0/3809799006 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800b620 con 0x7fbe34151840 2023-10-30T13:50:00.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe30ff9640 1 -- 172.21.15.31:0/3809799006 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2800b780 con 0x7fbe34151840 2023-10-30T13:50:00.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe30ff9640 1 -- 172.21.15.31:0/3809799006 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800bb10 con 0x7fbe34151840 2023-10-30T13:50:00.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.281+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe341f23c0 con 0x7fbe34151840 2023-10-30T13:50:00.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.285+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe341e05b0 con 0x7fbe34151840 2023-10-30T13:50:00.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.285+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe341f2b60 con 0x7fbe34151840 2023-10-30T13:50:00.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.285+0000 7fbe30ff9640 1 -- 172.21.15.31:0/3809799006 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe2800bc70 con 0x7fbe34151840 2023-10-30T13:50:00.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.285+0000 7fbe30ff9640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbe200664d0 0x7fbe20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:00.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.285+0000 7fbe30ff9640 1 -- 172.21.15.31:0/3809799006 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbe28084650 con 0x7fbe34151840 2023-10-30T13:50:00.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.285+0000 7fbe33fff640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbe200664d0 0x7fbe20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:00.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.289+0000 7fbe33fff640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbe200664d0 0x7fbe20068990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbe240b2000 tx=0x7fbe24009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:00.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.289+0000 7fbe30ff9640 1 -- 172.21.15.31:0/3809799006 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe280516b0 con 0x7fbe34151840 2023-10-30T13:50:00.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe34073c50 con 0x7fbe34151840 2023-10-30T13:50:00.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe30ff9640 1 -- 172.21.15.31:0/3809799006 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5705 (secure 0 0 0) 0x7fbe28054cd0 con 0x7fbe34151840 2023-10-30T13:50:00.485 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:50:00.485 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:56.226057+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 69 2023-10-30T13:50:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbe200664d0 msgr2=0x7fbe20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbe200664d0 0x7fbe20068990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbe240b2000 tx=0x7fbe24009040 comp rx=0 tx=0).stop 2023-10-30T13:50:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 msgr2=0x7fbe341f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe341f2e60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800d620 tx=0x7fbe2800dae0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 shutdown_connections 2023-10-30T13:50:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbe200664d0 0x7fbe20068990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe34151c20 0x7fbe341e4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe34151840 0x7fbe341f2e60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 --2- 172.21.15.31:0/3809799006 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbe340cb8b0 0x7fbe341eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 >> 172.21.15.31:0/3809799006 conn(0x7fbe340cac70 msgr2=0x7fbe34145950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 shutdown_connections 2023-10-30T13:50:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:00.485+0000 7fbe3a30f640 1 -- 172.21.15.31:0/3809799006 wait complete. 2023-10-30T13:50:00.506 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:02.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:02.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.761+0000 7f362bf3e640 1 Processor -- start 2023-10-30T13:50:02.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.761+0000 7f362bf3e640 1 -- start start 2023-10-30T13:50:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f362bf3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f36240c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f362bf3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3624151c20 0x7f3624072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f362bf3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240c5030 con 0x7f36240cb8b0 2023-10-30T13:50:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f362bf3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3624157830 con 0x7f3624151c20 2023-10-30T13:50:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f362bf3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3624157b10 con 0x7f3624151840 2023-10-30T13:50:02.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f36294d9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3624151c20 0x7f3624072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:02.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f3629cda640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f36240c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:02.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f3629cda640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f36240c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:47624/0 (socket says 172.21.15.31:47624) 2023-10-30T13:50:02.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f3629cda640 1 -- 172.21.15.31:0/3778489910 learned_addr learned my addr 172.21.15.31:0/3778489910 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:02.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f36294d9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3624151c20 0x7f3624072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41416/0 (socket says 172.21.15.31:41416) 2023-10-30T13:50:02.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f3629cda640 1 -- 172.21.15.31:0/3778489910 >> v2:172.21.15.31:3300/0 conn(0x7f3624151c20 msgr2=0x7f3624072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:02.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f3629cda640 1 --2- 172.21.15.31:0/3778489910 >> v2:172.21.15.31:3300/0 conn(0x7f3624151c20 0x7f3624072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:02.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f3629cda640 1 -- 172.21.15.31:0/3778489910 >> v1:172.21.15.31:6789/0 conn(0x7f36240cb8b0 legacy=0x7f362406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:02.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.765+0000 7f3629cda640 1 -- 172.21.15.31:0/3778489910 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624157df0 con 0x7f3624151840 2023-10-30T13:50:02.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.769+0000 7f3629cda640 1 --2- 172.21.15.31:0/3778489910 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f36240c49a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f361c009e50 tx=0x7f361c023600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d8574855335659e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f3628cd8640 1 -- 172.21.15.31:0/3778489910 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0b3430 con 0x7f3624151840 2023-10-30T13:50:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f3628cd8640 1 -- 172.21.15.31:0/3778489910 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c0b3590 con 0x7f3624151840 2023-10-30T13:50:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f3628cd8640 1 -- 172.21.15.31:0/3778489910 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0b3920 con 0x7f3624151840 2023-10-30T13:50:02.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 -- 172.21.15.31:0/3778489910 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 msgr2=0x7f36240c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:02.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 --2- 172.21.15.31:0/3778489910 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f36240c49a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f361c009e50 tx=0x7f361c023600 comp rx=0 tx=0).stop 2023-10-30T13:50:02.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 -- 172.21.15.31:0/3778489910 shutdown_connections 2023-10-30T13:50:02.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 --2- 172.21.15.31:0/3778489910 >> v2:172.21.15.31:3300/0 conn(0x7f3624151c20 0x7f3624072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:02.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 --2- 172.21.15.31:0/3778489910 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f36240c49a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:02.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 -- 172.21.15.31:0/3778489910 >> 172.21.15.31:0/3778489910 conn(0x7f36240cac70 msgr2=0x7f36240cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:02.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 -- 172.21.15.31:0/3778489910 shutdown_connections 2023-10-30T13:50:02.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.773+0000 7f362bf3e640 1 -- 172.21.15.31:0/3778489910 wait complete. 2023-10-30T13:50:02.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 Processor -- start 2023-10-30T13:50:02.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 -- start start 2023-10-30T13:50:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240cb8b0 0x7f3624156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f362414e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3624151c20 0x7f362414e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362414dce0 con 0x7f36240cb8b0 2023-10-30T13:50:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3624154d40 con 0x7f3624151c20 2023-10-30T13:50:02.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362bf3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36241ea990 con 0x7f3624151840 2023-10-30T13:50:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f3629cda640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240cb8b0 0x7f3624156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f36294d9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f362414e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3624151c20 0x7f362414e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3624151c20 0x7f362414e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:60714/0 (socket says 172.21.15.31:60714) 2023-10-30T13:50:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 -- 172.21.15.31:0/665507974 learned_addr learned my addr 172.21.15.31:0/665507974 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 -- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 msgr2=0x7f362414e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f362414e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 -- 172.21.15.31:0/665507974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240cb8b0 msgr2=0x7f3624156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240cb8b0 0x7f3624156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.777+0000 7f362a4db640 1 -- 172.21.15.31:0/665507974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c009b30 con 0x7f3624151c20 2023-10-30T13:50:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.781+0000 7f362a4db640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3624151c20 0x7f362414e6b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f362000c390 tx=0x7f362000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.781+0000 7f361a7fc640 1 -- 172.21.15.31:0/665507974 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36200052d0 con 0x7f3624151c20 2023-10-30T13:50:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.781+0000 7f361a7fc640 1 -- 172.21.15.31:0/665507974 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3620005430 con 0x7f3624151c20 2023-10-30T13:50:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.781+0000 7f361a7fc640 1 -- 172.21.15.31:0/665507974 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620007240 con 0x7f3624151c20 2023-10-30T13:50:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.781+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36241eac30 con 0x7f3624151c20 2023-10-30T13:50:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.781+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36241eb1f0 con 0x7f3624151c20 2023-10-30T13:50:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.781+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362414de50 con 0x7f3624151c20 2023-10-30T13:50:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.785+0000 7f361a7fc640 1 -- 172.21.15.31:0/665507974 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f36200073a0 con 0x7f3624151c20 2023-10-30T13:50:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.785+0000 7f361a7fc640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f36080665a0 0x7f3608068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.785+0000 7f361a7fc640 1 -- 172.21.15.31:0/665507974 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f36200833b0 con 0x7f3624151c20 2023-10-30T13:50:02.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.789+0000 7f3629cda640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f36080665a0 0x7f3608068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:02.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.789+0000 7f3629cda640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f36080665a0 0x7f3608068a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f361c0b5000 tx=0x7f361c0af000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:02.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:02.801+0000 7f361a7fc640 1 -- 172.21.15.31:0/665507974 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3620050660 con 0x7f3624151c20 2023-10-30T13:50:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.061+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f362404e170 con 0x7f3624151c20 2023-10-30T13:50:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f361a7fc640 1 -- 172.21.15.31:0/665507974 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f3620053c80 con 0x7f3624151c20 2023-10-30T13:50:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:50:03.065 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:49:56.226057+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 69 2023-10-30T13:50:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f36080665a0 msgr2=0x7f3608068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f36080665a0 0x7f3608068a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f361c0b5000 tx=0x7f361c0af000 comp rx=0 tx=0).stop 2023-10-30T13:50:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3624151c20 msgr2=0x7f362414e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3624151c20 0x7f362414e6b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f362000c390 tx=0x7f362000c850 comp rx=0 tx=0).stop 2023-10-30T13:50:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 shutdown_connections 2023-10-30T13:50:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f36080665a0 0x7f3608068a60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3624151c20 0x7f362414e6b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3624151840 0x7f362414e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 --2- 172.21.15.31:0/665507974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240cb8b0 0x7f3624156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 >> 172.21.15.31:0/665507974 conn(0x7f36240cac70 msgr2=0x7f36240cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 shutdown_connections 2023-10-30T13:50:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:03.065+0000 7f362bf3e640 1 -- 172.21.15.31:0/665507974 wait complete. 2023-10-30T13:50:03.086 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 69, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5889": { "addr": "172.21.15.31:6837/327248983", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 327248983, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 327248983, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5889, "incarnation": 66, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:49:56.226057+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5889 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:50:05.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bef65640 1 Processor -- start 2023-10-30T13:50:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bef65640 1 -- start start 2023-10-30T13:50:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bef65640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f46b80c4a10 0x7f46b815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bef65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c42a0 0x7f46b80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bef65640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b8159f20 con 0x7f46b8153c80 2023-10-30T13:50:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bef65640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b815dec0 con 0x7f46b80c4a10 2023-10-30T13:50:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bef65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b81599a0 con 0x7f46b80c42a0 2023-10-30T13:50:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46b7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f46b80c4a10 0x7f46b815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46b7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f46b80c4a10 0x7f46b815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55982/0 (socket says 172.21.15.31:55982) 2023-10-30T13:50:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46b7fff640 1 -- 172.21.15.31:0/1898697710 learned_addr learned my addr 172.21.15.31:0/1898697710 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46bd502640 1 --2- 172.21.15.31:0/1898697710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c42a0 0x7f46b80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46b7fff640 1 -- 172.21.15.31:0/1898697710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c42a0 msgr2=0x7f46b80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46b7fff640 1 --2- 172.21.15.31:0/1898697710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c42a0 0x7f46b80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46b7fff640 1 -- 172.21.15.31:0/1898697710 >> v1:172.21.15.33:6789/0 conn(0x7f46b8153c80 legacy=0x7f46b815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.245+0000 7f46b7fff640 1 -- 172.21.15.31:0/1898697710 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b815d620 con 0x7f46b80c4a10 2023-10-30T13:50:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46b7fff640 1 --2- 172.21.15.31:0/1898697710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b80c4a10 0x7f46b815cf90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f46a4009870 tx=0x7f46a401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be599d8fe18b0c11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46b77fe640 1 -- 172.21.15.31:0/1898697710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a40a8a30 con 0x7f46b80c4a10 2023-10-30T13:50:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46b77fe640 1 -- 172.21.15.31:0/1898697710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a40a8b90 con 0x7f46b80c4a10 2023-10-30T13:50:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46b77fe640 1 -- 172.21.15.31:0/1898697710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a40b0440 con 0x7f46b80c4a10 2023-10-30T13:50:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 -- 172.21.15.31:0/1898697710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b80c4a10 msgr2=0x7f46b815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 --2- 172.21.15.31:0/1898697710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b80c4a10 0x7f46b815cf90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f46a4009870 tx=0x7f46a401ed30 comp rx=0 tx=0).stop 2023-10-30T13:50:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 -- 172.21.15.31:0/1898697710 shutdown_connections 2023-10-30T13:50:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 --2- 172.21.15.31:0/1898697710 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c42a0 0x7f46b80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 --2- 172.21.15.31:0/1898697710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b80c4a10 0x7f46b815cf90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 -- 172.21.15.31:0/1898697710 >> 172.21.15.31:0/1898697710 conn(0x7f46b80715c0 msgr2=0x7f46b80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 -- 172.21.15.31:0/1898697710 shutdown_connections 2023-10-30T13:50:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.249+0000 7f46bef65640 1 -- 172.21.15.31:0/1898697710 wait complete. 2023-10-30T13:50:05.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 Processor -- start 2023-10-30T13:50:05.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 -- start start 2023-10-30T13:50:05.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b80c42a0 0x7f46b80c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c4a10 0x7f46b8149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b8153c80 0x7f46b814a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b814bf50 con 0x7f46b8153c80 2023-10-30T13:50:05.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b814bad0 con 0x7f46b80c42a0 2023-10-30T13:50:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c9410 con 0x7f46b80c4a10 2023-10-30T13:50:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b8153c80 0x7f46b814a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b8153c80 0x7f46b814a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55990/0 (socket says 172.21.15.31:55990) 2023-10-30T13:50:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 -- 172.21.15.31:0/3444270046 learned_addr learned my addr 172.21.15.31:0/3444270046 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46b7fff640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c4a10 0x7f46b8149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bcd01640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b80c42a0 0x7f46b80c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 -- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c4a10 msgr2=0x7f46b8149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c4a10 0x7f46b8149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 -- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b80c42a0 msgr2=0x7f46b80c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b80c42a0 0x7f46b80c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 -- 172.21.15.31:0/3444270046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40094d0 con 0x7f46b8153c80 2023-10-30T13:50:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bd502640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b8153c80 0x7f46b814a070 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f46b0007e40 tx=0x7f46b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46b57fa640 1 -- 172.21.15.31:0/3444270046 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0014620 con 0x7f46b8153c80 2023-10-30T13:50:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46b57fa640 1 -- 172.21.15.31:0/3444270046 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b00147a0 con 0x7f46b8153c80 2023-10-30T13:50:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46b57fa640 1 -- 172.21.15.31:0/3444270046 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0014b50 con 0x7f46b8153c80 2023-10-30T13:50:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.253+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b814ee80 con 0x7f46b8153c80 2023-10-30T13:50:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.257+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b814e5e0 con 0x7f46b8153c80 2023-10-30T13:50:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.257+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b814c090 con 0x7f46b8153c80 2023-10-30T13:50:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.257+0000 7f46b57fa640 1 -- 172.21.15.31:0/3444270046 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46b0014cb0 con 0x7f46b8153c80 2023-10-30T13:50:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.257+0000 7f46b57fa640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f469c0664a0 0x7f469c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.257+0000 7f46b57fa640 1 -- 172.21.15.31:0/3444270046 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f46b0084a90 con 0x7f46b8153c80 2023-10-30T13:50:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.257+0000 7f46bcd01640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f469c0664a0 0x7f469c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.261+0000 7f46bcd01640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f469c0664a0 0x7f469c068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f46ac007fd0 tx=0x7f46ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.261+0000 7f46b57fa640 1 -- 172.21.15.31:0/3444270046 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f46b814c090 con 0x7f46b8153c80 2023-10-30T13:50:05.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46b814c090 con 0x7f46b8153c80 2023-10-30T13:50:05.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46b57fa640 1 -- 172.21.15.31:0/3444270046 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f46b814c090 con 0x7f46b8153c80 2023-10-30T13:50:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:50:05.457 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:05.458 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 70 2023-10-30T13:50:05.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f469c0664a0 msgr2=0x7f469c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:05.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f469c0664a0 0x7f469c068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f46ac007fd0 tx=0x7f46ac009040 comp rx=0 tx=0).stop 2023-10-30T13:50:05.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b8153c80 msgr2=0x7f46b814a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:05.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b8153c80 0x7f46b814a070 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f46b0007e40 tx=0x7f46b000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 shutdown_connections 2023-10-30T13:50:05.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f469c0664a0 0x7f469c068960 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b8153c80 0x7f46b814a070 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f46b80c4a10 0x7f46b8149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 --2- 172.21.15.31:0/3444270046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46b80c42a0 0x7f46b80c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 >> 172.21.15.31:0/3444270046 conn(0x7f46b80715c0 msgr2=0x7f46b81591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:05.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 shutdown_connections 2023-10-30T13:50:05.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:05.457+0000 7f46bef65640 1 -- 172.21.15.31:0/3444270046 wait complete. 2023-10-30T13:50:05.479 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5842 2023-10-30T13:50:05.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.d 2023-10-30T13:50:11.004 INFO:tasks.ceph.mds.a.smithi031.stderr:2023-10-30T13:50:11.005+0000 7f9306c95640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:50:14.480 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:50:14.480 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:50:14.481 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:50:14.484 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:50:14.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:14.564 INFO:tasks.ceph.mds.d.smithi033.stdout:starting mds.d at 2023-10-30T13:50:14.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6eeb765640 1 Processor -- start 2023-10-30T13:50:14.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6eeb765640 1 -- start start 2023-10-30T13:50:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6eeb765640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee4151840 0x7f6ee40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6eeb765640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f6ee4151c20 0x7f6ee4072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:14.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6eeb765640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee40c5030 con 0x7f6ee40cb8b0 2023-10-30T13:50:14.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6eeb765640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee4157830 con 0x7f6ee4151c20 2023-10-30T13:50:14.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6eeb765640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee4157b10 con 0x7f6ee4151840 2023-10-30T13:50:14.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6ee9d02640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f6ee40cb8b0 0x7f6ee406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43062/0 (socket says 172.21.15.31:43062) 2023-10-30T13:50:14.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6ee9d02640 1 -- 172.21.15.31:0/3014616609 learned_addr learned my addr 172.21.15.31:0/3014616609 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:50:14.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6ee9501640 1 --2- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee4151840 0x7f6ee40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:14.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.649+0000 7f6ee8d00640 1 --2- 172.21.15.31:0/3014616609 >> v2:172.21.15.33:3300/0 conn(0x7f6ee4151c20 0x7f6ee4072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:14.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.653+0000 7f6edbfff640 1 -- 172.21.15.31:0/3014616609 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270514843 0 0) 0x7f6ee40c5030 con 0x7f6ee40cb8b0 2023-10-30T13:50:14.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.653+0000 7f6edbfff640 1 -- 172.21.15.31:0/3014616609 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed0003530 con 0x7f6ee40cb8b0 2023-10-30T13:50:14.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.653+0000 7f6ee8d00640 1 -- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee4151840 msgr2=0x7f6ee40c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:14.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.653+0000 7f6ee8d00640 1 --2- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee4151840 0x7f6ee40c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.653+0000 7f6ee8d00640 1 -- 172.21.15.31:0/3014616609 >> v1:172.21.15.31:6789/0 conn(0x7f6ee40cb8b0 legacy=0x7f6ee406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:14.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.653+0000 7f6ee8d00640 1 -- 172.21.15.31:0/3014616609 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4157df0 con 0x7f6ee4151c20 2023-10-30T13:50:14.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.653+0000 7f6ee8d00640 1 --2- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151c20 0x7f6ee4072000 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40097c0 tx=0x7f6ed401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=387bbb6624aa5c5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:14.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6edbfff640 1 -- 172.21.15.31:0/3014616609 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed40a8970 con 0x7f6ee4151c20 2023-10-30T13:50:14.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6edbfff640 1 -- 172.21.15.31:0/3014616609 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed40a8ad0 con 0x7f6ee4151c20 2023-10-30T13:50:14.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6edbfff640 1 -- 172.21.15.31:0/3014616609 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed40a8e40 con 0x7f6ee4151c20 2023-10-30T13:50:14.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151c20 msgr2=0x7f6ee4072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:14.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151c20 0x7f6ee4072000 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40097c0 tx=0x7f6ed401ec70 comp rx=0 tx=0).stop 2023-10-30T13:50:14.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- 172.21.15.31:0/3014616609 shutdown_connections 2023-10-30T13:50:14.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151c20 0x7f6ee4072000 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3014616609 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee4151840 0x7f6ee40c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- 172.21.15.31:0/3014616609 >> 172.21.15.31:0/3014616609 conn(0x7f6ee40cac70 msgr2=0x7f6ee40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:14.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- 172.21.15.31:0/3014616609 shutdown_connections 2023-10-30T13:50:14.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- 172.21.15.31:0/3014616609 wait complete. 2023-10-30T13:50:14.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 Processor -- start 2023-10-30T13:50:14.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- start start 2023-10-30T13:50:14.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee40cb8b0 0x7f6ee4147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:14.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151840 0x7f6ee40c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:14.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee4151c20 0x7f6ee40c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:14.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee41e3cc0 con 0x7f6ee4151c20 2023-10-30T13:50:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee41e33b0 con 0x7f6ee4151840 2023-10-30T13:50:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.657+0000 7f6eeb765640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee41e3670 con 0x7f6ee40cb8b0 2023-10-30T13:50:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee4151c20 0x7f6ee40c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee4151c20 0x7f6ee40c92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55994/0 (socket says 172.21.15.31:55994) 2023-10-30T13:50:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 -- 172.21.15.31:0/3921917089 learned_addr learned my addr 172.21.15.31:0/3921917089 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9501640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee40cb8b0 0x7f6ee4147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee8d00640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151840 0x7f6ee40c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:14.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 -- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee40cb8b0 msgr2=0x7f6ee4147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:14.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee40cb8b0 0x7f6ee4147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 -- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151840 msgr2=0x7f6ee40c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151840 0x7f6ee40c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 -- 172.21.15.31:0/3921917089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed4009420 con 0x7f6ee4151c20 2023-10-30T13:50:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ee9d02640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee4151c20 0x7f6ee40c92a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00cb40 tx=0x7f6edc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ed9ffb640 1 -- 172.21.15.31:0/3921917089 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00aca0 con 0x7f6ee4151c20 2023-10-30T13:50:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ed9ffb640 1 -- 172.21.15.31:0/3921917089 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc00ae00 con 0x7f6ee4151c20 2023-10-30T13:50:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ed9ffb640 1 -- 172.21.15.31:0/3921917089 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc012620 con 0x7f6ee4151c20 2023-10-30T13:50:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee41e4630 con 0x7f6ee4151c20 2023-10-30T13:50:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee40c8150 con 0x7f6ee4151c20 2023-10-30T13:50:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee41e3e00 con 0x7f6ee4151c20 2023-10-30T13:50:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.661+0000 7f6ed9ffb640 1 -- 172.21.15.31:0/3921917089 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6edc007070 con 0x7f6ee4151c20 2023-10-30T13:50:14.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.665+0000 7f6ed9ffb640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6ec80664d0 0x7f6ec8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:14.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.665+0000 7f6ed9ffb640 1 -- 172.21.15.31:0/3921917089 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6edc084150 con 0x7f6ee4151c20 2023-10-30T13:50:14.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.665+0000 7f6ee9501640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6ec80664d0 0x7f6ec8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:14.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.665+0000 7f6ee9501640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6ec80664d0 0x7f6ec8068990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009b60 tx=0x7f6ee0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:14.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.669+0000 7f6ed9ffb640 1 -- 172.21.15.31:0/3921917089 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6edc0512f0 con 0x7f6ee4151c20 2023-10-30T13:50:14.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.885+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee404e170 con 0x7f6ee4151c20 2023-10-30T13:50:14.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.885+0000 7f6ed9ffb640 1 -- 172.21.15.31:0/3921917089 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5699 (secure 0 0 0) 0x7f6edc054910 con 0x7f6ee4151c20 2023-10-30T13:50:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:50:14.887 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 70 2023-10-30T13:50:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6ec80664d0 msgr2=0x7f6ec8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6ec80664d0 0x7f6ec8068990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009b60 tx=0x7f6ee0009190 comp rx=0 tx=0).stop 2023-10-30T13:50:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee4151c20 msgr2=0x7f6ee40c92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee4151c20 0x7f6ee40c92a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00cb40 tx=0x7f6edc00a060 comp rx=0 tx=0).stop 2023-10-30T13:50:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 shutdown_connections 2023-10-30T13:50:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6ec80664d0 0x7f6ec8068990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee4151c20 0x7f6ee40c92a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee4151840 0x7f6ee40c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 --2- 172.21.15.31:0/3921917089 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6ee40cb8b0 0x7f6ee4147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 >> 172.21.15.31:0/3921917089 conn(0x7f6ee40cac70 msgr2=0x7f6ee40cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 shutdown_connections 2023-10-30T13:50:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:14.889+0000 7f6eeb765640 1 -- 172.21.15.31:0/3921917089 wait complete. 2023-10-30T13:50:14.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:50:16.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73f1d1d640 1 Processor -- start 2023-10-30T13:50:17.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73f1d1d640 1 -- start start 2023-10-30T13:50:17.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73f1d1d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f73ec153c80 0x7f73ec15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:17.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73f1d1d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f73ec0c42a0 0x7f73ec0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:17.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73f1d1d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec159f20 con 0x7f73ec0c4a10 2023-10-30T13:50:17.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73f1d1d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ec15dec0 con 0x7f73ec0c42a0 2023-10-30T13:50:17.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73f1d1d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ec1599a0 con 0x7f73ec153c80 2023-10-30T13:50:17.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73ebfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f73ec0c42a0 0x7f73ec0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:17.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73ebfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f73ec0c42a0 0x7f73ec0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37634/0 (socket says 172.21.15.31:37634) 2023-10-30T13:50:17.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73ebfff640 1 -- 172.21.15.31:0/2139987764 learned_addr learned my addr 172.21.15.31:0/2139987764 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:17.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73eb7fe640 1 --2- 172.21.15.31:0/2139987764 >> v2:172.21.15.33:3300/0 conn(0x7f73ec153c80 0x7f73ec15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:17.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73ebfff640 1 -- 172.21.15.31:0/2139987764 >> v2:172.21.15.33:3300/0 conn(0x7f73ec153c80 msgr2=0x7f73ec15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:17.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73ebfff640 1 --2- 172.21.15.31:0/2139987764 >> v2:172.21.15.33:3300/0 conn(0x7f73ec153c80 0x7f73ec15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73ebfff640 1 -- 172.21.15.31:0/2139987764 >> v1:172.21.15.31:6789/0 conn(0x7f73ec0c4a10 legacy=0x7f73ec15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:17.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.057+0000 7f73ebfff640 1 -- 172.21.15.31:0/2139987764 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec15d620 con 0x7f73ec0c42a0 2023-10-30T13:50:17.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73ebfff640 1 --2- 172.21.15.31:0/2139987764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec0c42a0 0x7f73ec0ca1e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0d00f0 tx=0x7f73e401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3be5f51a008ce3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:17.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73ea7fc640 1 -- 172.21.15.31:0/2139987764 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e40af400 con 0x7f73ec0c42a0 2023-10-30T13:50:17.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73ea7fc640 1 -- 172.21.15.31:0/2139987764 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e40ab050 con 0x7f73ec0c42a0 2023-10-30T13:50:17.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73ea7fc640 1 -- 172.21.15.31:0/2139987764 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e40af780 con 0x7f73ec0c42a0 2023-10-30T13:50:17.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 -- 172.21.15.31:0/2139987764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec0c42a0 msgr2=0x7f73ec0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:17.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/2139987764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec0c42a0 0x7f73ec0ca1e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0d00f0 tx=0x7f73e401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 -- 172.21.15.31:0/2139987764 shutdown_connections 2023-10-30T13:50:17.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/2139987764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec0c42a0 0x7f73ec0ca1e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/2139987764 >> v2:172.21.15.33:3300/0 conn(0x7f73ec153c80 0x7f73ec15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 -- 172.21.15.31:0/2139987764 >> 172.21.15.31:0/2139987764 conn(0x7f73ec0715c0 msgr2=0x7f73ec0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:17.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 -- 172.21.15.31:0/2139987764 shutdown_connections 2023-10-30T13:50:17.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.061+0000 7f73f1d1d640 1 -- 172.21.15.31:0/2139987764 wait complete. 2023-10-30T13:50:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 Processor -- start 2023-10-30T13:50:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 -- start start 2023-10-30T13:50:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73ec0c42a0 0x7f73ec0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73ec0c4a10 0x7f73ec1526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec153c80 0x7f73ec152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:17.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec0b2d00 con 0x7f73ec153c80 2023-10-30T13:50:17.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec0b2870 con 0x7f73ec0c4a10 2023-10-30T13:50:17.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec156420 con 0x7f73ec0c42a0 2023-10-30T13:50:17.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec153c80 0x7f73ec152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:17.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec153c80 0x7f73ec152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37646/0 (socket says 172.21.15.31:37646) 2023-10-30T13:50:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 -- 172.21.15.31:0/3530103985 learned_addr learned my addr 172.21.15.31:0/3530103985 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:50:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73eb7fe640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73ec0c42a0 0x7f73ec0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73eaffd640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73ec0c4a10 0x7f73ec1526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 -- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73ec0c42a0 msgr2=0x7f73ec0cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73ec0c42a0 0x7f73ec0cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 -- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73ec0c4a10 msgr2=0x7f73ec1526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73ec0c4a10 0x7f73ec1526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 -- 172.21.15.31:0/3530103985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e4009b30 con 0x7f73ec153c80 2023-10-30T13:50:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73ebfff640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec153c80 0x7f73ec152c30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f73e40b2040 tx=0x7f73e401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73d3fff640 1 -- 172.21.15.31:0/3530103985 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e40afdc0 con 0x7f73ec153c80 2023-10-30T13:50:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73d3fff640 1 -- 172.21.15.31:0/3530103985 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e4009510 con 0x7f73ec153c80 2023-10-30T13:50:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73d3fff640 1 -- 172.21.15.31:0/3530103985 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e40b8640 con 0x7f73ec153c80 2023-10-30T13:50:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ec153340 con 0x7f73ec153c80 2023-10-30T13:50:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ec0b2500 con 0x7f73ec153c80 2023-10-30T13:50:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.065+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ec157060 con 0x7f73ec153c80 2023-10-30T13:50:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.069+0000 7f73d3fff640 1 -- 172.21.15.31:0/3530103985 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73e40b6040 con 0x7f73ec153c80 2023-10-30T13:50:17.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.069+0000 7f73d3fff640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73d80664d0 0x7f73d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.069+0000 7f73d3fff640 1 -- 172.21.15.31:0/3530103985 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f73e4121ab0 con 0x7f73ec153c80 2023-10-30T13:50:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.069+0000 7f73eb7fe640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73d80664d0 0x7f73d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.069+0000 7f73eb7fe640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73d80664d0 0x7f73d8068990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00b930 tx=0x7f73dc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.073+0000 7f73d3fff640 1 -- 172.21.15.31:0/3530103985 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73ec157060 con 0x7f73ec153c80 2023-10-30T13:50:17.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.285+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73ec157060 con 0x7f73ec153c80 2023-10-30T13:50:17.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73d3fff640 1 -- 172.21.15.31:0/3530103985 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6490 (secure 0 0 0) 0x7f73ec157060 con 0x7f73ec153c80 2023-10-30T13:50:17.288 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:50:17.289 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5591,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:17.289 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2023-10-30T13:50:17.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73d80664d0 msgr2=0x7f73d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:17.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73d80664d0 0x7f73d8068990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00b930 tx=0x7f73dc008040 comp rx=0 tx=0).stop 2023-10-30T13:50:17.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec153c80 msgr2=0x7f73ec152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:17.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec153c80 0x7f73ec152c30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f73e40b2040 tx=0x7f73e401bf60 comp rx=0 tx=0).stop 2023-10-30T13:50:17.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 shutdown_connections 2023-10-30T13:50:17.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f73d80664d0 0x7f73d8068990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ec153c80 0x7f73ec152c30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73ec0c4a10 0x7f73ec1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 --2- 172.21.15.31:0/3530103985 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f73ec0c42a0 0x7f73ec0cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:17.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 >> 172.21.15.31:0/3530103985 conn(0x7f73ec0715c0 msgr2=0x7f73ec158790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:17.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 shutdown_connections 2023-10-30T13:50:17.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:50:17.289+0000 7f73f1d1d640 1 -- 172.21.15.31:0/3530103985 wait complete. 2023-10-30T13:50:17.319 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:50:17.320 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-10-30T13:50:27.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:27.251 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:57.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:57.319 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:09.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:09.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.464+0000 7f9b9fea9640 1 Processor -- start 2023-10-30T13:51:09.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.464+0000 7f9b9fea9640 1 -- start start 2023-10-30T13:51:09.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.464+0000 7f9b9fea9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 0x7f9b980cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.464+0000 7f9b9fea9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f9b980c0260 0x7f9b980cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.464+0000 7f9b9fea9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b980c1e60 con 0x7f9b9806a0f0 2023-10-30T13:51:09.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.464+0000 7f9b9fea9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9815a3f0 con 0x7f9b980c0260 2023-10-30T13:51:09.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.464+0000 7f9b9fea9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980cb2b0 con 0x7f9b9806a4d0 2023-10-30T13:51:09.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9dc45640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f9b9806a0f0 0x7f9b98153830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37472/0 (socket says 172.21.15.31:37472) 2023-10-30T13:51:09.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9dc45640 1 -- 172.21.15.31:0/44860911 learned_addr learned my addr 172.21.15.31:0/44860911 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:09.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9e446640 1 --2- 172.21.15.31:0/44860911 >> v2:172.21.15.33:3300/0 conn(0x7f9b980c0260 0x7f9b980cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9d444640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 0x7f9b980cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553756075 0 0) 0x7f9b980c1e60 con 0x7f9b9806a0f0 2023-10-30T13:51:09.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b84003530 con 0x7f9b9806a0f0 2023-10-30T13:51:09.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822925152 0 0) 0x7f9b84003530 con 0x7f9b9806a0f0 2023-10-30T13:51:09.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 msgr2=0x7f9b980cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:09.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 0x7f9b980cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 >> v2:172.21.15.33:3300/0 conn(0x7f9b980c0260 msgr2=0x7f9b980cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:09.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 --2- 172.21.15.31:0/44860911 >> v2:172.21.15.33:3300/0 conn(0x7f9b980c0260 0x7f9b980cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9815a8c0 con 0x7f9b9806a0f0 2023-10-30T13:51:09.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f9b94001680 con 0x7f9b9806a0f0 2023-10-30T13:51:09.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 >> v1:172.21.15.31:6789/0 conn(0x7f9b9806a0f0 legacy=0x7f9b98153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 0x7f9b84003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b84004d10 0x7f9b84005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980cb2b0 con 0x7f9b84003f00 2023-10-30T13:51:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8400b990 con 0x7f9b84004d10 2023-10-30T13:51:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8400bad0 con 0x7f9b9806a4d0 2023-10-30T13:51:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9d444640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9dc45640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 0x7f9b84003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9e446640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b84004d10 0x7f9b84005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9d444640 1 -- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 msgr2=0x7f9b84003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:09.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9d444640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b9806a4d0 0x7f9b84003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9d444640 1 -- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b84004d10 msgr2=0x7f9b84005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:09.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9d444640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b84004d10 0x7f9b84005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.468+0000 7f9b9d444640 1 -- 172.21.15.31:0/44860911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84003530 con 0x7f9b84003f00 2023-10-30T13:51:09.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9d444640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b840047d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009ab0 tx=0x7f9b8801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737e4f26050511b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:09.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880a5620 con 0x7f9b84003f00 2023-10-30T13:51:09.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b880a5780 con 0x7f9b84003f00 2023-10-30T13:51:09.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9cc43640 1 -- 172.21.15.31:0/44860911 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880a5b30 con 0x7f9b84003f00 2023-10-30T13:51:09.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 -- 172.21.15.31:0/44860911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 msgr2=0x7f9b840047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 --2- 172.21.15.31:0/44860911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b840047d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009ab0 tx=0x7f9b8801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9e446640 1 -- 172.21.15.31:0/44860911 reap_dead start 2023-10-30T13:51:09.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 -- 172.21.15.31:0/44860911 shutdown_connections 2023-10-30T13:51:09.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 -- 172.21.15.31:0/44860911 >> 172.21.15.31:0/44860911 conn(0x7f9b9806de90 msgr2=0x7f9b9806e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:09.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 -- 172.21.15.31:0/44860911 shutdown_connections 2023-10-30T13:51:09.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 -- 172.21.15.31:0/44860911 wait complete. 2023-10-30T13:51:09.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 Processor -- start 2023-10-30T13:51:09.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.472+0000 7f9b9fea9640 1 -- start start 2023-10-30T13:51:09.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b980af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b980709e0 0x7f9b98070e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b980714a0 0x7f9b98071920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9815b8f0 con 0x7f9b84003f00 2023-10-30T13:51:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98070550 con 0x7f9b980714a0 2023-10-30T13:51:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b981f2c10 con 0x7f9b980709e0 2023-10-30T13:51:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b980af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b980af8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43774/0 (socket says 172.21.15.31:43774) 2023-10-30T13:51:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 -- 172.21.15.31:0/3385477442 learned_addr learned my addr 172.21.15.31:0/3385477442 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9d444640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b980709e0 0x7f9b98070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9e446640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b980714a0 0x7f9b98071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 -- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b980709e0 msgr2=0x7f9b98070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b980709e0 0x7f9b98070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 -- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b980714a0 msgr2=0x7f9b98071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b980714a0 0x7f9b98071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 -- 172.21.15.31:0/3385477442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9815db10 con 0x7f9b84003f00 2023-10-30T13:51:09.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9dc45640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b980af8e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9b940122e0 tx=0x7f9b940127a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:09.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b8e7fc640 1 -- 172.21.15.31:0/3385477442 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94014750 con 0x7f9b84003f00 2023-10-30T13:51:09.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b8e7fc640 1 -- 172.21.15.31:0/3385477442 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b940148d0 con 0x7f9b84003f00 2023-10-30T13:51:09.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b981f2eb0 con 0x7f9b84003f00 2023-10-30T13:51:09.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b8e7fc640 1 -- 172.21.15.31:0/3385477442 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94014c80 con 0x7f9b84003f00 2023-10-30T13:51:09.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9815ddd0 con 0x7f9b84003f00 2023-10-30T13:51:09.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b981f30f0 con 0x7f9b84003f00 2023-10-30T13:51:09.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.476+0000 7f9b8e7fc640 1 -- 172.21.15.31:0/3385477442 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b940153e0 con 0x7f9b84003f00 2023-10-30T13:51:09.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.480+0000 7f9b8e7fc640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b7c066520 0x7f9b7c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.480+0000 7f9b8e7fc640 1 -- 172.21.15.31:0/3385477442 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9b94086e00 con 0x7f9b84003f00 2023-10-30T13:51:09.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.480+0000 7f9b9d444640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b7c066520 0x7f9b7c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.480+0000 7f9b9d444640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b7c066520 0x7f9b7c0689e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9b880af000 tx=0x7f9b880046d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:09.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.484+0000 7f9b8e7fc640 1 -- 172.21.15.31:0/3385477442 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b94053f90 con 0x7f9b84003f00 2023-10-30T13:51:09.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.672+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b98070300 con 0x7f9b84003f00 2023-10-30T13:51:09.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.672+0000 7f9b8e7fc640 1 -- 172.21.15.31:0/3385477442 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6490 (secure 0 0 0) 0x7f9b9401c030 con 0x7f9b84003f00 2023-10-30T13:51:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:09.676 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5591,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:09.676 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2023-10-30T13:51:09.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b7c066520 msgr2=0x7f9b7c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b7c066520 0x7f9b7c0689e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9b880af000 tx=0x7f9b880046d0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 msgr2=0x7f9b980af8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b980af8e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9b940122e0 tx=0x7f9b940127a0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 shutdown_connections 2023-10-30T13:51:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f9b7c066520 0x7f9b7c0689e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b980714a0 0x7f9b98071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9b980709e0 0x7f9b98070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 --2- 172.21.15.31:0/3385477442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b84003f00 0x7f9b980af8e0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 >> 172.21.15.31:0/3385477442 conn(0x7f9b9806de90 msgr2=0x7f9b98151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:09.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 shutdown_connections 2023-10-30T13:51:09.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.676+0000 7f9b9fea9640 1 -- 172.21.15.31:0/3385477442 wait complete. 2023-10-30T13:51:09.696 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:51:09.697 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:51:09.697 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:51:09.717 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:51:09.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:09.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.900+0000 7fcffeb3e640 1 Processor -- start 2023-10-30T13:51:09.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.900+0000 7fcffeb3e640 1 -- start start 2023-10-30T13:51:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.900+0000 7fcffeb3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 0x7fcff815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.900+0000 7fcffeb3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff80c2aa0 con 0x7fcff81537f0 2023-10-30T13:51:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.900+0000 7fcffeb3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff80708e0 con 0x7fcff806b2f0 2023-10-30T13:51:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.900+0000 7fcffeb3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80c3ab0 con 0x7fcff80701a0 2023-10-30T13:51:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffc8da640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fcff81537f0 0x7fcff80ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:37484/0 (socket says 172.21.15.31:37484) 2023-10-30T13:51:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffc8da640 1 -- 172.21.15.31:0/2952901963 learned_addr learned my addr 172.21.15.31:0/2952901963 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:09.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff7fff640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 0x7fcff815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752407665 0 0) 0x7fcff80c2aa0 con 0x7fcff81537f0 2023-10-30T13:51:09.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe8003530 con 0x7fcff81537f0 2023-10-30T13:51:09.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2626835450 0 0) 0x7fcfe8003530 con 0x7fcff81537f0 2023-10-30T13:51:09.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 msgr2=0x7fcff815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:09.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 0x7fcff815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 >> v1:172.21.15.33:6789/0 conn(0x7fcff806b2f0 legacy=0x7fcff81641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80c1eb0 con 0x7fcff81537f0 2023-10-30T13:51:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fcfec001680 con 0x7fcff81537f0 2023-10-30T13:51:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 >> v1:172.21.15.31:6789/0 conn(0x7fcff81537f0 legacy=0x7fcff80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 0x7fcfe80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfe8003ef0 0x7fcfe80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcfe8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe800b9c0 con 0x7fcfe8004d10 2023-10-30T13:51:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe800bb30 con 0x7fcfe8003ef0 2023-10-30T13:51:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe800bcf0 con 0x7fcff80701a0 2023-10-30T13:51:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffd0db640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcfe8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffc8da640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 0x7fcfe80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcff7fff640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfe8003ef0 0x7fcfe80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffd0db640 1 -- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 msgr2=0x7fcfe80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffd0db640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff80701a0 0x7fcfe80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffd0db640 1 -- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfe8003ef0 msgr2=0x7fcfe80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffd0db640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfe8003ef0 0x7fcfe80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.904+0000 7fcffd0db640 1 -- 172.21.15.31:0/2952901963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe8003530 con 0x7fcfe8004d10 2023-10-30T13:51:09.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffd0db640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcfe8005190 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcff0001f60 tx=0x7fcff001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85db0496c4aaabd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:09.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00a5750 con 0x7fcfe8004d10 2023-10-30T13:51:09.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff00a58b0 con 0x7fcfe8004d10 2023-10-30T13:51:09.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcff77fe640 1 -- 172.21.15.31:0/2952901963 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00a5ca0 con 0x7fcfe8004d10 2023-10-30T13:51:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 -- 172.21.15.31:0/2952901963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 msgr2=0x7fcfe8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 --2- 172.21.15.31:0/2952901963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcfe8005190 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcff0001f60 tx=0x7fcff001c860 comp rx=0 tx=0).stop 2023-10-30T13:51:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffd0db640 1 -- 172.21.15.31:0/2952901963 reap_dead start 2023-10-30T13:51:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 -- 172.21.15.31:0/2952901963 shutdown_connections 2023-10-30T13:51:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 -- 172.21.15.31:0/2952901963 >> 172.21.15.31:0/2952901963 conn(0x7fcff8071fb0 msgr2=0x7fcff80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 -- 172.21.15.31:0/2952901963 shutdown_connections 2023-10-30T13:51:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 -- 172.21.15.31:0/2952901963 wait complete. 2023-10-30T13:51:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 Processor -- start 2023-10-30T13:51:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.908+0000 7fcffeb3e640 1 -- start start 2023-10-30T13:51:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcff81f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff815aa20 0x7fcff81566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcff8156bf0 0x7fcff8149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8155190 con 0x7fcfe8004d10 2023-10-30T13:51:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8156220 con 0x7fcff8156bf0 2023-10-30T13:51:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8157070 con 0x7fcff815aa20 2023-10-30T13:51:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcff81f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcff81f2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43806/0 (socket says 172.21.15.31:43806) 2023-10-30T13:51:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 -- 172.21.15.31:0/1399623427 learned_addr learned my addr 172.21.15.31:0/1399623427 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffd0db640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcff8156bf0 0x7fcff8149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcff7fff640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff815aa20 0x7fcff81566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 -- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff815aa20 msgr2=0x7fcff81566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff815aa20 0x7fcff81566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 -- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcff8156bf0 msgr2=0x7fcff8149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcff8156bf0 0x7fcff8149730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 -- 172.21.15.31:0/1399623427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8155610 con 0x7fcfe8004d10 2023-10-30T13:51:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffd0db640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcff8156bf0 0x7fcff8149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffc8da640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcff81f2280 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0072f0 tx=0x7fcfec0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcff57fa640 1 -- 172.21.15.31:0/1399623427 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec00f960 con 0x7fcfe8004d10 2023-10-30T13:51:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcff57fa640 1 -- 172.21.15.31:0/1399623427 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec00fae0 con 0x7fcfe8004d10 2023-10-30T13:51:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcff57fa640 1 -- 172.21.15.31:0/1399623427 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec00fe90 con 0x7fcfe8004d10 2023-10-30T13:51:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff81558f0 con 0x7fcfe8004d10 2023-10-30T13:51:09.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff8155eb0 con 0x7fcfe8004d10 2023-10-30T13:51:09.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff804e170 con 0x7fcfe8004d10 2023-10-30T13:51:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.912+0000 7fcff57fa640 1 -- 172.21.15.31:0/1399623427 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcfec01d360 con 0x7fcfe8004d10 2023-10-30T13:51:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.916+0000 7fcff57fa640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcfdc0664a0 0x7fcfdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.916+0000 7fcff57fa640 1 -- 172.21.15.31:0/1399623427 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fcfec083b10 con 0x7fcfe8004d10 2023-10-30T13:51:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.916+0000 7fcff7fff640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcfdc0664a0 0x7fcfdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.916+0000 7fcff7fff640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcfdc0664a0 0x7fcfdc068960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcff8151780 tx=0x7fcfe4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:09.920+0000 7fcff57fa640 1 -- 172.21.15.31:0/1399623427 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcfec050c00 con 0x7fcfe8004d10 2023-10-30T13:51:10.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcff80c9b20 con 0x7fcfe8004d10 2023-10-30T13:51:10.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcff57fa640 1 -- 172.21.15.31:0/1399623427 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6490 (secure 0 0 0) 0x7fcfec01a090 con 0x7fcfe8004d10 2023-10-30T13:51:10.114 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:10.114 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5591,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:10.114 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2023-10-30T13:51:10.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcfdc0664a0 msgr2=0x7fcfdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcfdc0664a0 0x7fcfdc068960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcff8151780 tx=0x7fcfe4008040 comp rx=0 tx=0).stop 2023-10-30T13:51:10.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 msgr2=0x7fcff81f2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcff81f2280 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0072f0 tx=0x7fcfec0077b0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 shutdown_connections 2023-10-30T13:51:10.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcfdc0664a0 0x7fcfdc068960 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcff8156bf0 0x7fcff8149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fcff815aa20 0x7fcff81566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 --2- 172.21.15.31:0/1399623427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfe8004d10 0x7fcff81f2280 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 >> 172.21.15.31:0/1399623427 conn(0x7fcff8071fb0 msgr2=0x7fcff80c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:10.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 shutdown_connections 2023-10-30T13:51:10.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:10.112+0000 7fcffeb3e640 1 -- 172.21.15.31:0/1399623427 wait complete. 2023-10-30T13:51:10.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:51:12.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:12.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.288+0000 7fe741d94640 1 Processor -- start 2023-10-30T13:51:12.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.288+0000 7fe741d94640 1 -- start start 2023-10-30T13:51:12.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe741d94640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe73c0c5ba0 0x7fe73c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe741d94640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5f80 0x7fe73c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe741d94640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe73c0c9b40 0x7fe73c06a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe741d94640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73c15d990 con 0x7fe73c0c5ba0 2023-10-30T13:51:12.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe741d94640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73c153f60 con 0x7fe73c0c9b40 2023-10-30T13:51:12.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe741d94640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c15cda0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe73c0c5ba0 0x7fe73c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe73c0c5ba0 0x7fe73c0c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43810/0 (socket says 172.21.15.31:43810) 2023-10-30T13:51:12.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 -- 172.21.15.31:0/951194802 learned_addr learned my addr 172.21.15.31:0/951194802 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:12.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73bfff640 1 --2- 172.21.15.31:0/951194802 >> v2:172.21.15.33:3300/0 conn(0x7fe73c0c9b40 0x7fe73c06a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73affd640 1 --2- 172.21.15.31:0/951194802 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5f80 0x7fe73c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 -- 172.21.15.31:0/951194802 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5f80 msgr2=0x7fe73c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:12.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 --2- 172.21.15.31:0/951194802 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5f80 0x7fe73c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 -- 172.21.15.31:0/951194802 >> v2:172.21.15.33:3300/0 conn(0x7fe73c0c9b40 msgr2=0x7fe73c06a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:12.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 --2- 172.21.15.31:0/951194802 >> v2:172.21.15.33:3300/0 conn(0x7fe73c0c9b40 0x7fe73c06a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.292+0000 7fe73b7fe640 1 -- 172.21.15.31:0/951194802 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c06f490 con 0x7fe73c0c5ba0 2023-10-30T13:51:12.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe73b7fe640 1 --2- 172.21.15.31:0/951194802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5ba0 0x7fe73c0c70f0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0c7630 tx=0x7fe72c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3502b8843735dd7a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe73a7fc640 1 -- 172.21.15.31:0/951194802 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c0ab600 con 0x7fe73c0c5ba0 2023-10-30T13:51:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe73a7fc640 1 -- 172.21.15.31:0/951194802 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c0a7050 con 0x7fe73c0c5ba0 2023-10-30T13:51:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe73a7fc640 1 -- 172.21.15.31:0/951194802 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c0ab980 con 0x7fe73c0c5ba0 2023-10-30T13:51:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 -- 172.21.15.31:0/951194802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5ba0 msgr2=0x7fe73c0c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 --2- 172.21.15.31:0/951194802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5ba0 0x7fe73c0c70f0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0c7630 tx=0x7fe72c01c470 comp rx=0 tx=0).stop 2023-10-30T13:51:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 -- 172.21.15.31:0/951194802 shutdown_connections 2023-10-30T13:51:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 --2- 172.21.15.31:0/951194802 >> v2:172.21.15.33:3300/0 conn(0x7fe73c0c9b40 0x7fe73c06a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 --2- 172.21.15.31:0/951194802 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5f80 0x7fe73c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 --2- 172.21.15.31:0/951194802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5ba0 0x7fe73c0c70f0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 -- 172.21.15.31:0/951194802 >> 172.21.15.31:0/951194802 conn(0x7fe73c06afa0 msgr2=0x7fe73c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 -- 172.21.15.31:0/951194802 shutdown_connections 2023-10-30T13:51:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 -- 172.21.15.31:0/951194802 wait complete. 2023-10-30T13:51:12.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 Processor -- start 2023-10-30T13:51:12.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.296+0000 7fe741d94640 1 -- start start 2023-10-30T13:51:12.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5ba0 0x7fe73c0c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5f80 0x7fe73c151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe73c0c9b40 0x7fe73c157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c0c5450 con 0x7fe73c0c5f80 2023-10-30T13:51:12.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c0c4320 con 0x7fe73c0c9b40 2023-10-30T13:51:12.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c0c48d0 con 0x7fe73c0c5ba0 2023-10-30T13:51:12.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5f80 0x7fe73c151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5f80 0x7fe73c151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43822/0 (socket says 172.21.15.31:43822) 2023-10-30T13:51:12.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 -- 172.21.15.31:0/2318950585 learned_addr learned my addr 172.21.15.31:0/2318950585 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:12.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73b7fe640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5ba0 0x7fe73c0c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73bfff640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe73c0c9b40 0x7fe73c157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 -- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5ba0 msgr2=0x7fe73c0c3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5ba0 0x7fe73c0c3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 -- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe73c0c9b40 msgr2=0x7fe73c157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe73c0c9b40 0x7fe73c157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 -- 172.21.15.31:0/2318950585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c0094d0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe73affd640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5f80 0x7fe73c151d80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe73000d800 tx=0x7fe73000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:12.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe723fff640 1 -- 172.21.15.31:0/2318950585 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7300096a0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe723fff640 1 -- 172.21.15.31:0/2318950585 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe730009800 con 0x7fe73c0c5f80 2023-10-30T13:51:12.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe723fff640 1 -- 172.21.15.31:0/2318950585 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730009990 con 0x7fe73c0c5f80 2023-10-30T13:51:12.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73c0c4bb0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73c0c5170 con 0x7fe73c0c5f80 2023-10-30T13:51:12.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73c0c3ab0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.300+0000 7fe723fff640 1 -- 172.21.15.31:0/2318950585 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe730009af0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.304+0000 7fe723fff640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7280664a0 0x7fe728068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.304+0000 7fe723fff640 1 -- 172.21.15.31:0/2318950585 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe730084650 con 0x7fe73c0c5f80 2023-10-30T13:51:12.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.304+0000 7fe73b7fe640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7280664a0 0x7fe728068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.304+0000 7fe73b7fe640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7280664a0 0x7fe728068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe72c0ad000 tx=0x7fe72c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:12.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.308+0000 7fe723fff640 1 -- 172.21.15.31:0/2318950585 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe73c0c3ab0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.500+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe73c0c3ab0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.500+0000 7fe723fff640 1 -- 172.21.15.31:0/2318950585 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6490 (secure 0 0 0) 0x7fe73c0c3ab0 con 0x7fe73c0c5f80 2023-10-30T13:51:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:12.503 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5591,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:12.504 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2023-10-30T13:51:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7280664a0 msgr2=0x7fe728068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7280664a0 0x7fe728068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe72c0ad000 tx=0x7fe72c008040 comp rx=0 tx=0).stop 2023-10-30T13:51:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5f80 msgr2=0x7fe73c151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5f80 0x7fe73c151d80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe73000d800 tx=0x7fe73000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 shutdown_connections 2023-10-30T13:51:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7280664a0 0x7fe728068960 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe73c0c9b40 0x7fe73c157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe73c0c5f80 0x7fe73c151d80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 --2- 172.21.15.31:0/2318950585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe73c0c5ba0 0x7fe73c0c3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 >> 172.21.15.31:0/2318950585 conn(0x7fe73c06afa0 msgr2=0x7fe73c0befb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 shutdown_connections 2023-10-30T13:51:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:12.504+0000 7fe741d94640 1 -- 172.21.15.31:0/2318950585 wait complete. 2023-10-30T13:51:12.524 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:51:14.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:14.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a9b5640 1 Processor -- start 2023-10-30T13:51:14.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a9b5640 1 -- start start 2023-10-30T13:51:14.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a9b5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9415ac70 0x7eff9415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:14.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a9b5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7eff9406aeb0 0x7eff94069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:14.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a9b5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9415dcc0 con 0x7eff940c8eb0 2023-10-30T13:51:14.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a9b5640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff94159800 con 0x7eff9406aeb0 2023-10-30T13:51:14.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a9b5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff94159d80 con 0x7eff9415ac70 2023-10-30T13:51:14.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a1b4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7eff9406aeb0 0x7eff94069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:14.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a1b4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7eff9406aeb0 0x7eff94069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43824/0 (socket says 172.21.15.31:43824) 2023-10-30T13:51:14.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a1b4640 1 -- 172.21.15.31:0/1064458645 learned_addr learned my addr 172.21.15.31:0/1064458645 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:14.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff991b2640 1 --2- 172.21.15.31:0/1064458645 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9415ac70 0x7eff9415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:14.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a1b4640 1 -- 172.21.15.31:0/1064458645 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9415ac70 msgr2=0x7eff9415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:14.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a1b4640 1 --2- 172.21.15.31:0/1064458645 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9415ac70 0x7eff9415cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a1b4640 1 -- 172.21.15.31:0/1064458645 >> v1:172.21.15.33:6789/0 conn(0x7eff940c8eb0 legacy=0x7eff9415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:14.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.680+0000 7eff9a1b4640 1 -- 172.21.15.31:0/1064458645 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9415d420 con 0x7eff9406aeb0 2023-10-30T13:51:14.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a1b4640 1 --2- 172.21.15.31:0/1064458645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9406aeb0 0x7eff94069e90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009e50 tx=0x7eff8c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85af2e914979fd95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:14.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff989b1640 1 -- 172.21.15.31:0/1064458645 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0af400 con 0x7eff9406aeb0 2023-10-30T13:51:14.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff989b1640 1 -- 172.21.15.31:0/1064458645 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c0ab050 con 0x7eff9406aeb0 2023-10-30T13:51:14.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff989b1640 1 -- 172.21.15.31:0/1064458645 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0af780 con 0x7eff9406aeb0 2023-10-30T13:51:14.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 -- 172.21.15.31:0/1064458645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9406aeb0 msgr2=0x7eff94069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:14.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/1064458645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9406aeb0 0x7eff94069e90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009e50 tx=0x7eff8c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 -- 172.21.15.31:0/1064458645 shutdown_connections 2023-10-30T13:51:14.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/1064458645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9406aeb0 0x7eff94069e90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/1064458645 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9415ac70 0x7eff9415cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 -- 172.21.15.31:0/1064458645 >> 172.21.15.31:0/1064458645 conn(0x7eff940cef00 msgr2=0x7eff940cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:14.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 -- 172.21.15.31:0/1064458645 shutdown_connections 2023-10-30T13:51:14.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 -- 172.21.15.31:0/1064458645 wait complete. 2023-10-30T13:51:14.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.684+0000 7eff9a9b5640 1 Processor -- start 2023-10-30T13:51:14.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 -- start start 2023-10-30T13:51:14.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9406aeb0 0x7eff940c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff940c8eb0 0x7eff940c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff9415ac70 0x7eff940c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9414e070 con 0x7eff940c8eb0 2023-10-30T13:51:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9414e280 con 0x7eff9415ac70 2023-10-30T13:51:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9414e560 con 0x7eff9406aeb0 2023-10-30T13:51:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff940c8eb0 0x7eff940c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff940c8eb0 0x7eff940c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43830/0 (socket says 172.21.15.31:43830) 2023-10-30T13:51:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 -- 172.21.15.31:0/4047752345 learned_addr learned my addr 172.21.15.31:0/4047752345 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff999b3640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9406aeb0 0x7eff940c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a1b4640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff9415ac70 0x7eff940c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:14.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 -- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9406aeb0 msgr2=0x7eff940c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:14.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9406aeb0 0x7eff940c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 -- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff9415ac70 msgr2=0x7eff940c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:14.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff9415ac70 0x7eff940c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 -- 172.21.15.31:0/4047752345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8c009b30 con 0x7eff940c8eb0 2023-10-30T13:51:14.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a1b4640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff9415ac70 0x7eff940c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:51:14.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff991b2640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff940c8eb0 0x7eff940c4530 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7eff8400d7d0 tx=0x7eff8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:14.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff8a7fc640 1 -- 172.21.15.31:0/4047752345 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400b6c0 con 0x7eff940c8eb0 2023-10-30T13:51:14.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff8a7fc640 1 -- 172.21.15.31:0/4047752345 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8400b820 con 0x7eff940c8eb0 2023-10-30T13:51:14.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff8a7fc640 1 -- 172.21.15.31:0/4047752345 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400bbb0 con 0x7eff940c8eb0 2023-10-30T13:51:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9414e840 con 0x7eff940c8eb0 2023-10-30T13:51:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9414ee00 con 0x7eff940c8eb0 2023-10-30T13:51:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.688+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff9414efc0 con 0x7eff940c8eb0 2023-10-30T13:51:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.692+0000 7eff8a7fc640 1 -- 172.21.15.31:0/4047752345 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff8400bd10 con 0x7eff940c8eb0 2023-10-30T13:51:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.692+0000 7eff8a7fc640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7eff780664f0 0x7eff780689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.692+0000 7eff8a7fc640 1 -- 172.21.15.31:0/4047752345 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7eff84084660 con 0x7eff940c8eb0 2023-10-30T13:51:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.692+0000 7eff999b3640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7eff780664f0 0x7eff780689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.692+0000 7eff999b3640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7eff780664f0 0x7eff780689b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7eff90000f70 tx=0x7eff90009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.696+0000 7eff8a7fc640 1 -- 172.21.15.31:0/4047752345 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff840517d0 con 0x7eff940c8eb0 2023-10-30T13:51:14.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.884+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff9404e170 con 0x7eff940c8eb0 2023-10-30T13:51:14.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.884+0000 7eff8a7fc640 1 -- 172.21.15.31:0/4047752345 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6490 (secure 0 0 0) 0x7eff84054df0 con 0x7eff940c8eb0 2023-10-30T13:51:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:14.887 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5591,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:14.887 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2023-10-30T13:51:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7eff780664f0 msgr2=0x7eff780689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7eff780664f0 0x7eff780689b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7eff90000f70 tx=0x7eff90009040 comp rx=0 tx=0).stop 2023-10-30T13:51:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff940c8eb0 msgr2=0x7eff940c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff940c8eb0 0x7eff940c4530 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7eff8400d7d0 tx=0x7eff8400dc90 comp rx=0 tx=0).stop 2023-10-30T13:51:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 shutdown_connections 2023-10-30T13:51:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7eff780664f0 0x7eff780689b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff9415ac70 0x7eff940c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff940c8eb0 0x7eff940c4530 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 --2- 172.21.15.31:0/4047752345 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7eff9406aeb0 0x7eff940c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 >> 172.21.15.31:0/4047752345 conn(0x7eff940cef00 msgr2=0x7eff941585c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 shutdown_connections 2023-10-30T13:51:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:14.888+0000 7eff9a9b5640 1 -- 172.21.15.31:0/4047752345 wait complete. 2023-10-30T13:51:14.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:51:16.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:17.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.064+0000 7ffb22a46640 1 Processor -- start 2023-10-30T13:51:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.064+0000 7ffb22a46640 1 -- start start 2023-10-30T13:51:17.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb22a46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffb1c06b760 0x7ffb1c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb22a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06bb40 0x7ffb1c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb22a46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ffb1c06c420 0x7ffb1c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb22a46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c071e00 con 0x7ffb1c06b760 2023-10-30T13:51:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb22a46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c070020 con 0x7ffb1c06c420 2023-10-30T13:51:17.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb22a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c15eb90 con 0x7ffb1c06bb40 2023-10-30T13:51:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffb1c06b760 0x7ffb1c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffb1c06b760 0x7ffb1c15e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41926/0 (socket says 172.21.15.31:41926) 2023-10-30T13:51:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 -- 172.21.15.31:0/834208008 learned_addr learned my addr 172.21.15.31:0/834208008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb20fe3640 1 --2- 172.21.15.31:0/834208008 >> v2:172.21.15.33:3300/0 conn(0x7ffb1c06c420 0x7ffb1c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1b7fe640 1 --2- 172.21.15.31:0/834208008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06bb40 0x7ffb1c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 -- 172.21.15.31:0/834208008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06bb40 msgr2=0x7ffb1c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 --2- 172.21.15.31:0/834208008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06bb40 0x7ffb1c157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 -- 172.21.15.31:0/834208008 >> v2:172.21.15.33:3300/0 conn(0x7ffb1c06c420 msgr2=0x7ffb1c157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 --2- 172.21.15.31:0/834208008 >> v2:172.21.15.33:3300/0 conn(0x7ffb1c06c420 0x7ffb1c157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 -- 172.21.15.31:0/834208008 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c070f10 con 0x7ffb1c06b760 2023-10-30T13:51:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.068+0000 7ffb1bfff640 1 --2- 172.21.15.31:0/834208008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06b760 0x7ffb1c15e650 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c15b7a0 tx=0x7ffb0c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b2644c06e79b9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb1affd640 1 -- 172.21.15.31:0/834208008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0aa5f0 con 0x7ffb1c06b760 2023-10-30T13:51:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb1affd640 1 -- 172.21.15.31:0/834208008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c0aa750 con 0x7ffb1c06b760 2023-10-30T13:51:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb1affd640 1 -- 172.21.15.31:0/834208008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0aaae0 con 0x7ffb1c06b760 2023-10-30T13:51:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- 172.21.15.31:0/834208008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06b760 msgr2=0x7ffb1c15e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 --2- 172.21.15.31:0/834208008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06b760 0x7ffb1c15e650 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c15b7a0 tx=0x7ffb0c01c4d0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- 172.21.15.31:0/834208008 shutdown_connections 2023-10-30T13:51:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 --2- 172.21.15.31:0/834208008 >> v2:172.21.15.33:3300/0 conn(0x7ffb1c06c420 0x7ffb1c157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 --2- 172.21.15.31:0/834208008 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06bb40 0x7ffb1c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 --2- 172.21.15.31:0/834208008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06b760 0x7ffb1c15e650 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- 172.21.15.31:0/834208008 >> 172.21.15.31:0/834208008 conn(0x7ffb1c0cac70 msgr2=0x7ffb1c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- 172.21.15.31:0/834208008 shutdown_connections 2023-10-30T13:51:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- 172.21.15.31:0/834208008 wait complete. 2023-10-30T13:51:17.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 Processor -- start 2023-10-30T13:51:17.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- start start 2023-10-30T13:51:17.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c06b760 0x7ffb1c159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06bb40 0x7ffb1c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06c420 0x7ffb1c158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0c4c80 con 0x7ffb1c06bb40 2023-10-30T13:51:17.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1f0400 con 0x7ffb1c06b760 2023-10-30T13:51:17.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb22a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0c4730 con 0x7ffb1c06c420 2023-10-30T13:51:17.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.072+0000 7ffb1b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06bb40 0x7ffb1c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06bb40 0x7ffb1c15a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41938/0 (socket says 172.21.15.31:41938) 2023-10-30T13:51:17.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 -- 172.21.15.31:0/228628914 learned_addr learned my addr 172.21.15.31:0/228628914 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:17.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1bfff640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c06b760 0x7ffb1c159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb20fe3640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06c420 0x7ffb1c158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 -- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06c420 msgr2=0x7ffb1c158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06c420 0x7ffb1c158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 -- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c06b760 msgr2=0x7ffb1c159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c06b760 0x7ffb1c159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 -- 172.21.15.31:0/228628914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0094d0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb1b7fe640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06bb40 0x7ffb1c15a260 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000d7d0 tx=0x7ffb1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:17.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb18ff9640 1 -- 172.21.15.31:0/228628914 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10004020 con 0x7ffb1c06bb40 2023-10-30T13:51:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb18ff9640 1 -- 172.21.15.31:0/228628914 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10004180 con 0x7ffb1c06bb40 2023-10-30T13:51:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb18ff9640 1 -- 172.21.15.31:0/228628914 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10004330 con 0x7ffb1c06bb40 2023-10-30T13:51:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c0c49a0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c1f0090 con 0x7ffb1c06bb40 2023-10-30T13:51:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c158ac0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb18ff9640 1 -- 172.21.15.31:0/228628914 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb100096a0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb18ff9640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb080664a0 0x7ffb08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.076+0000 7ffb18ff9640 1 -- 172.21.15.31:0/228628914 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ffb100844a0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.080+0000 7ffb1bfff640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb080664a0 0x7ffb08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.080+0000 7ffb1bfff640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb080664a0 0x7ffb08068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0ae000 tx=0x7ffb0c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:17.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.084+0000 7ffb18ff9640 1 -- 172.21.15.31:0/228628914 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb1c158ac0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.272+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb1c158ac0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.272+0000 7ffb18ff9640 1 -- 172.21.15.31:0/228628914 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6490 (secure 0 0 0) 0x7ffb1c158ac0 con 0x7ffb1c06bb40 2023-10-30T13:51:17.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:17.275 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5591,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2023-10-30T13:51:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb080664a0 msgr2=0x7ffb08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb080664a0 0x7ffb08068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0ae000 tx=0x7ffb0c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:51:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06bb40 msgr2=0x7ffb1c15a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06bb40 0x7ffb1c15a260 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000d7d0 tx=0x7ffb1000dc90 comp rx=0 tx=0).stop 2023-10-30T13:51:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 shutdown_connections 2023-10-30T13:51:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffb080664a0 0x7ffb08068960 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffb1c06c420 0x7ffb1c158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1c06bb40 0x7ffb1c15a260 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 --2- 172.21.15.31:0/228628914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c06b760 0x7ffb1c159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 >> 172.21.15.31:0/228628914 conn(0x7ffb1c0cac70 msgr2=0x7ffb1c0736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:17.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 shutdown_connections 2023-10-30T13:51:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:17.276+0000 7ffb22a46640 1 -- 172.21.15.31:0/228628914 wait complete. 2023-10-30T13:51:17.297 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:51:19.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:19.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc343e46640 1 Processor -- start 2023-10-30T13:51:19.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc343e46640 1 -- start start 2023-10-30T13:51:19.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc343e46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc33c06b760 0x7fc33c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc343e46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06bb40 0x7fc33c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc343e46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c071e00 con 0x7fc33c06c420 2023-10-30T13:51:19.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc343e46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33c070020 con 0x7fc33c06b760 2023-10-30T13:51:19.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc343e46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c15eb90 con 0x7fc33c06bb40 2023-10-30T13:51:19.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc341be2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc33c06b760 0x7fc33c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc341be2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc33c06b760 0x7fc33c15e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41954/0 (socket says 172.21.15.31:41954) 2023-10-30T13:51:19.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc341be2640 1 -- 172.21.15.31:0/2571562023 learned_addr learned my addr 172.21.15.31:0/2571562023 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:19.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc3413e1640 1 --2- 172.21.15.31:0/2571562023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06bb40 0x7fc33c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc341be2640 1 -- 172.21.15.31:0/2571562023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06bb40 msgr2=0x7fc33c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:19.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc341be2640 1 --2- 172.21.15.31:0/2571562023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06bb40 0x7fc33c157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.452+0000 7fc341be2640 1 -- 172.21.15.31:0/2571562023 >> v1:172.21.15.33:6789/0 conn(0x7fc33c06c420 legacy=0x7fc33c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc341be2640 1 -- 172.21.15.31:0/2571562023 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c070f10 con 0x7fc33c06b760 2023-10-30T13:51:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc341be2640 1 --2- 172.21.15.31:0/2571562023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06b760 0x7fc33c15e650 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc338009870 tx=0x7fc33801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c89864aff9d266 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc340be0640 1 -- 172.21.15.31:0/2571562023 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3380aa670 con 0x7fc33c06b760 2023-10-30T13:51:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc340be0640 1 -- 172.21.15.31:0/2571562023 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3380aa7d0 con 0x7fc33c06b760 2023-10-30T13:51:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc340be0640 1 -- 172.21.15.31:0/2571562023 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3380aab60 con 0x7fc33c06b760 2023-10-30T13:51:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc343e46640 1 -- 172.21.15.31:0/2571562023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06b760 msgr2=0x7fc33c15e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc343e46640 1 --2- 172.21.15.31:0/2571562023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06b760 0x7fc33c15e650 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc338009870 tx=0x7fc33801c510 comp rx=0 tx=0).stop 2023-10-30T13:51:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc343e46640 1 -- 172.21.15.31:0/2571562023 shutdown_connections 2023-10-30T13:51:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc343e46640 1 --2- 172.21.15.31:0/2571562023 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06bb40 0x7fc33c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc343e46640 1 --2- 172.21.15.31:0/2571562023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06b760 0x7fc33c15e650 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.456+0000 7fc343e46640 1 -- 172.21.15.31:0/2571562023 >> 172.21.15.31:0/2571562023 conn(0x7fc33c0cac70 msgr2=0x7fc33c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 -- 172.21.15.31:0/2571562023 shutdown_connections 2023-10-30T13:51:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 -- 172.21.15.31:0/2571562023 wait complete. 2023-10-30T13:51:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 Processor -- start 2023-10-30T13:51:19.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 -- start start 2023-10-30T13:51:19.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06b760 0x7fc33c14b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc33c06bb40 0x7fc33c14e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06c420 0x7fc33c14d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c1f6dd0 con 0x7fc33c06c420 2023-10-30T13:51:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c15bfc0 con 0x7fc33c06bb40 2023-10-30T13:51:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc343e46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c1f5f40 con 0x7fc33c06b760 2023-10-30T13:51:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06c420 0x7fc33c14d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3413e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc33c06bb40 0x7fc33c14e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06c420 0x7fc33c14d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41956/0 (socket says 172.21.15.31:41956) 2023-10-30T13:51:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 -- 172.21.15.31:0/3482720052 learned_addr learned my addr 172.21.15.31:0/3482720052 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc341be2640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06b760 0x7fc33c14b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 -- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06b760 msgr2=0x7fc33c14b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06b760 0x7fc33c14b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 -- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc33c06bb40 msgr2=0x7fc33c14e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc33c06bb40 0x7fc33c14e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 -- 172.21.15.31:0/3482720052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3380094d0 con 0x7fc33c06c420 2023-10-30T13:51:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3423e3640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06c420 0x7fc33c14d440 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c760 tx=0x7fc33400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3327fc640 1 -- 172.21.15.31:0/3482720052 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334003b80 con 0x7fc33c06c420 2023-10-30T13:51:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3327fc640 1 -- 172.21.15.31:0/3482720052 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc334003ce0 con 0x7fc33c06c420 2023-10-30T13:51:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.460+0000 7fc3327fc640 1 -- 172.21.15.31:0/3482720052 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33400aa90 con 0x7fc33c06c420 2023-10-30T13:51:19.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.464+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33c1f6150 con 0x7fc33c06c420 2023-10-30T13:51:19.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.464+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c1f6710 con 0x7fc33c06c420 2023-10-30T13:51:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.464+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c04e170 con 0x7fc33c06c420 2023-10-30T13:51:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.464+0000 7fc3327fc640 1 -- 172.21.15.31:0/3482720052 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc33400abf0 con 0x7fc33c06c420 2023-10-30T13:51:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.464+0000 7fc3327fc640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3200664d0 0x7fc320068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.464+0000 7fc3327fc640 1 -- 172.21.15.31:0/3482720052 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc334083e30 con 0x7fc33c06c420 2023-10-30T13:51:19.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.464+0000 7fc341be2640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3200664d0 0x7fc320068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.468+0000 7fc341be2640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3200664d0 0x7fc320068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc3380ae000 tx=0x7fc3380b5040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.468+0000 7fc3327fc640 1 -- 172.21.15.31:0/3482720052 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc334050fd0 con 0x7fc33c06c420 2023-10-30T13:51:19.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.656+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc33c072270 con 0x7fc33c06c420 2023-10-30T13:51:19.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.656+0000 7fc3327fc640 1 -- 172.21.15.31:0/3482720052 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6490 (secure 0 0 0) 0x7fc3340545f0 con 0x7fc33c06c420 2023-10-30T13:51:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:19.660 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5591,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:50:02.763196+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5889},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5889":{"gid":5889,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/327248983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":327248983},{"type":"v1","addr":"172.21.15.31:6837","nonce":327248983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:19.660 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2023-10-30T13:51:19.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3200664d0 msgr2=0x7fc320068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3200664d0 0x7fc320068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc3380ae000 tx=0x7fc3380b5040 comp rx=0 tx=0).stop 2023-10-30T13:51:19.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06c420 msgr2=0x7fc33c14d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06c420 0x7fc33c14d440 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c760 tx=0x7fc33400cc20 comp rx=0 tx=0).stop 2023-10-30T13:51:19.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 shutdown_connections 2023-10-30T13:51:19.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3200664d0 0x7fc320068990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc33c06c420 0x7fc33c14d440 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc33c06bb40 0x7fc33c14e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 --2- 172.21.15.31:0/3482720052 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc33c06b760 0x7fc33c14b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 >> 172.21.15.31:0/3482720052 conn(0x7fc33c0cac70 msgr2=0x7fc33c0c62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 shutdown_connections 2023-10-30T13:51:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:19.660+0000 7fc343e46640 1 -- 172.21.15.31:0/3482720052 wait complete. 2023-10-30T13:51:19.681 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:51:21.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:21.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dc1646640 1 Processor -- start 2023-10-30T13:51:21.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dc1646640 1 -- start start 2023-10-30T13:51:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dc1646640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7dbc06ba00 0x7f7dbc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dc1646640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc0c2aa0 con 0x7f7dbc0701a0 2023-10-30T13:51:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dc1646640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc0708e0 con 0x7f7dbc1537f0 2023-10-30T13:51:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dc1646640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0c3ab0 con 0x7f7dbc06ba00 2023-10-30T13:51:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dba7fc640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7dbc0701a0 0x7f7dbc06b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46180/0 (socket says 172.21.15.31:46180) 2023-10-30T13:51:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dba7fc640 1 -- 172.21.15.31:0/824845019 learned_addr learned my addr 172.21.15.31:0/824845019 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7dbb7fe640 1 --2- 172.21.15.31:0/824845019 >> v2:172.21.15.33:3300/0 conn(0x7f7dbc06ba00 0x7f7dbc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277975587 0 0) 0x7f7dbc0c2aa0 con 0x7f7dbc0701a0 2023-10-30T13:51:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da4003530 con 0x7f7dbc0701a0 2023-10-30T13:51:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205539480 0 0) 0x7f7dbc0708e0 con 0x7f7dbc1537f0 2023-10-30T13:51:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dbc0c2aa0 con 0x7f7dbc1537f0 2023-10-30T13:51:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496088204 0 0) 0x7f7da4003530 con 0x7f7dbc0701a0 2023-10-30T13:51:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 >> v2:172.21.15.33:3300/0 conn(0x7f7dbc06ba00 msgr2=0x7f7dbc15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 --2- 172.21.15.31:0/824845019 >> v2:172.21.15.33:3300/0 conn(0x7f7dbc06ba00 0x7f7dbc15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 >> v1:172.21.15.33:6789/0 conn(0x7f7dbc1537f0 legacy=0x7f7dbc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0c1eb0 con 0x7f7dbc0701a0 2023-10-30T13:51:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f7db0001680 con 0x7f7dbc0701a0 2023-10-30T13:51:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 >> v1:172.21.15.31:6789/0 conn(0x7f7dbc0701a0 legacy=0x7f7dbc06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4003b10 0x7f7da4003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7da40047d0 0x7f7da4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7da4005190 0x7f7da400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da400bf80 con 0x7f7da4005190 2023-10-30T13:51:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da400c0c0 con 0x7f7da40047d0 2023-10-30T13:51:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.836+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da400c280 con 0x7f7da4003b10 2023-10-30T13:51:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7da4005190 0x7f7da400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dba7fc640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7da40047d0 0x7f7da4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbaffd640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4003b10 0x7f7da4003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 -- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4003b10 msgr2=0x7f7da4003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4003b10 0x7f7da4003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 -- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7da40047d0 msgr2=0x7f7da4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7da40047d0 0x7f7da4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 -- 172.21.15.31:0/824845019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da4003530 con 0x7f7da4005190 2023-10-30T13:51:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 -- 172.21.15.31:0/824845019 reap_dead start 2023-10-30T13:51:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7dbb7fe640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7da4005190 0x7f7da400b990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009f90 tx=0x7f7db401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3f73274049e1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40a5c70 con 0x7f7da4005190 2023-10-30T13:51:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400f040 con 0x7f7da4005190 2023-10-30T13:51:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.840+0000 7f7db9ffb640 1 -- 172.21.15.31:0/824845019 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40039a0 con 0x7f7da4005190 2023-10-30T13:51:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- 172.21.15.31:0/824845019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7da4005190 msgr2=0x7f7da400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7da4005190 0x7f7da400b990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009f90 tx=0x7f7db401cc60 comp rx=0 tx=0).stop 2023-10-30T13:51:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- 172.21.15.31:0/824845019 shutdown_connections 2023-10-30T13:51:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 --2- 172.21.15.31:0/824845019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7da4005190 0x7f7da400b990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- 172.21.15.31:0/824845019 >> 172.21.15.31:0/824845019 conn(0x7f7dbc071fb0 msgr2=0x7f7dbc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- 172.21.15.31:0/824845019 shutdown_connections 2023-10-30T13:51:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- 172.21.15.31:0/824845019 wait complete. 2023-10-30T13:51:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 Processor -- start 2023-10-30T13:51:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- start start 2023-10-30T13:51:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4005190 0x7f7dbc158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7dbc154150 0x7f7dbc14ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7dbc1ef0f0 0x7f7dbc146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0c9840 con 0x7f7dbc154150 2023-10-30T13:51:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0b2c10 con 0x7f7dbc1ef0f0 2023-10-30T13:51:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dc1646640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0b2e70 con 0x7f7da4005190 2023-10-30T13:51:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7dbc154150 0x7f7dbc14ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7dbc154150 0x7f7dbc14ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41964/0 (socket says 172.21.15.31:41964) 2023-10-30T13:51:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 -- 172.21.15.31:0/1585156167 learned_addr learned my addr 172.21.15.31:0/1585156167 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dbaffd640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4005190 0x7f7dbc158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dbb7fe640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7dbc1ef0f0 0x7f7dbc146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 -- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4005190 msgr2=0x7f7dbc158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4005190 0x7f7dbc158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 -- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7dbc1ef0f0 msgr2=0x7f7dbc146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7dbc1ef0f0 0x7f7dbc146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 -- 172.21.15.31:0/1585156167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0ca1c0 con 0x7f7dbc154150 2023-10-30T13:51:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.844+0000 7f7dba7fc640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7dbc154150 0x7f7dbc14ed50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7db000e780 tx=0x7f7db000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7da37fe640 1 -- 172.21.15.31:0/1585156167 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000ee90 con 0x7f7dbc154150 2023-10-30T13:51:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7da37fe640 1 -- 172.21.15.31:0/1585156167 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db000c8d0 con 0x7f7dbc154150 2023-10-30T13:51:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc158550 con 0x7f7dbc154150 2023-10-30T13:51:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7da37fe640 1 -- 172.21.15.31:0/1585156167 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000cc80 con 0x7f7dbc154150 2023-10-30T13:51:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc0c9a50 con 0x7f7dbc154150 2023-10-30T13:51:21.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc157dd0 con 0x7f7dbc154150 2023-10-30T13:51:21.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7da37fe640 1 -- 172.21.15.31:0/1585156167 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7db001d460 con 0x7f7dbc154150 2023-10-30T13:51:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7da37fe640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7da80664a0 0x7f7da8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7da37fe640 1 -- 172.21.15.31:0/1585156167 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f7db0084080 con 0x7f7dbc154150 2023-10-30T13:51:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.848+0000 7f7dbaffd640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7da80664a0 0x7f7da8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.852+0000 7f7dbaffd640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7da80664a0 0x7f7da8068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7dac007860 tx=0x7f7dac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:21.852+0000 7f7da37fe640 1 -- 172.21.15.31:0/1585156167 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7db00b5090 con 0x7f7dbc154150 2023-10-30T13:51:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dbc0c2700 con 0x7f7dbc154150 2023-10-30T13:51:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7da37fe640 1 -- 172.21.15.31:0/1585156167 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f7db0054c90 con 0x7f7dbc154150 2023-10-30T13:51:22.050 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:22.050 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:22.020359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:22.050 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2023-10-30T13:51:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7da80664a0 msgr2=0x7f7da8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7da80664a0 0x7f7da8068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7dac007860 tx=0x7f7dac009040 comp rx=0 tx=0).stop 2023-10-30T13:51:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7dbc154150 msgr2=0x7f7dbc14ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7dbc154150 0x7f7dbc14ed50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7db000e780 tx=0x7f7db000ec40 comp rx=0 tx=0).stop 2023-10-30T13:51:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 shutdown_connections 2023-10-30T13:51:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7da80664a0 0x7f7da8068960 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7dbc1ef0f0 0x7f7dbc146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7dbc154150 0x7f7dbc14ed50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.048+0000 7f7dc1646640 1 --2- 172.21.15.31:0/1585156167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7da4005190 0x7f7dbc158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.052+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 >> 172.21.15.31:0/1585156167 conn(0x7f7dbc071fb0 msgr2=0x7f7dbc1515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.052+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 shutdown_connections 2023-10-30T13:51:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.052+0000 7f7dc1646640 1 -- 172.21.15.31:0/1585156167 wait complete. 2023-10-30T13:51:22.071 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:51:22.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:51:22.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.212+0000 7f7b46d2a640 1 Processor -- start 2023-10-30T13:51:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.212+0000 7f7b46d2a640 1 -- start start 2023-10-30T13:51:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b46d2a640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7b4006c420 0x7f7b400c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b46d2a640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7b400c7170 0x7f7b40159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b46d2a640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400c5f80 con 0x7f7b40153bb0 2023-10-30T13:51:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b46d2a640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b400c7580 con 0x7f7b4006c420 2023-10-30T13:51:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b46d2a640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b400c5690 con 0x7f7b400c7170 2023-10-30T13:51:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7b4006c420 0x7f7b400c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7b4006c420 0x7f7b400c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41976/0 (socket says 172.21.15.31:41976) 2023-10-30T13:51:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b44ac6640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7b40153bb0 0x7f7b400c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46186/0 (socket says 172.21.15.31:46186) 2023-10-30T13:51:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3ffff640 1 -- 172.21.15.31:0/1304647266 learned_addr learned my addr 172.21.15.31:0/1304647266 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b452c7640 1 --2- 172.21.15.31:0/1304647266 >> v2:172.21.15.33:3300/0 conn(0x7f7b400c7170 0x7f7b40159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3ffff640 1 -- 172.21.15.31:0/1304647266 >> v2:172.21.15.33:3300/0 conn(0x7f7b400c7170 msgr2=0x7f7b40159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3ffff640 1 --2- 172.21.15.31:0/1304647266 >> v2:172.21.15.33:3300/0 conn(0x7f7b400c7170 0x7f7b40159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3ffff640 1 -- 172.21.15.31:0/1304647266 >> v1:172.21.15.31:6789/0 conn(0x7f7b40153bb0 legacy=0x7f7b400c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3ffff640 1 -- 172.21.15.31:0/1304647266 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4015a580 con 0x7f7b4006c420 2023-10-30T13:51:22.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b3f7fe640 1 -- 172.21.15.31:0/1304647266 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694548594 0 0) 0x7f7b400c5f80 con 0x7f7b40153bb0 2023-10-30T13:51:22.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.216+0000 7f7b452c7640 1 --2- 172.21.15.31:0/1304647266 >> v2:172.21.15.33:3300/0 conn(0x7f7b400c7170 0x7f7b40159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:22.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b3ffff640 1 --2- 172.21.15.31:0/1304647266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b4006c420 0x7f7b400c6c30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7b400c8b20 tx=0x7f7b2c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4686cec45ec75f98 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b3f7fe640 1 -- 172.21.15.31:0/1304647266 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0a89f0 con 0x7f7b4006c420 2023-10-30T13:51:22.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b3f7fe640 1 -- 172.21.15.31:0/1304647266 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c0a8b50 con 0x7f7b4006c420 2023-10-30T13:51:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b3f7fe640 1 -- 172.21.15.31:0/1304647266 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0b0440 con 0x7f7b4006c420 2023-10-30T13:51:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 -- 172.21.15.31:0/1304647266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b4006c420 msgr2=0x7f7b400c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/1304647266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b4006c420 0x7f7b400c6c30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7b400c8b20 tx=0x7f7b2c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 -- 172.21.15.31:0/1304647266 shutdown_connections 2023-10-30T13:51:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/1304647266 >> v2:172.21.15.33:3300/0 conn(0x7f7b400c7170 0x7f7b40159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/1304647266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b4006c420 0x7f7b400c6c30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 -- 172.21.15.31:0/1304647266 >> 172.21.15.31:0/1304647266 conn(0x7f7b400cac70 msgr2=0x7f7b400cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:22.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 -- 172.21.15.31:0/1304647266 shutdown_connections 2023-10-30T13:51:22.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 -- 172.21.15.31:0/1304647266 wait complete. 2023-10-30T13:51:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 Processor -- start 2023-10-30T13:51:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.220+0000 7f7b46d2a640 1 -- start start 2023-10-30T13:51:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7b4006c420 0x7f7b401e2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b400c7170 0x7f7b400c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b40153bb0 0x7f7b400c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400c2bd0 con 0x7f7b40153bb0 2023-10-30T13:51:22.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b40158130 con 0x7f7b400c7170 2023-10-30T13:51:22.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b401ef650 con 0x7f7b4006c420 2023-10-30T13:51:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b40153bb0 0x7f7b400c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b40153bb0 0x7f7b400c2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41986/0 (socket says 172.21.15.31:41986) 2023-10-30T13:51:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 -- 172.21.15.31:0/3518920433 learned_addr learned my addr 172.21.15.31:0/3518920433 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b3ffff640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b400c7170 0x7f7b400c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b44ac6640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7b4006c420 0x7f7b401e2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 -- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7b4006c420 msgr2=0x7f7b401e2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7b4006c420 0x7f7b401e2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 -- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b400c7170 msgr2=0x7f7b400c40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b400c7170 0x7f7b400c40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 -- 172.21.15.31:0/3518920433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0094d0 con 0x7f7b40153bb0 2023-10-30T13:51:22.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b452c7640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b40153bb0 0x7f7b400c2ff0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7b38007df0 tx=0x7f7b3800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b3d7fa640 1 -- 172.21.15.31:0/3518920433 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b380039a0 con 0x7f7b40153bb0 2023-10-30T13:51:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b3d7fa640 1 -- 172.21.15.31:0/3518920433 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3800f040 con 0x7f7b40153bb0 2023-10-30T13:51:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b3d7fa640 1 -- 172.21.15.31:0/3518920433 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38003d40 con 0x7f7b40153bb0 2023-10-30T13:51:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b40157890 con 0x7f7b40153bb0 2023-10-30T13:51:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b40157e50 con 0x7f7b40153bb0 2023-10-30T13:51:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b400c2d10 con 0x7f7b40153bb0 2023-10-30T13:51:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.224+0000 7f7b3d7fa640 1 -- 172.21.15.31:0/3518920433 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b38004030 con 0x7f7b40153bb0 2023-10-30T13:51:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.228+0000 7f7b3d7fa640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7b240664a0 0x7f7b24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.228+0000 7f7b3d7fa640 1 -- 172.21.15.31:0/3518920433 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f7b38084520 con 0x7f7b40153bb0 2023-10-30T13:51:22.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.228+0000 7f7b44ac6640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7b240664a0 0x7f7b24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.228+0000 7f7b44ac6640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7b240664a0 0x7f7b24068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7b34007fd0 tx=0x7f7b34007d60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.232+0000 7f7b3d7fa640 1 -- 172.21.15.31:0/3518920433 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7b400c2d10 con 0x7f7b40153bb0 2023-10-30T13:51:22.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b400c2d10 con 0x7f7b40153bb0 2023-10-30T13:51:22.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b3d7fa640 1 -- 172.21.15.31:0/3518920433 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f7b400c2d10 con 0x7f7b40153bb0 2023-10-30T13:51:22.425 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:22.425 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:22.020359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:22.425 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2023-10-30T13:51:22.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7b240664a0 msgr2=0x7f7b24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7b240664a0 0x7f7b24068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7b34007fd0 tx=0x7f7b34007d60 comp rx=0 tx=0).stop 2023-10-30T13:51:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b40153bb0 msgr2=0x7f7b400c2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b40153bb0 0x7f7b400c2ff0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7b38007df0 tx=0x7f7b3800d4a0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 shutdown_connections 2023-10-30T13:51:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7b240664a0 0x7f7b24068960 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b40153bb0 0x7f7b400c2ff0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b400c7170 0x7f7b400c40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 --2- 172.21.15.31:0/3518920433 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7b4006c420 0x7f7b401e2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 >> 172.21.15.31:0/3518920433 conn(0x7f7b400cac70 msgr2=0x7f7b40152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:22.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 shutdown_connections 2023-10-30T13:51:22.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:22.424+0000 7f7b46d2a640 1 -- 172.21.15.31:0/3518920433 wait complete. 2023-10-30T13:51:22.446 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5591": { "addr": "172.21.15.33:6836/3928996203", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3928996203, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3928996203, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5591, "incarnation": 72, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:51:22.020359+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5591 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:51:24.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:24.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f94946d6640 1 Processor -- start 2023-10-30T13:51:24.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f94946d6640 1 -- start start 2023-10-30T13:51:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f94946d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f948c070520 0x7f948c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f94946d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948c0c2310 con 0x7f948c0cb7d0 2023-10-30T13:51:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f94946d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948c156710 con 0x7f948c150830 2023-10-30T13:51:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f94946d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948c06bc10 con 0x7f948c070520 2023-10-30T13:51:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f9491c71640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f948c0cb7d0 0x7f948c0c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:46196/0 (socket says 172.21.15.31:46196) 2023-10-30T13:51:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.604+0000 7f9491c71640 1 -- 172.21.15.31:0/4077930088 learned_addr learned my addr 172.21.15.31:0/4077930088 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9492c73640 1 --2- 172.21.15.31:0/4077930088 >> v2:172.21.15.33:3300/0 conn(0x7f948c070520 0x7f948c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192093282 0 0) 0x7f948c0c2310 con 0x7f948c0cb7d0 2023-10-30T13:51:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9478003530 con 0x7f948c0cb7d0 2023-10-30T13:51:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1035627039 0 0) 0x7f9478003530 con 0x7f948c0cb7d0 2023-10-30T13:51:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 >> v2:172.21.15.33:3300/0 conn(0x7f948c070520 msgr2=0x7f948c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 --2- 172.21.15.31:0/4077930088 >> v2:172.21.15.33:3300/0 conn(0x7f948c070520 0x7f948c070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 >> v1:172.21.15.33:6789/0 conn(0x7f948c150830 legacy=0x7f948c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c0c4360 con 0x7f948c0cb7d0 2023-10-30T13:51:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9492c73640 1 --2- 172.21.15.31:0/4077930088 >> v2:172.21.15.33:3300/0 conn(0x7f948c070520 0x7f948c070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:51:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f947c0016a0 con 0x7f948c0cb7d0 2023-10-30T13:51:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 >> v1:172.21.15.31:6789/0 conn(0x7f948c0cb7d0 legacy=0x7f948c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9478003980 0x7f9478003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780047d0 0x7f9478004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478005190 0x7f947800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c06bc10 con 0x7f94780047d0 2023-10-30T13:51:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947800c040 con 0x7f9478005190 2023-10-30T13:51:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947800c200 con 0x7f9478003980 2023-10-30T13:51:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491c71640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780047d0 0x7f9478004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9492c73640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478005190 0x7f947800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9492472640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9478003980 0x7f9478003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491c71640 1 -- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9478003980 msgr2=0x7f9478003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491c71640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9478003980 0x7f9478003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491c71640 1 -- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478005190 msgr2=0x7f947800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491c71640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478005190 0x7f947800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9491c71640 1 -- 172.21.15.31:0/4077930088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9478003530 con 0x7f94780047d0 2023-10-30T13:51:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9492c73640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478005190 0x7f947800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9492472640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f9478003980 0x7f9478003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:24.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.608+0000 7f9492c73640 1 -- 172.21.15.31:0/4077930088 reap_dead start 2023-10-30T13:51:24.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f9491c71640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780047d0 0x7f9478004c50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f948c0c24a0 tx=0x7f947c01cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3b07e04a1a9224 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947c005ae0 con 0x7f94780047d0 2023-10-30T13:51:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947c005c40 con 0x7f94780047d0 2023-10-30T13:51:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f9491470640 1 -- 172.21.15.31:0/4077930088 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947c0a58c0 con 0x7f94780047d0 2023-10-30T13:51:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 -- 172.21.15.31:0/4077930088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780047d0 msgr2=0x7f9478004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780047d0 0x7f9478004c50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f948c0c24a0 tx=0x7f947c01cbc0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 -- 172.21.15.31:0/4077930088 shutdown_connections 2023-10-30T13:51:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 --2- 172.21.15.31:0/4077930088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780047d0 0x7f9478004c50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 -- 172.21.15.31:0/4077930088 >> 172.21.15.31:0/4077930088 conn(0x7f948c0cac70 msgr2=0x7f948c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 -- 172.21.15.31:0/4077930088 shutdown_connections 2023-10-30T13:51:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 -- 172.21.15.31:0/4077930088 wait complete. 2023-10-30T13:51:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 Processor -- start 2023-10-30T13:51:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.612+0000 7f94946d6640 1 -- start start 2023-10-30T13:51:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94780047d0 0x7f948c1415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f948c14ef20 0x7f948c0c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f948c14dd80 0x7f948c0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c14e4a0 con 0x7f948c14dd80 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c14f9c0 con 0x7f94780047d0 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948c14ea20 con 0x7f948c14ef20 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f948c14dd80 0x7f948c0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f948c14dd80 0x7f948c0ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42004/0 (socket says 172.21.15.31:42004) 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 -- 172.21.15.31:0/3742683791 learned_addr learned my addr 172.21.15.31:0/3742683791 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9491c71640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f948c14ef20 0x7f948c0c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492472640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94780047d0 0x7f948c1415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 -- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f948c14ef20 msgr2=0x7f948c0c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f948c14ef20 0x7f948c0c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 -- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94780047d0 msgr2=0x7f948c1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94780047d0 0x7f948c1415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 -- 172.21.15.31:0/3742683791 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c141050 con 0x7f948c14dd80 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9492c73640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f948c14dd80 0x7f948c0ca0b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f948c1541f0 tx=0x7f9484009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9482ffd640 1 -- 172.21.15.31:0/3742683791 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948400e070 con 0x7f948c14dd80 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9482ffd640 1 -- 172.21.15.31:0/3742683791 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9484011e60 con 0x7f948c14dd80 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9482ffd640 1 -- 172.21.15.31:0/3742683791 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94840039a0 con 0x7f948c14dd80 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948c14fc10 con 0x7f948c14dd80 2023-10-30T13:51:24.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948c14e6b0 con 0x7f948c14dd80 2023-10-30T13:51:24.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948c04e170 con 0x7f948c14dd80 2023-10-30T13:51:24.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.616+0000 7f9482ffd640 1 -- 172.21.15.31:0/3742683791 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9484003b00 con 0x7f948c14dd80 2023-10-30T13:51:24.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.620+0000 7f9482ffd640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94700664d0 0x7f9470068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.620+0000 7f9482ffd640 1 -- 172.21.15.31:0/3742683791 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f9484084ca0 con 0x7f948c14dd80 2023-10-30T13:51:24.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.620+0000 7f9492472640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94700664d0 0x7f9470068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.620+0000 7f9492472640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94700664d0 0x7f9470068990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f948c063a60 tx=0x7f9488009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:24.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.624+0000 7f9482ffd640 1 -- 172.21.15.31:0/3742683791 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94840557d0 con 0x7f948c14dd80 2023-10-30T13:51:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f948c14eb60 con 0x7f948c14dd80 2023-10-30T13:51:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f9482ffd640 1 -- 172.21.15.31:0/3742683791 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f9484003db0 con 0x7f948c14dd80 2023-10-30T13:51:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:24.813 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:22.020359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2023-10-30T13:51:24.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94700664d0 msgr2=0x7f9470068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94700664d0 0x7f9470068990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f948c063a60 tx=0x7f9488009040 comp rx=0 tx=0).stop 2023-10-30T13:51:24.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f948c14dd80 msgr2=0x7f948c0ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f948c14dd80 0x7f948c0ca0b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f948c1541f0 tx=0x7f9484009f90 comp rx=0 tx=0).stop 2023-10-30T13:51:24.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 shutdown_connections 2023-10-30T13:51:24.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f94700664d0 0x7f9470068990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f948c14dd80 0x7f948c0ca0b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f948c14ef20 0x7f948c0c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 --2- 172.21.15.31:0/3742683791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94780047d0 0x7f948c1415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 >> 172.21.15.31:0/3742683791 conn(0x7f948c0cac70 msgr2=0x7f948c0c5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:24.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 shutdown_connections 2023-10-30T13:51:24.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:24.812+0000 7f94946d6640 1 -- 172.21.15.31:0/3742683791 wait complete. 2023-10-30T13:51:24.834 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:26.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:26.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.988+0000 7f05c68d6640 1 Processor -- start 2023-10-30T13:51:26.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.988+0000 7f05c68d6640 1 -- start start 2023-10-30T13:51:26.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c68d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c68d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f05c00cb8b0 0x7f05c006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c68d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c00c5030 con 0x7f05c0151c20 2023-10-30T13:51:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c68d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c0157830 con 0x7f05c00cb8b0 2023-10-30T13:51:26.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c68d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c0157b10 con 0x7f05c0151840 2023-10-30T13:51:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05bffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c4e73640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f05c00cb8b0 0x7f05c006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:26.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c4e73640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f05c00cb8b0 0x7f05c006b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:34946/0 (socket says 172.21.15.31:34946) 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c4e73640 1 -- 172.21.15.31:0/274563256 learned_addr learned my addr 172.21.15.31:0/274563256 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05bffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:43086/0 (socket says 172.21.15.31:43086) 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05bf7fe640 1 --1- 172.21.15.31:0/274563256 >> v1:172.21.15.33:6789/0 conn(0x7f05c0151c20 0x7f05c0072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:54706/0 (socket says 172.21.15.31:54706) 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05bffff640 1 -- 172.21.15.31:0/274563256 >> v2:172.21.15.33:3300/0 conn(0x7f05c00cb8b0 msgr2=0x7f05c006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05bffff640 1 --2- 172.21.15.31:0/274563256 >> v2:172.21.15.33:3300/0 conn(0x7f05c00cb8b0 0x7f05c006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05bffff640 1 -- 172.21.15.31:0/274563256 >> v1:172.21.15.33:6789/0 conn(0x7f05c0151c20 legacy=0x7f05c0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05bffff640 1 -- 172.21.15.31:0/274563256 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c0157df0 con 0x7f05c0151840 2023-10-30T13:51:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.992+0000 7f05c4e73640 1 --2- 172.21.15.31:0/274563256 >> v2:172.21.15.33:3300/0 conn(0x7f05c00cb8b0 0x7f05c006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:51:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05bffff640 1 --2- 172.21.15.31:0/274563256 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c49a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097c0 tx=0x7f05b001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e0677c29d28ded3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05beffd640 1 -- 172.21.15.31:0/274563256 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00a8970 con 0x7f05c0151840 2023-10-30T13:51:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05beffd640 1 -- 172.21.15.31:0/274563256 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b00a8ad0 con 0x7f05c0151840 2023-10-30T13:51:26.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05beffd640 1 -- 172.21.15.31:0/274563256 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00a8e60 con 0x7f05c0151840 2023-10-30T13:51:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 -- 172.21.15.31:0/274563256 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 msgr2=0x7f05c00c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:26.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 --2- 172.21.15.31:0/274563256 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c49a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097c0 tx=0x7f05b001ec70 comp rx=0 tx=0).stop 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 -- 172.21.15.31:0/274563256 shutdown_connections 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 --2- 172.21.15.31:0/274563256 >> v2:172.21.15.33:3300/0 conn(0x7f05c00cb8b0 0x7f05c006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 --2- 172.21.15.31:0/274563256 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c49a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 -- 172.21.15.31:0/274563256 >> 172.21.15.31:0/274563256 conn(0x7f05c00cac70 msgr2=0x7f05c00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 -- 172.21.15.31:0/274563256 shutdown_connections 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 -- 172.21.15.31:0/274563256 wait complete. 2023-10-30T13:51:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 Processor -- start 2023-10-30T13:51:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:26.996+0000 7f05c68d6640 1 -- start start 2023-10-30T13:51:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c00cb8b0 0x7f05c014bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05c0151c20 0x7f05c013d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c014c940 con 0x7f05c00cb8b0 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c014d770 con 0x7f05c0151c20 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c014c660 con 0x7f05c0151840 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c00cb8b0 0x7f05c014bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c00cb8b0 0x7f05c014bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47346/0 (socket says 172.21.15.31:47346) 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 -- 172.21.15.31:0/2150891171 learned_addr learned my addr 172.21.15.31:0/2150891171 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bf7fe640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c4e73640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05c0151c20 0x7f05c013d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 -- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 msgr2=0x7f05c00c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 -- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05c0151c20 msgr2=0x7f05c013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05c0151c20 0x7f05c013d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 -- 172.21.15.31:0/2150891171 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b8009b30 con 0x7f05c00cb8b0 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bffff640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c00cb8b0 0x7f05c014bfd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f05b001c8a0 tx=0x7f05b001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bcff9640 1 -- 172.21.15.31:0/2150891171 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00b0e10 con 0x7f05c00cb8b0 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bcff9640 1 -- 172.21.15.31:0/2150891171 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b00b9410 con 0x7f05c00cb8b0 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b0009420 con 0x7f05c00cb8b0 2023-10-30T13:51:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bcff9640 1 -- 172.21.15.31:0/2150891171 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00b97c0 con 0x7f05c00cb8b0 2023-10-30T13:51:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c014d120 con 0x7f05c00cb8b0 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c0066510 con 0x7f05c00cb8b0 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.000+0000 7f05bcff9640 1 -- 172.21.15.31:0/2150891171 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05b00b7040 con 0x7f05c00cb8b0 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.004+0000 7f05bcff9640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f05ac0664a0 0x7f05ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.004+0000 7f05bcff9640 1 -- 172.21.15.31:0/2150891171 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f05b0121f10 con 0x7f05c00cb8b0 2023-10-30T13:51:27.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.004+0000 7f05bf7fe640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f05ac0664a0 0x7f05ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.004+0000 7f05bf7fe640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f05ac0664a0 0x7f05ac068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f05c0161da0 tx=0x7f05b4007cb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:27.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.008+0000 7f05bcff9640 1 -- 172.21.15.31:0/2150891171 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05b00eef20 con 0x7f05c00cb8b0 2023-10-30T13:51:27.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.204+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05c004e170 con 0x7f05c00cb8b0 2023-10-30T13:51:27.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.204+0000 7f05bcff9640 1 -- 172.21.15.31:0/2150891171 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f05b00b5070 con 0x7f05c00cb8b0 2023-10-30T13:51:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:27.207 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:22.020359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:27.208 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2023-10-30T13:51:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f05ac0664a0 msgr2=0x7f05ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f05ac0664a0 0x7f05ac068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f05c0161da0 tx=0x7f05b4007cb0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c00cb8b0 msgr2=0x7f05c014bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c00cb8b0 0x7f05c014bfd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f05b001c8a0 tx=0x7f05b001ba10 comp rx=0 tx=0).stop 2023-10-30T13:51:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 shutdown_connections 2023-10-30T13:51:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f05ac0664a0 0x7f05ac068960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05c0151c20 0x7f05c013d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f05c0151840 0x7f05c00c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 --2- 172.21.15.31:0/2150891171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c00cb8b0 0x7f05c014bfd0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 >> 172.21.15.31:0/2150891171 conn(0x7f05c00cac70 msgr2=0x7f05c01509a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 shutdown_connections 2023-10-30T13:51:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:27.208+0000 7f05c68d6640 1 -- 172.21.15.31:0/2150891171 wait complete. 2023-10-30T13:51:27.228 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:27.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:27.374 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:29.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:29.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0adb3e640 1 Processor -- start 2023-10-30T13:51:29.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0adb3e640 1 -- start start 2023-10-30T13:51:29.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0adb3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff0a8151c20 0x7ff0a80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0adb3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a80c5030 con 0x7ff0a8151840 2023-10-30T13:51:29.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0adb3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a8157830 con 0x7ff0a80cb8b0 2023-10-30T13:51:29.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0adb3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a8157b10 con 0x7ff0a8151c20 2023-10-30T13:51:29.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0a77fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ff0a8151840 0x7ff0a8072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58840/0 (socket says 172.21.15.31:58840) 2023-10-30T13:51:29.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0a77fe640 1 -- 172.21.15.31:0/56502339 learned_addr learned my addr 172.21.15.31:0/56502339 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:29.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0a6ffd640 1 --2- 172.21.15.31:0/56502339 >> v2:172.21.15.31:3300/0 conn(0x7ff0a8151c20 0x7ff0a80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0a6ffd640 1 -- 172.21.15.31:0/56502339 >> v1:172.21.15.33:6789/0 conn(0x7ff0a80cb8b0 legacy=0x7ff0a806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0a6ffd640 1 -- 172.21.15.31:0/56502339 >> v1:172.21.15.31:6789/0 conn(0x7ff0a8151840 legacy=0x7ff0a8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0a6ffd640 1 -- 172.21.15.31:0/56502339 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a8157df0 con 0x7ff0a8151c20 2023-10-30T13:51:29.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.380+0000 7ff0a67fc640 1 -- 172.21.15.31:0/56502339 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844085169 0 0) 0x7ff0a80c5030 con 0x7ff0a8151840 2023-10-30T13:51:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0a6ffd640 1 --2- 172.21.15.31:0/56502339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a80c49a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0097c0 tx=0x7ff09c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27d0ef6d6d220cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0a67fc640 1 -- 172.21.15.31:0/56502339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c0a8970 con 0x7ff0a8151c20 2023-10-30T13:51:29.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0a67fc640 1 -- 172.21.15.31:0/56502339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c0a8ad0 con 0x7ff0a8151c20 2023-10-30T13:51:29.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0a67fc640 1 -- 172.21.15.31:0/56502339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c0a8e60 con 0x7ff0a8151c20 2023-10-30T13:51:29.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0adb3e640 1 -- 172.21.15.31:0/56502339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 msgr2=0x7ff0a80c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/56502339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a80c49a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0097c0 tx=0x7ff09c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:51:29.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0adb3e640 1 -- 172.21.15.31:0/56502339 shutdown_connections 2023-10-30T13:51:29.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/56502339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a80c49a0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0adb3e640 1 -- 172.21.15.31:0/56502339 >> 172.21.15.31:0/56502339 conn(0x7ff0a80cac70 msgr2=0x7ff0a80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:29.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0adb3e640 1 -- 172.21.15.31:0/56502339 shutdown_connections 2023-10-30T13:51:29.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.384+0000 7ff0adb3e640 1 -- 172.21.15.31:0/56502339 wait complete. 2023-10-30T13:51:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 Processor -- start 2023-10-30T13:51:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 -- start start 2023-10-30T13:51:29.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a80cb8b0 0x7ff0a8156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0a8151840 0x7ff0a80c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a8145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a81dfb10 con 0x7ff0a8151c20 2023-10-30T13:51:29.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a8155cb0 con 0x7ff0a80cb8b0 2023-10-30T13:51:29.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0adb3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a81df2b0 con 0x7ff0a8151840 2023-10-30T13:51:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a8145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a6ffd640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0a8151840 0x7ff0a80c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a6ffd640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0a8151840 0x7ff0a80c8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:43102/0 (socket says 172.21.15.31:43102) 2023-10-30T13:51:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a6ffd640 1 -- 172.21.15.31:0/3778253877 learned_addr learned my addr 172.21.15.31:0/3778253877 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a77fe640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a80cb8b0 0x7ff0a8156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a7fff640 1 -- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0a8151840 msgr2=0x7ff0a80c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a7fff640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0a8151840 0x7ff0a80c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a7fff640 1 -- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a80cb8b0 msgr2=0x7ff0a8156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a7fff640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a80cb8b0 0x7ff0a8156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a7fff640 1 -- 172.21.15.31:0/3778253877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c009420 con 0x7ff0a8151c20 2023-10-30T13:51:29.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.388+0000 7ff0a7fff640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a8145900 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000c730 tx=0x7ff0a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff08ffff640 1 -- 172.21.15.31:0/3778253877 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0003b80 con 0x7ff0a8151c20 2023-10-30T13:51:29.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff08ffff640 1 -- 172.21.15.31:0/3778253877 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a0003ce0 con 0x7ff0a8151c20 2023-10-30T13:51:29.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff08ffff640 1 -- 172.21.15.31:0/3778253877 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a000aa30 con 0x7ff0a8151c20 2023-10-30T13:51:29.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a81e0a50 con 0x7ff0a8151c20 2023-10-30T13:51:29.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a81df830 con 0x7ff0a8151c20 2023-10-30T13:51:29.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a804e170 con 0x7ff0a8151c20 2023-10-30T13:51:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff08ffff640 1 -- 172.21.15.31:0/3778253877 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0a000ab90 con 0x7ff0a8151c20 2023-10-30T13:51:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff08ffff640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0940664d0 0x7ff094068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff08ffff640 1 -- 172.21.15.31:0/3778253877 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7ff0a0083eb0 con 0x7ff0a8151c20 2023-10-30T13:51:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.392+0000 7ff0a77fe640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0940664d0 0x7ff094068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.396+0000 7ff0a77fe640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0940664d0 0x7ff094068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff098000f70 tx=0x7ff098007240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.396+0000 7ff08ffff640 1 -- 172.21.15.31:0/3778253877 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff0a0051000 con 0x7ff0a8151c20 2023-10-30T13:51:29.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0a80c6640 con 0x7ff0a8151c20 2023-10-30T13:51:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff08ffff640 1 -- 172.21.15.31:0/3778253877 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5701 (secure 0 0 0) 0x7ff0a0054620 con 0x7ff0a8151c20 2023-10-30T13:51:29.589 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:29.590 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:22.020359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2023-10-30T13:51:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0940664d0 msgr2=0x7ff094068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0940664d0 0x7ff094068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff098000f70 tx=0x7ff098007240 comp rx=0 tx=0).stop 2023-10-30T13:51:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 msgr2=0x7ff0a8145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a8145900 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000c730 tx=0x7ff0a000cbf0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 shutdown_connections 2023-10-30T13:51:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff0940664d0 0x7ff094068990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0a8151c20 0x7ff0a8145900 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff0a8151840 0x7ff0a80c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 --2- 172.21.15.31:0/3778253877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a80cb8b0 0x7ff0a8156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 >> 172.21.15.31:0/3778253877 conn(0x7ff0a80cac70 msgr2=0x7ff0a80c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:29.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 shutdown_connections 2023-10-30T13:51:29.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:29.588+0000 7ff0adb3e640 1 -- 172.21.15.31:0/3778253877 wait complete. 2023-10-30T13:51:29.611 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:31.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:31.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.764+0000 7f0d72e17640 1 Processor -- start 2023-10-30T13:51:31.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.764+0000 7f0d72e17640 1 -- start start 2023-10-30T13:51:31.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d72e17640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f0d6c151840 0x7f0d6c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:31.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d72e17640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0c5030 con 0x7f0d6c0cb8b0 2023-10-30T13:51:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d72e17640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c157830 con 0x7f0d6c151c20 2023-10-30T13:51:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d72e17640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c157b10 con 0x7f0d6c151840 2023-10-30T13:51:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d713b4640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f0d6c0cb8b0 0x7f0d6c06b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58844/0 (socket says 172.21.15.31:58844) 2023-10-30T13:51:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d713b4640 1 -- 172.21.15.31:0/781157491 learned_addr learned my addr 172.21.15.31:0/781157491 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d70bb3640 1 --2- 172.21.15.31:0/781157491 >> v2:172.21.15.33:3300/0 conn(0x7f0d6c151840 0x7f0d6c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:31.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/781157491 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782818211 0 0) 0x7f0d6c0c5030 con 0x7f0d6c0cb8b0 2023-10-30T13:51:31.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/781157491 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d5c003530 con 0x7f0d6c0cb8b0 2023-10-30T13:51:31.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/781157491 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258879385 0 0) 0x7f0d6c157830 con 0x7f0d6c151c20 2023-10-30T13:51:31.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/781157491 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c0c5030 con 0x7f0d6c151c20 2023-10-30T13:51:31.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d70bb3640 1 -- 172.21.15.31:0/781157491 >> v1:172.21.15.33:6789/0 conn(0x7f0d6c151c20 legacy=0x7f0d6c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:31.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d70bb3640 1 -- 172.21.15.31:0/781157491 >> v1:172.21.15.31:6789/0 conn(0x7f0d6c0cb8b0 legacy=0x7f0d6c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:31.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.768+0000 7f0d70bb3640 1 -- 172.21.15.31:0/781157491 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c157df0 con 0x7f0d6c151840 2023-10-30T13:51:31.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d70bb3640 1 --2- 172.21.15.31:0/781157491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c151840 0x7f0d6c0c49a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0d600097c0 tx=0x7f0d6001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=742c96b3660329cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:31.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/781157491 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d600a8970 con 0x7f0d6c151840 2023-10-30T13:51:31.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/781157491 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d600a8ad0 con 0x7f0d6c151840 2023-10-30T13:51:31.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/781157491 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d600a8e60 con 0x7f0d6c151840 2023-10-30T13:51:31.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 -- 172.21.15.31:0/781157491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c151840 msgr2=0x7f0d6c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:31.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 --2- 172.21.15.31:0/781157491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c151840 0x7f0d6c0c49a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0d600097c0 tx=0x7f0d6001ec70 comp rx=0 tx=0).stop 2023-10-30T13:51:31.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 -- 172.21.15.31:0/781157491 shutdown_connections 2023-10-30T13:51:31.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 --2- 172.21.15.31:0/781157491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c151840 0x7f0d6c0c49a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 -- 172.21.15.31:0/781157491 >> 172.21.15.31:0/781157491 conn(0x7f0d6c0cac70 msgr2=0x7f0d6c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 -- 172.21.15.31:0/781157491 shutdown_connections 2023-10-30T13:51:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 -- 172.21.15.31:0/781157491 wait complete. 2023-10-30T13:51:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 Processor -- start 2023-10-30T13:51:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.772+0000 7f0d72e17640 1 -- start start 2023-10-30T13:51:31.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb8b0 0x7f0d6c1ef430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:31.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c151840 0x7f0d6c14e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c151c20 0x7f0d6c14e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c071eb0 con 0x7f0d6c151c20 2023-10-30T13:51:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c1eea20 con 0x7f0d6c0cb8b0 2023-10-30T13:51:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c071930 con 0x7f0d6c151840 2023-10-30T13:51:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c151c20 0x7f0d6c14e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c151c20 0x7f0d6c14e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47370/0 (socket says 172.21.15.31:47370) 2023-10-30T13:51:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 -- 172.21.15.31:0/2969715438 learned_addr learned my addr 172.21.15.31:0/2969715438 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d6bfff640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c151840 0x7f0d6c14e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d70bb3640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb8b0 0x7f0d6c1ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 -- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c151840 msgr2=0x7f0d6c14e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c151840 0x7f0d6c14e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:31.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 -- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb8b0 msgr2=0x7f0d6c1ef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:31.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb8b0 0x7f0d6c1ef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:31.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 -- 172.21.15.31:0/2969715438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d60009420 con 0x7f0d6c151c20 2023-10-30T13:51:31.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d713b4640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c151c20 0x7f0d6c14e6f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400cb40 tx=0x7f0d6400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:31.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d697fa640 1 -- 172.21.15.31:0/2969715438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6400aca0 con 0x7f0d6c151c20 2023-10-30T13:51:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d697fa640 1 -- 172.21.15.31:0/2969715438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6400ae00 con 0x7f0d6c151c20 2023-10-30T13:51:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d697fa640 1 -- 172.21.15.31:0/2969715438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64012600 con 0x7f0d6c151c20 2023-10-30T13:51:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c1ef970 con 0x7f0d6c151c20 2023-10-30T13:51:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c14cc20 con 0x7f0d6c151c20 2023-10-30T13:51:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.776+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c04e170 con 0x7f0d6c151c20 2023-10-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.780+0000 7f0d697fa640 1 -- 172.21.15.31:0/2969715438 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d64007070 con 0x7f0d6c151c20 2023-10-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.780+0000 7f0d697fa640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d500664a0 0x7f0d50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.780+0000 7f0d697fa640 1 -- 172.21.15.31:0/2969715438 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f0d64084170 con 0x7f0d6c151c20 2023-10-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.780+0000 7f0d70bb3640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d500664a0 0x7f0d50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.780+0000 7f0d70bb3640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d500664a0 0x7f0d50068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0d600b2000 tx=0x7f0d600aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.784+0000 7f0d697fa640 1 -- 172.21.15.31:0/2969715438 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d640512c0 con 0x7f0d6c151c20 2023-10-30T13:51:31.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d6c073a30 con 0x7f0d6c151c20 2023-10-30T13:51:31.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d697fa640 1 -- 172.21.15.31:0/2969715438 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f0d640548e0 con 0x7f0d6c151c20 2023-10-30T13:51:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:31.977 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:22.020359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:31.978 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2023-10-30T13:51:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d500664a0 msgr2=0x7f0d50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d500664a0 0x7f0d50068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0d600b2000 tx=0x7f0d600aa070 comp rx=0 tx=0).stop 2023-10-30T13:51:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c151c20 msgr2=0x7f0d6c14e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c151c20 0x7f0d6c14e6f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400cb40 tx=0x7f0d6400a060 comp rx=0 tx=0).stop 2023-10-30T13:51:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 shutdown_connections 2023-10-30T13:51:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d500664a0 0x7f0d50068960 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c151c20 0x7f0d6c14e6f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c151840 0x7f0d6c14e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 --2- 172.21.15.31:0/2969715438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb8b0 0x7f0d6c1ef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 >> 172.21.15.31:0/2969715438 conn(0x7f0d6c0cac70 msgr2=0x7f0d6c0c6320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 shutdown_connections 2023-10-30T13:51:31.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:31.976+0000 7f0d72e17640 1 -- 172.21.15.31:0/2969715438 wait complete. 2023-10-30T13:51:31.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:33.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:34.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.148+0000 7fc56fc17640 1 Processor -- start 2023-10-30T13:51:34.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.148+0000 7fc56fc17640 1 -- start start 2023-10-30T13:51:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56fc17640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc568150830 0x7fc56806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56fc17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680cb7d0 0x7fc5680c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56fc17640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc5680c3e40 0x7fc568070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56fc17640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5680c2310 con 0x7fc5680c3e40 2023-10-30T13:51:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56fc17640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc568156710 con 0x7fc568150830 2023-10-30T13:51:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56fc17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680c4360 con 0x7fc5680cb7d0 2023-10-30T13:51:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc5680c3e40 0x7fc568070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc5680c3e40 0x7fc568070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47386/0 (socket says 172.21.15.31:47386) 2023-10-30T13:51:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 -- 172.21.15.31:0/4281383383 learned_addr learned my addr 172.21.15.31:0/4281383383 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56d1b2640 1 --2- 172.21.15.31:0/4281383383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680cb7d0 0x7fc5680c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56d9b3640 1 --2- 172.21.15.31:0/4281383383 >> v2:172.21.15.33:3300/0 conn(0x7fc568150830 0x7fc56806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 -- 172.21.15.31:0/4281383383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680cb7d0 msgr2=0x7fc5680c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:34.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 --2- 172.21.15.31:0/4281383383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680cb7d0 0x7fc5680c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 -- 172.21.15.31:0/4281383383 >> v2:172.21.15.33:3300/0 conn(0x7fc568150830 msgr2=0x7fc56806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:34.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 --2- 172.21.15.31:0/4281383383 >> v2:172.21.15.33:3300/0 conn(0x7fc568150830 0x7fc56806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.152+0000 7fc56e1b4640 1 -- 172.21.15.31:0/4281383383 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5681569b0 con 0x7fc5680c3e40 2023-10-30T13:51:34.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56e1b4640 1 --2- 172.21.15.31:0/4281383383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680c3e40 0x7fc568070520 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc56806fa40 tx=0x7fc56001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7f69c3a3f65d31 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:34.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56c9b1640 1 -- 172.21.15.31:0/4281383383 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5600af400 con 0x7fc5680c3e40 2023-10-30T13:51:34.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56c9b1640 1 -- 172.21.15.31:0/4281383383 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5600ab050 con 0x7fc5680c3e40 2023-10-30T13:51:34.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56c9b1640 1 -- 172.21.15.31:0/4281383383 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5600af780 con 0x7fc5680c3e40 2023-10-30T13:51:34.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 -- 172.21.15.31:0/4281383383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680c3e40 msgr2=0x7fc568070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 --2- 172.21.15.31:0/4281383383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680c3e40 0x7fc568070520 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc56806fa40 tx=0x7fc56001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 -- 172.21.15.31:0/4281383383 shutdown_connections 2023-10-30T13:51:34.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 --2- 172.21.15.31:0/4281383383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680c3e40 0x7fc568070520 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 --2- 172.21.15.31:0/4281383383 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680cb7d0 0x7fc5680c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 --2- 172.21.15.31:0/4281383383 >> v2:172.21.15.33:3300/0 conn(0x7fc568150830 0x7fc56806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 -- 172.21.15.31:0/4281383383 >> 172.21.15.31:0/4281383383 conn(0x7fc5680cac70 msgr2=0x7fc5680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:34.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 -- 172.21.15.31:0/4281383383 shutdown_connections 2023-10-30T13:51:34.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 -- 172.21.15.31:0/4281383383 wait complete. 2023-10-30T13:51:34.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 Processor -- start 2023-10-30T13:51:34.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.156+0000 7fc56fc17640 1 -- start start 2023-10-30T13:51:34.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680c3e40 0x7fc5681469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680cb7d0 0x7fc568146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc568150830 0x7fc56813e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5681480c0 con 0x7fc5680cb7d0 2023-10-30T13:51:34.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680c6160 con 0x7fc568150830 2023-10-30T13:51:34.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc568146240 con 0x7fc5680c3e40 2023-10-30T13:51:34.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680cb7d0 0x7fc568146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680cb7d0 0x7fc568146f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47388/0 (socket says 172.21.15.31:47388) 2023-10-30T13:51:34.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 -- 172.21.15.31:0/1137640329 learned_addr learned my addr 172.21.15.31:0/1137640329 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:34.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d9b3640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680c3e40 0x7fc5681469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56e1b4640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc568150830 0x7fc56813e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 -- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680c3e40 msgr2=0x7fc5681469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680c3e40 0x7fc5681469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 -- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc568150830 msgr2=0x7fc56813e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc568150830 0x7fc56813e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 -- 172.21.15.31:0/1137640329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc560009b30 con 0x7fc5680cb7d0 2023-10-30T13:51:34.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56e1b4640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc568150830 0x7fc56813e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:34.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56d1b2640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680cb7d0 0x7fc568146f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc55800d800 tx=0x7fc55800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:34.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc55e7fc640 1 -- 172.21.15.31:0/1137640329 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580096a0 con 0x7fc5680cb7d0 2023-10-30T13:51:34.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc55e7fc640 1 -- 172.21.15.31:0/1137640329 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558009800 con 0x7fc5680cb7d0 2023-10-30T13:51:34.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc55e7fc640 1 -- 172.21.15.31:0/1137640329 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558009b70 con 0x7fc5680cb7d0 2023-10-30T13:51:34.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc568147de0 con 0x7fc5680cb7d0 2023-10-30T13:51:34.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc568162250 con 0x7fc5680cb7d0 2023-10-30T13:51:34.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56804e170 con 0x7fc5680cb7d0 2023-10-30T13:51:34.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.160+0000 7fc55e7fc640 1 -- 172.21.15.31:0/1137640329 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc558009cd0 con 0x7fc5680cb7d0 2023-10-30T13:51:34.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.164+0000 7fc55e7fc640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc54c0664a0 0x7fc54c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.164+0000 7fc55e7fc640 1 -- 172.21.15.31:0/1137640329 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fc558084820 con 0x7fc5680cb7d0 2023-10-30T13:51:34.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.164+0000 7fc56d9b3640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc54c0664a0 0x7fc54c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.164+0000 7fc56d9b3640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc54c0664a0 0x7fc54c068960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc5640079c0 tx=0x7fc564008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:34.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.168+0000 7fc55e7fc640 1 -- 172.21.15.31:0/1137640329 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5580519d0 con 0x7fc5680cb7d0 2023-10-30T13:51:34.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5680c2b80 con 0x7fc5680cb7d0 2023-10-30T13:51:34.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc55e7fc640 1 -- 172.21.15.31:0/1137640329 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5705 (secure 0 0 0) 0x7fc558054ff0 con 0x7fc5680cb7d0 2023-10-30T13:51:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:34.361 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:31.953498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:34.362 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 73 2023-10-30T13:51:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc54c0664a0 msgr2=0x7fc54c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc54c0664a0 0x7fc54c068960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc5640079c0 tx=0x7fc564008040 comp rx=0 tx=0).stop 2023-10-30T13:51:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680cb7d0 msgr2=0x7fc568146f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680cb7d0 0x7fc568146f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc55800d800 tx=0x7fc55800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 shutdown_connections 2023-10-30T13:51:34.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc54c0664a0 0x7fc54c068960 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc568150830 0x7fc56813e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5680cb7d0 0x7fc568146f30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 --2- 172.21.15.31:0/1137640329 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc5680c3e40 0x7fc5681469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 >> 172.21.15.31:0/1137640329 conn(0x7fc5680cac70 msgr2=0x7fc568144db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:34.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 shutdown_connections 2023-10-30T13:51:34.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:34.360+0000 7fc56fc17640 1 -- 172.21.15.31:0/1137640329 wait complete. 2023-10-30T13:51:34.382 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5591": { "addr": "172.21.15.33:6836/3928996203", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3928996203, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3928996203, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5591, "incarnation": 72, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:51:31.953498+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5591 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:51:36.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:36.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71633e4640 1 Processor -- start 2023-10-30T13:51:36.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71633e4640 1 -- start start 2023-10-30T13:51:36.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71633e4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c153b50 0x7f715c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71633e4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c15a770 con 0x7f715c0c62b0 2023-10-30T13:51:36.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71633e4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0c9d40 con 0x7f715c0cb180 2023-10-30T13:51:36.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71633e4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c159cf0 con 0x7f715c153b50 2023-10-30T13:51:36.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7161be1640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f715c0c62b0 0x7f715c0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43668/0 (socket says 172.21.15.31:43668) 2023-10-30T13:51:36.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7161be1640 1 -- 172.21.15.31:0/239984085 learned_addr learned my addr 172.21.15.31:0/239984085 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:36.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71623e2640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c153b50 0x7f715c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116713510 0 0) 0x7f714c001120 con 0x7f715c0c62b0 2023-10-30T13:51:36.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7148003530 con 0x7f715c0c62b0 2023-10-30T13:51:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284686246 0 0) 0x7f715c0c9d40 con 0x7f715c0cb180 2023-10-30T13:51:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c001120 con 0x7f715c0cb180 2023-10-30T13:51:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415589076 0 0) 0x7f7148003530 con 0x7f715c0c62b0 2023-10-30T13:51:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c153b50 msgr2=0x7f715c1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c153b50 0x7f715c1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 >> v1:172.21.15.33:6789/0 conn(0x7f715c0cb180 legacy=0x7f715c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c153f30 con 0x7f715c0c62b0 2023-10-30T13:51:36.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f714c001810 con 0x7f715c0c62b0 2023-10-30T13:51:36.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 >> v1:172.21.15.31:6789/0 conn(0x7f715c0c62b0 legacy=0x7f715c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148003b40 0x7f7148003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f71480047d0 0x7f7148004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7148005190 0x7f714800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c159cf0 con 0x7f7148005190 2023-10-30T13:51:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714800bf80 con 0x7f7148003b40 2023-10-30T13:51:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714800c140 con 0x7f71480047d0 2023-10-30T13:51:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71623e2640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c153b50 0x7f715c1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:51:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7162be3640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7148005190 0x7f714800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7161be1640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f71480047d0 0x7f7148004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71623e2640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148003b40 0x7f7148003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7162be3640 1 -- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f71480047d0 msgr2=0x7f7148004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7162be3640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f71480047d0 0x7f7148004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7162be3640 1 -- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148003b40 msgr2=0x7f7148003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:36.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7162be3640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148003b40 0x7f7148003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f7162be3640 1 -- 172.21.15.31:0/239984085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148003530 con 0x7f7148005190 2023-10-30T13:51:36.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.540+0000 7f71623e2640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148003b40 0x7f7148003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:36.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f7162be3640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7148005190 0x7f714800b990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7154001fd0 tx=0x7f715401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465397431a92511a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:36.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154003b80 con 0x7f7148005190 2023-10-30T13:51:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7154003d00 con 0x7f7148005190 2023-10-30T13:51:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71613e0640 1 -- 172.21.15.31:0/239984085 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154005310 con 0x7f7148005190 2023-10-30T13:51:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71633e4640 1 -- 172.21.15.31:0/239984085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7148005190 msgr2=0x7f714800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71633e4640 1 --2- 172.21.15.31:0/239984085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7148005190 0x7f714800b990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7154001fd0 tx=0x7f715401c8e0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f7162be3640 1 -- 172.21.15.31:0/239984085 reap_dead start 2023-10-30T13:51:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71633e4640 1 -- 172.21.15.31:0/239984085 shutdown_connections 2023-10-30T13:51:36.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71633e4640 1 -- 172.21.15.31:0/239984085 >> 172.21.15.31:0/239984085 conn(0x7f715c065a80 msgr2=0x7f715c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:36.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71633e4640 1 -- 172.21.15.31:0/239984085 shutdown_connections 2023-10-30T13:51:36.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.544+0000 7f71633e4640 1 -- 172.21.15.31:0/239984085 wait complete. 2023-10-30T13:51:36.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 Processor -- start 2023-10-30T13:51:36.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 -- start start 2023-10-30T13:51:36.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148005190 0x7f715c0c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c14c210 0x7f715c14c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715c158480 0x7f715c158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c14cbb0 con 0x7f715c158480 2023-10-30T13:51:36.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c157ff0 con 0x7f7148005190 2023-10-30T13:51:36.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c158e40 con 0x7f715c14c210 2023-10-30T13:51:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715c158480 0x7f715c158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715c158480 0x7f715c158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40354/0 (socket says 172.21.15.31:40354) 2023-10-30T13:51:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 -- 172.21.15.31:0/193442385 learned_addr learned my addr 172.21.15.31:0/193442385 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7161be1640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c14c210 0x7f715c14c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71623e2640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148005190 0x7f715c0c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 -- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c14c210 msgr2=0x7f715c14c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c14c210 0x7f715c14c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 -- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148005190 msgr2=0x7f715c0c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148005190 0x7f715c0c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 -- 172.21.15.31:0/193442385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c1eeba0 con 0x7f715c158480 2023-10-30T13:51:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7161be1640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c14c210 0x7f715c14c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7162be3640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715c158480 0x7f715c158900 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7154001fd0 tx=0x7f7148005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7152ffd640 1 -- 172.21.15.31:0/193442385 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71540b4070 con 0x7f715c158480 2023-10-30T13:51:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7152ffd640 1 -- 172.21.15.31:0/193442385 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7154003e70 con 0x7f715c158480 2023-10-30T13:51:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c1eee60 con 0x7f715c158480 2023-10-30T13:51:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f7152ffd640 1 -- 172.21.15.31:0/193442385 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154006d60 con 0x7f715c158480 2023-10-30T13:51:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.548+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c1579a0 con 0x7f715c158480 2023-10-30T13:51:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.552+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c150d00 con 0x7f715c158480 2023-10-30T13:51:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.552+0000 7f7152ffd640 1 -- 172.21.15.31:0/193442385 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71540052e0 con 0x7f715c158480 2023-10-30T13:51:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.552+0000 7f7152ffd640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f71400664a0 0x7f7140068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.552+0000 7f7152ffd640 1 -- 172.21.15.31:0/193442385 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f71540aa070 con 0x7f715c158480 2023-10-30T13:51:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.552+0000 7f71623e2640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f71400664a0 0x7f7140068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.552+0000 7f71623e2640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f71400664a0 0x7f7140068960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f715c066890 tx=0x7f7158008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.556+0000 7f7152ffd640 1 -- 172.21.15.31:0/193442385 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71540ea860 con 0x7f715c158480 2023-10-30T13:51:36.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.748+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f715c067540 con 0x7f715c158480 2023-10-30T13:51:36.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.748+0000 7f7152ffd640 1 -- 172.21.15.31:0/193442385 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f71540ede80 con 0x7f715c158480 2023-10-30T13:51:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:36.750 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:31.953498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:36.750 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 73 2023-10-30T13:51:36.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f71400664a0 msgr2=0x7f7140068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f71400664a0 0x7f7140068960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f715c066890 tx=0x7f7158008040 comp rx=0 tx=0).stop 2023-10-30T13:51:36.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715c158480 msgr2=0x7f715c158900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715c158480 0x7f715c158900 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7154001fd0 tx=0x7f7148005800 comp rx=0 tx=0).stop 2023-10-30T13:51:36.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 shutdown_connections 2023-10-30T13:51:36.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f71400664a0 0x7f7140068960 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715c158480 0x7f715c158900 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f715c14c210 0x7f715c14c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 --2- 172.21.15.31:0/193442385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7148005190 0x7f715c0c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 >> 172.21.15.31:0/193442385 conn(0x7f715c065a80 msgr2=0x7f715c150190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:36.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 shutdown_connections 2023-10-30T13:51:36.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:36.752+0000 7f71633e4640 1 -- 172.21.15.31:0/193442385 wait complete. 2023-10-30T13:51:36.771 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:38.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:38.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.924+0000 7f4bcfbd3640 1 Processor -- start 2023-10-30T13:51:38.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.924+0000 7f4bcfbd3640 1 -- start start 2023-10-30T13:51:38.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bcfbd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc80cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bcfbd3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4bc80c2500 0x7f4bc80c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bcfbd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc8151aa0 con 0x7f4bc81516c0 2023-10-30T13:51:38.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bcfbd3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80c74a0 con 0x7f4bc80c2500 2023-10-30T13:51:38.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bcfbd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80c2e90 con 0x7f4bc80ccf60 2023-10-30T13:51:38.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bce170640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4bc80c2500 0x7f4bc80c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bce170640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4bc80c2500 0x7f4bc80c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40370/0 (socket says 172.21.15.31:40370) 2023-10-30T13:51:38.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bce170640 1 -- 172.21.15.31:0/4049695589 learned_addr learned my addr 172.21.15.31:0/4049695589 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:38.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bcd16e640 1 --2- 172.21.15.31:0/4049695589 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc80cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bce170640 1 -- 172.21.15.31:0/4049695589 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 msgr2=0x7f4bc80cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:38.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bce170640 1 --2- 172.21.15.31:0/4049695589 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc80cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bce170640 1 -- 172.21.15.31:0/4049695589 >> v1:172.21.15.33:6789/0 conn(0x7f4bc81516c0 legacy=0x7f4bc80c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.928+0000 7f4bce170640 1 -- 172.21.15.31:0/4049695589 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8158af0 con 0x7f4bc80c2500 2023-10-30T13:51:38.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bce170640 1 --2- 172.21.15.31:0/4049695589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc80c2950 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009e50 tx=0x7f4bc001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e26abff0133a27a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:38.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcc96d640 1 -- 172.21.15.31:0/4049695589 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00a56e0 con 0x7f4bc80c2500 2023-10-30T13:51:38.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcc96d640 1 -- 172.21.15.31:0/4049695589 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc00a5840 con 0x7f4bc80c2500 2023-10-30T13:51:38.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcc96d640 1 -- 172.21.15.31:0/4049695589 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00a5bd0 con 0x7f4bc80c2500 2023-10-30T13:51:38.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/4049695589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 msgr2=0x7f4bc80c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/4049695589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc80c2950 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009e50 tx=0x7f4bc001ca10 comp rx=0 tx=0).stop 2023-10-30T13:51:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/4049695589 shutdown_connections 2023-10-30T13:51:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/4049695589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc80c2950 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/4049695589 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc80cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/4049695589 >> 172.21.15.31:0/4049695589 conn(0x7f4bc8071fb0 msgr2=0x7f4bc80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/4049695589 shutdown_connections 2023-10-30T13:51:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/4049695589 wait complete. 2023-10-30T13:51:38.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 Processor -- start 2023-10-30T13:51:38.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- start start 2023-10-30T13:51:38.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc815b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc81f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bc81516c0 0x7f4bc81f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc814f750 con 0x7f4bc80c2500 2023-10-30T13:51:38.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc814f9a0 con 0x7f4bc81516c0 2023-10-30T13:51:38.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.932+0000 7f4bcfbd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc814fc80 con 0x7f4bc80ccf60 2023-10-30T13:51:38.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc815b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc815b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40380/0 (socket says 172.21.15.31:40380) 2023-10-30T13:51:38.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 -- 172.21.15.31:0/2113274300 learned_addr learned my addr 172.21.15.31:0/2113274300 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:38.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd16e640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc81f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bce170640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bc81516c0 0x7f4bc81f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 -- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 msgr2=0x7f4bc81f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc81f5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 -- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bc81516c0 msgr2=0x7f4bc81f5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bc81516c0 0x7f4bc81f5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 -- 172.21.15.31:0/2113274300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc0009b30 con 0x7f4bc80c2500 2023-10-30T13:51:38.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcd96f640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc815b0d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c500 tx=0x7f4bc400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:38.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bbe7fc640 1 -- 172.21.15.31:0/2113274300 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4005220 con 0x7f4bc80c2500 2023-10-30T13:51:38.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bbe7fc640 1 -- 172.21.15.31:0/2113274300 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc4005380 con 0x7f4bc80c2500 2023-10-30T13:51:38.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bbe7fc640 1 -- 172.21.15.31:0/2113274300 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4007270 con 0x7f4bc80c2500 2023-10-30T13:51:38.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc814ff60 con 0x7f4bc80c2500 2023-10-30T13:51:38.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc8150520 con 0x7f4bc80c2500 2023-10-30T13:51:38.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc8073b80 con 0x7f4bc80c2500 2023-10-30T13:51:38.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.936+0000 7f4bbe7fc640 1 -- 172.21.15.31:0/2113274300 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bc40073d0 con 0x7f4bc80c2500 2023-10-30T13:51:38.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.940+0000 7f4bbe7fc640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bac0664a0 0x7f4bac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.940+0000 7f4bbe7fc640 1 -- 172.21.15.31:0/2113274300 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f4bc4083a60 con 0x7f4bc80c2500 2023-10-30T13:51:38.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.940+0000 7f4bcd16e640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bac0664a0 0x7f4bac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.940+0000 7f4bcd16e640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bac0664a0 0x7f4bac068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8072500 tx=0x7f4bb8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:38.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:38.944+0000 7f4bbe7fc640 1 -- 172.21.15.31:0/2113274300 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bc4050bc0 con 0x7f4bc80c2500 2023-10-30T13:51:39.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.136+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc804e170 con 0x7f4bc80c2500 2023-10-30T13:51:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.136+0000 7f4bbe7fc640 1 -- 172.21.15.31:0/2113274300 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f4bc40541e0 con 0x7f4bc80c2500 2023-10-30T13:51:39.138 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:39.139 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:37.206424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 74 2023-10-30T13:51:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bac0664a0 msgr2=0x7f4bac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bac0664a0 0x7f4bac068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8072500 tx=0x7f4bb8008040 comp rx=0 tx=0).stop 2023-10-30T13:51:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 msgr2=0x7f4bc815b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc815b0d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c500 tx=0x7f4bc400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 shutdown_connections 2023-10-30T13:51:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4bac0664a0 0x7f4bac068960 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bc81516c0 0x7f4bc81f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4bc80ccf60 0x7f4bc81f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 --2- 172.21.15.31:0/2113274300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bc80c2500 0x7f4bc815b0d0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 >> 172.21.15.31:0/2113274300 conn(0x7f4bc8071fb0 msgr2=0x7f4bc81617f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 shutdown_connections 2023-10-30T13:51:39.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:39.140+0000 7f4bcfbd3640 1 -- 172.21.15.31:0/2113274300 wait complete. 2023-10-30T13:51:39.160 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:41.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:41.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.312+0000 7fe9adb0f640 1 Processor -- start 2023-10-30T13:51:41.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9adb0f640 1 -- start start 2023-10-30T13:51:41.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9adb0f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 0x7fe9a80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9adb0f640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a80c5030 con 0x7fe9a80cb8b0 2023-10-30T13:51:41.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9adb0f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a8157830 con 0x7fe9a8151840 2023-10-30T13:51:41.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9adb0f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8157b10 con 0x7fe9a8151c20 2023-10-30T13:51:41.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe9a80cb8b0 0x7fe9a806b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43678/0 (socket says 172.21.15.31:43678) 2023-10-30T13:51:41.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 -- 172.21.15.31:0/762102686 learned_addr learned my addr 172.21.15.31:0/762102686 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:41.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a6ffd640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 0x7fe9a80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307928542 0 0) 0x7fe9a80c5030 con 0x7fe9a80cb8b0 2023-10-30T13:51:41.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe990003530 con 0x7fe9a80cb8b0 2023-10-30T13:51:41.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3690490323 0 0) 0x7fe990003530 con 0x7fe9a80cb8b0 2023-10-30T13:51:41.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 msgr2=0x7fe9a80c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:41.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 0x7fe9a80c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 >> v1:172.21.15.33:6789/0 conn(0x7fe9a8151840 legacy=0x7fe9a8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:41.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8157df0 con 0x7fe9a80cb8b0 2023-10-30T13:51:41.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fe9a0001a00 con 0x7fe9a80cb8b0 2023-10-30T13:51:41.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 >> v1:172.21.15.31:6789/0 conn(0x7fe9a80cb8b0 legacy=0x7fe9a806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:41.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9900039b0 0x7fe990003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 0x7fe9900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe990004d10 0x7fe990005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8157b10 con 0x7fe990004d10 2023-10-30T13:51:41.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99000b9c0 con 0x7fe9900039b0 2023-10-30T13:51:41.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99000bb80 con 0x7fe9a8151c20 2023-10-30T13:51:41.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe990004d10 0x7fe990005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a77fe640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9900039b0 0x7fe990003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a6ffd640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 0x7fe9900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 -- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 msgr2=0x7fe9900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:41.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a8151c20 0x7fe9900047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 -- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9900039b0 msgr2=0x7fe990003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:41.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9900039b0 0x7fe990003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a7fff640 1 -- 172.21.15.31:0/762102686 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990003530 con 0x7fe990004d10 2023-10-30T13:51:41.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.316+0000 7fe9a77fe640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9900039b0 0x7fe990003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:41.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9a7fff640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe990004d10 0x7fe990005190 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000a830 tx=0x7fe9a001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad2b2231210cae0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:41.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a00ad630 con 0x7fe990004d10 2023-10-30T13:51:41.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a00a8070 con 0x7fe990004d10 2023-10-30T13:51:41.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9a67fc640 1 -- 172.21.15.31:0/762102686 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a00ada70 con 0x7fe990004d10 2023-10-30T13:51:41.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9adb0f640 1 -- 172.21.15.31:0/762102686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe990004d10 msgr2=0x7fe990005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:41.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9adb0f640 1 --2- 172.21.15.31:0/762102686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe990004d10 0x7fe990005190 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000a830 tx=0x7fe9a001d340 comp rx=0 tx=0).stop 2023-10-30T13:51:41.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9a7fff640 1 -- 172.21.15.31:0/762102686 reap_dead start 2023-10-30T13:51:41.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9adb0f640 1 -- 172.21.15.31:0/762102686 shutdown_connections 2023-10-30T13:51:41.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9adb0f640 1 -- 172.21.15.31:0/762102686 >> 172.21.15.31:0/762102686 conn(0x7fe9a80cac70 msgr2=0x7fe9a80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:41.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9adb0f640 1 -- 172.21.15.31:0/762102686 shutdown_connections 2023-10-30T13:51:41.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.320+0000 7fe9adb0f640 1 -- 172.21.15.31:0/762102686 wait complete. 2023-10-30T13:51:41.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 Processor -- start 2023-10-30T13:51:41.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 -- start start 2023-10-30T13:51:41.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe990004d10 0x7fe9a8070050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9a814c470 0x7fe9a81e0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a81e1510 0x7fe9a81454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81f2ac0 con 0x7fe9a814c470 2023-10-30T13:51:41.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8071cb0 con 0x7fe990004d10 2023-10-30T13:51:41.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81f1bd0 con 0x7fe9a81e1510 2023-10-30T13:51:41.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9a814c470 0x7fe9a81e0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9a814c470 0x7fe9a81e0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40396/0 (socket says 172.21.15.31:40396) 2023-10-30T13:51:41.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 -- 172.21.15.31:0/47793824 learned_addr learned my addr 172.21.15.31:0/47793824 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:41.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a7fff640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a81e1510 0x7fe9a81454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a77fe640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe990004d10 0x7fe9a8070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 -- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a81e1510 msgr2=0x7fe9a81454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:41.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a81e1510 0x7fe9a81454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 -- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe990004d10 msgr2=0x7fe9a8070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:41.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe990004d10 0x7fe9a8070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 -- 172.21.15.31:0/47793824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a81f1eb0 con 0x7fe9a814c470 2023-10-30T13:51:41.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a77fe640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe990004d10 0x7fe9a8070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:41.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9a6ffd640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9a814c470 0x7fe9a81e0fd0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe99c011590 tx=0x7fe99c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:41.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe98ffff640 1 -- 172.21.15.31:0/47793824 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c004220 con 0x7fe9a814c470 2023-10-30T13:51:41.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe98ffff640 1 -- 172.21.15.31:0/47793824 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c004380 con 0x7fe9a814c470 2023-10-30T13:51:41.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a81f2190 con 0x7fe9a814c470 2023-10-30T13:51:41.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe98ffff640 1 -- 172.21.15.31:0/47793824 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c015870 con 0x7fe9a814c470 2023-10-30T13:51:41.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.324+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a81f2750 con 0x7fe9a814c470 2023-10-30T13:51:41.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.328+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a80c3e40 con 0x7fe9a814c470 2023-10-30T13:51:41.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.328+0000 7fe98ffff640 1 -- 172.21.15.31:0/47793824 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe99c0159d0 con 0x7fe9a814c470 2023-10-30T13:51:41.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.328+0000 7fe98ffff640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9940664a0 0x7fe994068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:41.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.328+0000 7fe98ffff640 1 -- 172.21.15.31:0/47793824 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fe99c084360 con 0x7fe9a814c470 2023-10-30T13:51:41.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.328+0000 7fe9a77fe640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9940664a0 0x7fe994068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:41.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.328+0000 7fe9a77fe640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9940664a0 0x7fe994068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe998005500 tx=0x7fe998009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:41.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.332+0000 7fe98ffff640 1 -- 172.21.15.31:0/47793824 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe99c051490 con 0x7fe9a814c470 2023-10-30T13:51:41.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9a804e170 con 0x7fe9a814c470 2023-10-30T13:51:41.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe98ffff640 1 -- 172.21.15.31:0/47793824 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fe99c054ab0 con 0x7fe9a814c470 2023-10-30T13:51:41.529 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:41.530 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:37.206424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:41.530 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 74 2023-10-30T13:51:41.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9940664a0 msgr2=0x7fe994068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:41.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9940664a0 0x7fe994068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe998005500 tx=0x7fe998009040 comp rx=0 tx=0).stop 2023-10-30T13:51:41.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9a814c470 msgr2=0x7fe9a81e0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:41.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9a814c470 0x7fe9a81e0fd0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe99c011590 tx=0x7fe99c011a50 comp rx=0 tx=0).stop 2023-10-30T13:51:41.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 shutdown_connections 2023-10-30T13:51:41.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9940664a0 0x7fe994068960 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe9a81e1510 0x7fe9a81454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9a814c470 0x7fe9a81e0fd0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 --2- 172.21.15.31:0/47793824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe990004d10 0x7fe9a8070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 >> 172.21.15.31:0/47793824 conn(0x7fe9a80cac70 msgr2=0x7fe9a8147810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 shutdown_connections 2023-10-30T13:51:41.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:41.528+0000 7fe9adb0f640 1 -- 172.21.15.31:0/47793824 wait complete. 2023-10-30T13:51:41.551 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:43.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:43.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.704+0000 7f8968fee640 1 Processor -- start 2023-10-30T13:51:43.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.704+0000 7f8968fee640 1 -- start start 2023-10-30T13:51:43.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8968fee640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 0x7f8964159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8968fee640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89640c5f80 con 0x7f8964153bb0 2023-10-30T13:51:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8968fee640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896415a310 con 0x7f896406c420 2023-10-30T13:51:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8968fee640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c5690 con 0x7f8964159980 2023-10-30T13:51:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f896259c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8964153bb0 0x7f89640c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:43686/0 (socket says 172.21.15.31:43686) 2023-10-30T13:51:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f896259c640 1 -- 172.21.15.31:0/3123144233 learned_addr learned my addr 172.21.15.31:0/3123144233 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8962d9d640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 0x7f8964159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658776105 0 0) 0x7f89640c5f80 con 0x7f8964153bb0 2023-10-30T13:51:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8950003530 con 0x7f8964153bb0 2023-10-30T13:51:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2382101645 0 0) 0x7f8950003530 con 0x7f8964153bb0 2023-10-30T13:51:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 msgr2=0x7f8964159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 0x7f8964159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 >> v1:172.21.15.33:6789/0 conn(0x7f896406c420 legacy=0x7f89640c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896415a580 con 0x7f8964153bb0 2023-10-30T13:51:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f8954001680 con 0x7f8964153bb0 2023-10-30T13:51:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 >> v1:172.21.15.31:6789/0 conn(0x7f8964153bb0 legacy=0x7f89640c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89500039b0 0x7f8950003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89500047d0 0x7f8950004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 0x7f8950005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c5690 con 0x7f89500047d0 2023-10-30T13:51:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895000bad0 con 0x7f89500039b0 2023-10-30T13:51:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895000bc90 con 0x7f8964159980 2023-10-30T13:51:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f895bfff640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89500047d0 0x7f8950004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f896259c640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89500039b0 0x7f8950003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f8962d9d640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 0x7f8950005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f895bfff640 1 -- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 msgr2=0x7f8950005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f895bfff640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8964159980 0x7f8950005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f895bfff640 1 -- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89500039b0 msgr2=0x7f8950003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f895bfff640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89500039b0 0x7f8950003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.708+0000 7f895bfff640 1 -- 172.21.15.31:0/3123144233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8950003530 con 0x7f89500047d0 2023-10-30T13:51:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f895bfff640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89500047d0 0x7f8950004c50 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f894c001750 tx=0x7f894c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1d96630e84cff8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c0052d0 con 0x7f89500047d0 2023-10-30T13:51:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894c0a7050 con 0x7f89500047d0 2023-10-30T13:51:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8961d9b640 1 -- 172.21.15.31:0/3123144233 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c003600 con 0x7f89500047d0 2023-10-30T13:51:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 -- 172.21.15.31:0/3123144233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89500047d0 msgr2=0x7f8950004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:43.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 --2- 172.21.15.31:0/3123144233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89500047d0 0x7f8950004c50 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f894c001750 tx=0x7f894c01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8962d9d640 1 -- 172.21.15.31:0/3123144233 reap_dead start 2023-10-30T13:51:43.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 -- 172.21.15.31:0/3123144233 shutdown_connections 2023-10-30T13:51:43.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 -- 172.21.15.31:0/3123144233 >> 172.21.15.31:0/3123144233 conn(0x7f89640cac70 msgr2=0x7f89640cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:43.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 -- 172.21.15.31:0/3123144233 shutdown_connections 2023-10-30T13:51:43.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 -- 172.21.15.31:0/3123144233 wait complete. 2023-10-30T13:51:43.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 Processor -- start 2023-10-30T13:51:43.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.712+0000 7f8968fee640 1 -- start start 2023-10-30T13:51:43.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89500047d0 0x7f89641eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89641f2240 0x7f896414ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89640bc310 0x7f89640bc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640bd110 con 0x7f89641f2240 2023-10-30T13:51:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c3bb0 con 0x7f89640bc310 2023-10-30T13:51:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c1f00 con 0x7f89500047d0 2023-10-30T13:51:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89641f2240 0x7f896414ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89641f2240 0x7f896414ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40416/0 (socket says 172.21.15.31:40416) 2023-10-30T13:51:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 -- 172.21.15.31:0/4040429725 learned_addr learned my addr 172.21.15.31:0/4040429725 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f896259c640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89500047d0 0x7f89641eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8962d9d640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89640bc310 0x7f89640bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 -- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89500047d0 msgr2=0x7f89641eecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89500047d0 0x7f89641eecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 -- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89640bc310 msgr2=0x7f89640bc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89640bc310 0x7f89640bc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 -- 172.21.15.31:0/4040429725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89641f2780 con 0x7f89641f2240 2023-10-30T13:51:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895bfff640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89641f2240 0x7f896414ea40 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f894c005040 tx=0x7f8950005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895affd640 1 -- 172.21.15.31:0/4040429725 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c007a40 con 0x7f89641f2240 2023-10-30T13:51:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895affd640 1 -- 172.21.15.31:0/4040429725 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894c007680 con 0x7f89641f2240 2023-10-30T13:51:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89641f2a60 con 0x7f89641f2240 2023-10-30T13:51:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f895affd640 1 -- 172.21.15.31:0/4040429725 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c006a20 con 0x7f89641f2240 2023-10-30T13:51:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89641f3020 con 0x7f89641f2240 2023-10-30T13:51:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.716+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896404e170 con 0x7f89641f2240 2023-10-30T13:51:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.720+0000 7f895affd640 1 -- 172.21.15.31:0/4040429725 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f894c006b80 con 0x7f89641f2240 2023-10-30T13:51:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.720+0000 7f895affd640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89440664a0 0x7f8944068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.720+0000 7f895affd640 1 -- 172.21.15.31:0/4040429725 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f894c11e860 con 0x7f89641f2240 2023-10-30T13:51:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.720+0000 7f896259c640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89440664a0 0x7f8944068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.720+0000 7f896259c640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89440664a0 0x7f8944068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8954001210 tx=0x7f8954007040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.724+0000 7f895affd640 1 -- 172.21.15.31:0/4040429725 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f894c0eb9c0 con 0x7f89641f2240 2023-10-30T13:51:43.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8964151f90 con 0x7f89641f2240 2023-10-30T13:51:43.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f895affd640 1 -- 172.21.15.31:0/4040429725 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5708 (secure 0 0 0) 0x7f894c0eefe0 con 0x7f89641f2240 2023-10-30T13:51:43.913 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:43.913 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:43.204125+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:43.914 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 75 2023-10-30T13:51:43.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89440664a0 msgr2=0x7f8944068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:43.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89440664a0 0x7f8944068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8954001210 tx=0x7f8954007040 comp rx=0 tx=0).stop 2023-10-30T13:51:43.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89641f2240 msgr2=0x7f896414ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:43.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89641f2240 0x7f896414ea40 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f894c005040 tx=0x7f8950005800 comp rx=0 tx=0).stop 2023-10-30T13:51:43.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 shutdown_connections 2023-10-30T13:51:43.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89440664a0 0x7f8944068960 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89640bc310 0x7f89640bc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89641f2240 0x7f896414ea40 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 --2- 172.21.15.31:0/4040429725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89500047d0 0x7f89641eecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:43.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 >> 172.21.15.31:0/4040429725 conn(0x7f89640cac70 msgr2=0x7f8964150dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:43.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 shutdown_connections 2023-10-30T13:51:43.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:43.912+0000 7f8968fee640 1 -- 172.21.15.31:0/4040429725 wait complete. 2023-10-30T13:51:43.934 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:45.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.088+0000 7fe622fc4640 1 Processor -- start 2023-10-30T13:51:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe622fc4640 1 -- start start 2023-10-30T13:51:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe622fc4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe61c151840 0x7fe61c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe622fc4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c0cb8b0 0x7fe61c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe622fc4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c0c5030 con 0x7fe61c151c20 2023-10-30T13:51:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe622fc4640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61c157830 con 0x7fe61c151840 2023-10-30T13:51:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe622fc4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c157b10 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe620d60640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe61c151840 0x7fe61c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe621561640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c0cb8b0 0x7fe61c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe620d60640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe61c151840 0x7fe61c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:36118/0 (socket says 172.21.15.31:36118) 2023-10-30T13:51:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe620d60640 1 -- 172.21.15.31:0/3786159461 learned_addr learned my addr 172.21.15.31:0/3786159461 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe621561640 1 -- 172.21.15.31:0/3786159461 >> v2:172.21.15.33:3300/0 conn(0x7fe61c151840 msgr2=0x7fe61c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe621561640 1 --2- 172.21.15.31:0/3786159461 >> v2:172.21.15.33:3300/0 conn(0x7fe61c151840 0x7fe61c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe621561640 1 -- 172.21.15.31:0/3786159461 >> v1:172.21.15.33:6789/0 conn(0x7fe61c151c20 legacy=0x7fe61c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe621561640 1 -- 172.21.15.31:0/3786159461 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c157df0 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.092+0000 7fe620d60640 1 --2- 172.21.15.31:0/3786159461 >> v2:172.21.15.33:3300/0 conn(0x7fe61c151840 0x7fe61c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:51:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe621561640 1 --2- 172.21.15.31:0/3786159461 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c0cb8b0 0x7fe61c06b600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe614009d90 tx=0x7fe61401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d32f81e85b8d4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe61b7fe640 1 -- 172.21.15.31:0/3786159461 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140af600 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe61b7fe640 1 -- 172.21.15.31:0/3786159461 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6140ab050 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe61b7fe640 1 -- 172.21.15.31:0/3786159461 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140af980 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 -- 172.21.15.31:0/3786159461 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c0cb8b0 msgr2=0x7fe61c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 --2- 172.21.15.31:0/3786159461 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c0cb8b0 0x7fe61c06b600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe614009d90 tx=0x7fe61401f130 comp rx=0 tx=0).stop 2023-10-30T13:51:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 -- 172.21.15.31:0/3786159461 shutdown_connections 2023-10-30T13:51:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 --2- 172.21.15.31:0/3786159461 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c0cb8b0 0x7fe61c06b600 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 --2- 172.21.15.31:0/3786159461 >> v2:172.21.15.33:3300/0 conn(0x7fe61c151840 0x7fe61c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 -- 172.21.15.31:0/3786159461 >> 172.21.15.31:0/3786159461 conn(0x7fe61c0cac70 msgr2=0x7fe61c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 -- 172.21.15.31:0/3786159461 shutdown_connections 2023-10-30T13:51:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.096+0000 7fe622fc4640 1 -- 172.21.15.31:0/3786159461 wait complete. 2023-10-30T13:51:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 Processor -- start 2023-10-30T13:51:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 -- start start 2023-10-30T13:51:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe61c0cb8b0 0x7fe61c143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe61c151840 0x7fe61c156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c151c20 0x7fe61c1e4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c155e90 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c143ed0 con 0x7fe61c151840 2023-10-30T13:51:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c1e30c0 con 0x7fe61c151c20 2023-10-30T13:51:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe61c0cb8b0 0x7fe61c143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe61c0cb8b0 0x7fe61c143990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46376/0 (socket says 172.21.15.31:46376) 2023-10-30T13:51:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 -- 172.21.15.31:0/98163205 learned_addr learned my addr 172.21.15.31:0/98163205 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe621561640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c151c20 0x7fe61c1e4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe61bfff640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe61c151840 0x7fe61c156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 -- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c151c20 msgr2=0x7fe61c1e4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c151c20 0x7fe61c1e4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 -- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe61c151840 msgr2=0x7fe61c156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe61c151840 0x7fe61c156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 -- 172.21.15.31:0/98163205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6100094d0 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe620d60640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe61c0cb8b0 0x7fe61c143990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe61000ea00 tx=0x7fe61000eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe6197fa640 1 -- 172.21.15.31:0/98163205 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000ca00 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe6197fa640 1 -- 172.21.15.31:0/98163205 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61000cb60 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe614009a70 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe6197fa640 1 -- 172.21.15.31:0/98163205 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000ced0 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c1e32f0 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.100+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61c070970 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.104+0000 7fe6197fa640 1 -- 172.21.15.31:0/98163205 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe610013470 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.104+0000 7fe6197fa640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6000664a0 0x7fe600068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.104+0000 7fe6197fa640 1 -- 172.21.15.31:0/98163205 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fe610084600 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.104+0000 7fe61bfff640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6000664a0 0x7fe600068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.104+0000 7fe61bfff640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6000664a0 0x7fe600068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0708e0 tx=0x7fe608009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.108+0000 7fe6197fa640 1 -- 172.21.15.31:0/98163205 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe61c070970 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe61c070970 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe6197fa640 1 -- 172.21.15.31:0/98163205 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5708 (secure 0 0 0) 0x7fe61c070970 con 0x7fe61c0cb8b0 2023-10-30T13:51:46.301 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:46.301 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:43.204125+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:46.301 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 75 2023-10-30T13:51:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6000664a0 msgr2=0x7fe600068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6000664a0 0x7fe600068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0708e0 tx=0x7fe608009040 comp rx=0 tx=0).stop 2023-10-30T13:51:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe61c0cb8b0 msgr2=0x7fe61c143990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe61c0cb8b0 0x7fe61c143990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe61000ea00 tx=0x7fe61000eec0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 shutdown_connections 2023-10-30T13:51:46.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6000664a0 0x7fe600068960 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe61c151c20 0x7fe61c1e4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe61c151840 0x7fe61c156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 --2- 172.21.15.31:0/98163205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe61c0cb8b0 0x7fe61c143990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 >> 172.21.15.31:0/98163205 conn(0x7fe61c0cac70 msgr2=0x7fe61c06b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:46.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 shutdown_connections 2023-10-30T13:51:46.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:46.300+0000 7fe622fc4640 1 -- 172.21.15.31:0/98163205 wait complete. 2023-10-30T13:51:46.323 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 75, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 75, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5591": { "addr": "172.21.15.33:6836/3928996203", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 3928996203, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 3928996203, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5591, "incarnation": 72, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:51:43.204125+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5591 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:51:48.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:48.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f9001217640 1 Processor -- start 2023-10-30T13:51:48.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f9001217640 1 -- start start 2023-10-30T13:51:48.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f9001217640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8ffc15ac70 0x7f8ffc15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:48.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f9001217640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc15dcc0 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f9001217640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc159d80 con 0x7f8ffc06aeb0 2023-10-30T13:51:48.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f9001217640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc159800 con 0x7f8ffc15ac70 2023-10-30T13:51:48.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffbfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8ffc0c8eb0 0x7f8ffc15a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49284/0 (socket says 172.21.15.31:49284) 2023-10-30T13:51:48.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffbfff640 1 -- 172.21.15.31:0/4141087264 learned_addr learned my addr 172.21.15.31:0/4141087264 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:48.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffb7fe640 1 --2- 172.21.15.31:0/4141087264 >> v2:172.21.15.33:3300/0 conn(0x7f8ffc15ac70 0x7f8ffc15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:48.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffaffd640 1 -- 172.21.15.31:0/4141087264 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442023466 0 0) 0x7f8fec001120 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffaffd640 1 -- 172.21.15.31:0/4141087264 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe4003530 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffb7fe640 1 -- 172.21.15.31:0/4141087264 >> v1:172.21.15.33:6789/0 conn(0x7f8ffc06aeb0 legacy=0x7f8ffc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffb7fe640 1 -- 172.21.15.31:0/4141087264 >> v1:172.21.15.31:6789/0 conn(0x7f8ffc0c8eb0 legacy=0x7f8ffc15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.480+0000 7f8ffb7fe640 1 -- 172.21.15.31:0/4141087264 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc15d420 con 0x7f8ffc15ac70 2023-10-30T13:51:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f8ffb7fe640 1 --2- 172.21.15.31:0/4141087264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc15cd70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097c0 tx=0x7f8ff001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dd9d8dbdaf30cee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:48.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f8ffaffd640 1 -- 172.21.15.31:0/4141087264 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00a8970 con 0x7f8ffc15ac70 2023-10-30T13:51:48.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f8ffaffd640 1 -- 172.21.15.31:0/4141087264 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff00a8ad0 con 0x7f8ffc15ac70 2023-10-30T13:51:48.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f8ffaffd640 1 -- 172.21.15.31:0/4141087264 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00a8e60 con 0x7f8ffc15ac70 2023-10-30T13:51:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f9001217640 1 -- 172.21.15.31:0/4141087264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc15ac70 msgr2=0x7f8ffc15cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f9001217640 1 --2- 172.21.15.31:0/4141087264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc15cd70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097c0 tx=0x7f8ff001ec70 comp rx=0 tx=0).stop 2023-10-30T13:51:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f9001217640 1 -- 172.21.15.31:0/4141087264 shutdown_connections 2023-10-30T13:51:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f9001217640 1 --2- 172.21.15.31:0/4141087264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc15cd70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f9001217640 1 -- 172.21.15.31:0/4141087264 >> 172.21.15.31:0/4141087264 conn(0x7f8ffc0cef00 msgr2=0x7f8ffc0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f9001217640 1 -- 172.21.15.31:0/4141087264 shutdown_connections 2023-10-30T13:51:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.484+0000 7f9001217640 1 -- 172.21.15.31:0/4141087264 wait complete. 2023-10-30T13:51:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 Processor -- start 2023-10-30T13:51:48.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 -- start start 2023-10-30T13:51:48.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ffc06aeb0 0x7f8ffc1efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc0c8eb0 0x7f8ffc1e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc14d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc1ef620 con 0x7f8ffc15ac70 2023-10-30T13:51:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc1e3f10 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc1e41f0 con 0x7f8ffc06aeb0 2023-10-30T13:51:48.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc0c8eb0 0x7f8ffc1e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:48.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9000a16640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc14d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:48.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffbfff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ffc06aeb0 0x7f8ffc1efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:48.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc0c8eb0 0x7f8ffc1e4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:36152/0 (socket says 172.21.15.31:36152) 2023-10-30T13:51:48.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 -- 172.21.15.31:0/4135401895 learned_addr learned my addr 172.21.15.31:0/4135401895 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:48.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 -- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ffc06aeb0 msgr2=0x7f8ffc1efab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:48.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ffc06aeb0 0x7f8ffc1efab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 -- 172.21.15.31:0/4135401895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ffc15ac70 msgr2=0x7f8ffc14d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:48.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc14d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 -- 172.21.15.31:0/4135401895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff0009420 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9000a16640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:51:48.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ffb7fe640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc0c8eb0 0x7f8ffc1e4f70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ff001b880 tx=0x7f8ff001ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:48.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ff8ff9640 1 -- 172.21.15.31:0/4135401895 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00b1d60 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ff8ff9640 1 -- 172.21.15.31:0/4135401895 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff00b9410 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f8ff8ff9640 1 -- 172.21.15.31:0/4135401895 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00b97c0 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc1e44d0 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.488+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc1efff0 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.492+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc04e170 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.492+0000 7f8ff8ff9640 1 -- 172.21.15.31:0/4135401895 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ff00b7040 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.492+0000 7f8ff8ff9640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fe80664a0 0x7f8fe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:48.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.492+0000 7f8ff8ff9640 1 -- 172.21.15.31:0/4135401895 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f8ff0122750 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.492+0000 7f8ffbfff640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fe80664a0 0x7f8fe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:48.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.492+0000 7f8ffbfff640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fe80664a0 0x7f8fe8068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8fec001280 tx=0x7f8fec007bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:48.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.500+0000 7f8ff8ff9640 1 -- 172.21.15.31:0/4135401895 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ff00efad0 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.688+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ffc06a690 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f8ff8ff9640 1 -- 172.21.15.31:0/4135401895 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5708 (secure 0 0 0) 0x7f8ff00f30f0 con 0x7f8ffc0c8eb0 2023-10-30T13:51:48.692 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:48.692 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:43.204125+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:48.693 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 75 2023-10-30T13:51:48.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fe80664a0 msgr2=0x7f8fe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:48.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fe80664a0 0x7f8fe8068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8fec001280 tx=0x7f8fec007bc0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc0c8eb0 msgr2=0x7f8ffc1e4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:48.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc0c8eb0 0x7f8ffc1e4f70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ff001b880 tx=0x7f8ff001ba10 comp rx=0 tx=0).stop 2023-10-30T13:51:48.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 shutdown_connections 2023-10-30T13:51:48.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8fe80664a0 0x7f8fe8068960 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ffc15ac70 0x7f8ffc14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ffc0c8eb0 0x7f8ffc1e4f70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 --2- 172.21.15.31:0/4135401895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8ffc06aeb0 0x7f8ffc1efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:48.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 >> 172.21.15.31:0/4135401895 conn(0x7f8ffc0cef00 msgr2=0x7f8ffc15ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:48.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 shutdown_connections 2023-10-30T13:51:48.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:48.692+0000 7f9001217640 1 -- 172.21.15.31:0/4135401895 wait complete. 2023-10-30T13:51:48.713 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:50.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:50.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.867+0000 7fba32e17640 1 Processor -- start 2023-10-30T13:51:50.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.867+0000 7fba32e17640 1 -- start start 2023-10-30T13:51:50.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.867+0000 7fba32e17640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fba2c151840 0x7fba2c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:50.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba32e17640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fba2c0cb8b0 0x7fba2c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:50.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba32e17640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c0c5030 con 0x7fba2c151c20 2023-10-30T13:51:50.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba32e17640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2c157830 con 0x7fba2c0cb8b0 2023-10-30T13:51:50.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba32e17640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2c157b10 con 0x7fba2c151840 2023-10-30T13:51:50.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba2bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fba2c151c20 0x7fba2c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49300/0 (socket says 172.21.15.31:49300) 2023-10-30T13:51:50.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba2bfff640 1 -- 172.21.15.31:0/1073171304 learned_addr learned my addr 172.21.15.31:0/1073171304 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:50.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba313b4640 1 --2- 172.21.15.31:0/1073171304 >> v2:172.21.15.31:3300/0 conn(0x7fba2c0cb8b0 0x7fba2c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:50.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba30bb3640 1 --2- 172.21.15.31:0/1073171304 >> v2:172.21.15.33:3300/0 conn(0x7fba2c151840 0x7fba2c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:50.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba2b7fe640 1 -- 172.21.15.31:0/1073171304 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765813178 0 0) 0x7fba2c0c5030 con 0x7fba2c151c20 2023-10-30T13:51:50.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba2b7fe640 1 -- 172.21.15.31:0/1073171304 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba1c003530 con 0x7fba2c151c20 2023-10-30T13:51:50.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba313b4640 1 -- 172.21.15.31:0/1073171304 >> v2:172.21.15.33:3300/0 conn(0x7fba2c151840 msgr2=0x7fba2c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:50.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba313b4640 1 --2- 172.21.15.31:0/1073171304 >> v2:172.21.15.33:3300/0 conn(0x7fba2c151840 0x7fba2c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:50.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba313b4640 1 -- 172.21.15.31:0/1073171304 >> v1:172.21.15.31:6789/0 conn(0x7fba2c151c20 legacy=0x7fba2c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:50.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba313b4640 1 -- 172.21.15.31:0/1073171304 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c157df0 con 0x7fba2c0cb8b0 2023-10-30T13:51:50.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.871+0000 7fba313b4640 1 --2- 172.21.15.31:0/1073171304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c0cb8b0 0x7fba2c06b600 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fba2c072b40 tx=0x7fba2401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125000fc3ba35ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:50.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba2b7fe640 1 -- 172.21.15.31:0/1073171304 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240b0440 con 0x7fba2c0cb8b0 2023-10-30T13:51:50.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba2b7fe640 1 -- 172.21.15.31:0/1073171304 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba240b05a0 con 0x7fba2c0cb8b0 2023-10-30T13:51:50.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba2b7fe640 1 -- 172.21.15.31:0/1073171304 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240b0930 con 0x7fba2c0cb8b0 2023-10-30T13:51:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- 172.21.15.31:0/1073171304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c0cb8b0 msgr2=0x7fba2c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 --2- 172.21.15.31:0/1073171304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c0cb8b0 0x7fba2c06b600 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fba2c072b40 tx=0x7fba2401f0f0 comp rx=0 tx=0).stop 2023-10-30T13:51:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- 172.21.15.31:0/1073171304 shutdown_connections 2023-10-30T13:51:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 --2- 172.21.15.31:0/1073171304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c0cb8b0 0x7fba2c06b600 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 --2- 172.21.15.31:0/1073171304 >> v2:172.21.15.33:3300/0 conn(0x7fba2c151840 0x7fba2c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:50.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- 172.21.15.31:0/1073171304 >> 172.21.15.31:0/1073171304 conn(0x7fba2c0cac70 msgr2=0x7fba2c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:50.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- 172.21.15.31:0/1073171304 shutdown_connections 2023-10-30T13:51:50.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- 172.21.15.31:0/1073171304 wait complete. 2023-10-30T13:51:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 Processor -- start 2023-10-30T13:51:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- start start 2023-10-30T13:51:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba2c0cb8b0 0x7fba2c146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fba2c151840 0x7fba2c14fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c151c20 0x7fba2c1ef560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:50.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c147590 con 0x7fba2c151c20 2023-10-30T13:51:50.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c1ef0d0 con 0x7fba2c0cb8b0 2023-10-30T13:51:50.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba32e17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c0c9010 con 0x7fba2c151840 2023-10-30T13:51:50.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba313b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c151c20 0x7fba2c1ef560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:50.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba313b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c151c20 0x7fba2c1ef560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46402/0 (socket says 172.21.15.31:46402) 2023-10-30T13:51:50.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.875+0000 7fba313b4640 1 -- 172.21.15.31:0/3965998410 learned_addr learned my addr 172.21.15.31:0/3965998410 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:50.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba30bb3640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba2c0cb8b0 0x7fba2c146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:50.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba2bfff640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fba2c151840 0x7fba2c14fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:50.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba313b4640 1 -- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fba2c151840 msgr2=0x7fba2c14fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:50.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba313b4640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fba2c151840 0x7fba2c14fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:50.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba313b4640 1 -- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba2c0cb8b0 msgr2=0x7fba2c146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:50.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba313b4640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba2c0cb8b0 0x7fba2c146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:50.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba313b4640 1 -- 172.21.15.31:0/3965998410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba24009a70 con 0x7fba2c151c20 2023-10-30T13:51:50.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba2bfff640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fba2c151840 0x7fba2c14fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:51:50.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba313b4640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c151c20 0x7fba2c1ef560 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fba2401bd00 tx=0x7fba2401be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:50.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba297fa640 1 -- 172.21.15.31:0/3965998410 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240b0440 con 0x7fba2c151c20 2023-10-30T13:51:50.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba297fa640 1 -- 172.21.15.31:0/3965998410 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba240bc650 con 0x7fba2c151c20 2023-10-30T13:51:50.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba297fa640 1 -- 172.21.15.31:0/3965998410 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240bca00 con 0x7fba2c151c20 2023-10-30T13:51:50.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c1eed80 con 0x7fba2c151c20 2023-10-30T13:51:50.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c0c92b0 con 0x7fba2c151c20 2023-10-30T13:51:50.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2c1ef280 con 0x7fba2c151c20 2023-10-30T13:51:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba297fa640 1 -- 172.21.15.31:0/3965998410 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba240bcb60 con 0x7fba2c151c20 2023-10-30T13:51:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba297fa640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fba100664a0 0x7fba10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.879+0000 7fba297fa640 1 -- 172.21.15.31:0/3965998410 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fba24122d40 con 0x7fba2c151c20 2023-10-30T13:51:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.883+0000 7fba30bb3640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fba100664a0 0x7fba10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:50.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.883+0000 7fba30bb3640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fba100664a0 0x7fba10068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fba20009af0 tx=0x7fba20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:50.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:50.887+0000 7fba297fa640 1 -- 172.21.15.31:0/3965998410 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba2c1ef280 con 0x7fba2c151c20 2023-10-30T13:51:51.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba2c1ef280 con 0x7fba2c151c20 2023-10-30T13:51:51.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba297fa640 1 -- 172.21.15.31:0/3965998410 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fba2c1ef280 con 0x7fba2c151c20 2023-10-30T13:51:51.081 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:51.082 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:51.083 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 76 2023-10-30T13:51:51.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fba100664a0 msgr2=0x7fba10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fba100664a0 0x7fba10068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fba20009af0 tx=0x7fba20008040 comp rx=0 tx=0).stop 2023-10-30T13:51:51.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c151c20 msgr2=0x7fba2c1ef560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c151c20 0x7fba2c1ef560 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fba2401bd00 tx=0x7fba2401be90 comp rx=0 tx=0).stop 2023-10-30T13:51:51.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 shutdown_connections 2023-10-30T13:51:51.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fba100664a0 0x7fba10068960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2c151c20 0x7fba2c1ef560 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fba2c151840 0x7fba2c14fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 --2- 172.21.15.31:0/3965998410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba2c0cb8b0 0x7fba2c146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 >> 172.21.15.31:0/3965998410 conn(0x7fba2c0cac70 msgr2=0x7fba2c0cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:51.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 shutdown_connections 2023-10-30T13:51:51.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:51.079+0000 7fba32e17640 1 -- 172.21.15.31:0/3965998410 wait complete. 2023-10-30T13:51:51.103 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5889 2023-10-30T13:51:51.103 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2023-10-30T13:51:55.259 INFO:tasks.ceph.mds.d.smithi033.stderr:2023-10-30T13:51:55.259+0000 7f0e695c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:51:57.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:57.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:58.104 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:51:58.105 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:51:58.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:51:58.109 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:51:58.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:58.207 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2023-10-30T13:51:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc596f3e640 1 Processor -- start 2023-10-30T13:51:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc596f3e640 1 -- start start 2023-10-30T13:51:58.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc596f3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc59006bd10 0x7fc5900c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc596f3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5900c5030 con 0x7fc590151840 2023-10-30T13:51:58.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc596f3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc590157830 con 0x7fc590151c20 2023-10-30T13:51:58.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc596f3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc590157b10 con 0x7fc59006bd10 2023-10-30T13:51:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc5954db640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc59006bd10 0x7fc5900c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc594cda640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc590151840 0x7fc590072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:39906/0 (socket says 172.21.15.31:39906) 2023-10-30T13:51:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc594cda640 1 -- 172.21.15.31:0/901434077 learned_addr learned my addr 172.21.15.31:0/901434077 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:51:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc5954db640 1 -- 172.21.15.31:0/901434077 >> v1:172.21.15.33:6789/0 conn(0x7fc590151c20 legacy=0x7fc59006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc5954db640 1 -- 172.21.15.31:0/901434077 >> v1:172.21.15.31:6789/0 conn(0x7fc590151840 legacy=0x7fc590072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc5954db640 1 -- 172.21.15.31:0/901434077 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590157df0 con 0x7fc59006bd10 2023-10-30T13:51:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.263+0000 7fc58f7fe640 1 -- 172.21.15.31:0/901434077 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088268531 0 0) 0x7fc5900c5030 con 0x7fc590151840 2023-10-30T13:51:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc5954db640 1 --2- 172.21.15.31:0/901434077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5900c49a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc588009d90 tx=0x7fc58801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91dda99a97df7753 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc58f7fe640 1 -- 172.21.15.31:0/901434077 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5880af600 con 0x7fc59006bd10 2023-10-30T13:51:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc58f7fe640 1 -- 172.21.15.31:0/901434077 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5880ab050 con 0x7fc59006bd10 2023-10-30T13:51:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc58f7fe640 1 -- 172.21.15.31:0/901434077 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5880af980 con 0x7fc59006bd10 2023-10-30T13:51:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc596f3e640 1 -- 172.21.15.31:0/901434077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 msgr2=0x7fc5900c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc596f3e640 1 --2- 172.21.15.31:0/901434077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5900c49a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc588009d90 tx=0x7fc58801f130 comp rx=0 tx=0).stop 2023-10-30T13:51:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc596f3e640 1 -- 172.21.15.31:0/901434077 shutdown_connections 2023-10-30T13:51:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc596f3e640 1 --2- 172.21.15.31:0/901434077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5900c49a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc596f3e640 1 -- 172.21.15.31:0/901434077 >> 172.21.15.31:0/901434077 conn(0x7fc5900cac70 msgr2=0x7fc5900cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc596f3e640 1 -- 172.21.15.31:0/901434077 shutdown_connections 2023-10-30T13:51:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.267+0000 7fc596f3e640 1 -- 172.21.15.31:0/901434077 wait complete. 2023-10-30T13:51:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 Processor -- start 2023-10-30T13:51:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 -- start start 2023-10-30T13:51:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5901ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590151840 0x7fc5900c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc590151c20 0x7fc590071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc590070fa0 con 0x7fc59006bd10 2023-10-30T13:51:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5901f23e0 con 0x7fc590151840 2023-10-30T13:51:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5901f26c0 con 0x7fc590151c20 2023-10-30T13:51:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5901ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5901ee840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60782/0 (socket says 172.21.15.31:60782) 2023-10-30T13:51:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 -- 172.21.15.31:0/1311452980 learned_addr learned my addr 172.21.15.31:0/1311452980 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:51:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc58ffff640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590151840 0x7fc5900c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc5954db640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc590151c20 0x7fc590071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 -- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc590151c20 msgr2=0x7fc590071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc590151c20 0x7fc590071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 -- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590151840 msgr2=0x7fc5900c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590151840 0x7fc5900c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 -- 172.21.15.31:0/1311452980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588009a70 con 0x7fc59006bd10 2023-10-30T13:51:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc58ffff640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590151840 0x7fc5900c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc594cda640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5901ee840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc58400c4a0 tx=0x7fc58400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc58d7fa640 1 -- 172.21.15.31:0/1311452980 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5840074e0 con 0x7fc59006bd10 2023-10-30T13:51:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc58d7fa640 1 -- 172.21.15.31:0/1311452980 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc584007640 con 0x7fc59006bd10 2023-10-30T13:51:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc58d7fa640 1 -- 172.21.15.31:0/1311452980 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584004a30 con 0x7fc59006bd10 2023-10-30T13:51:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5901f29a0 con 0x7fc59006bd10 2023-10-30T13:51:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.271+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc590070c30 con 0x7fc59006bd10 2023-10-30T13:51:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.275+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59004e170 con 0x7fc59006bd10 2023-10-30T13:51:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.275+0000 7fc58d7fa640 1 -- 172.21.15.31:0/1311452980 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc584004b90 con 0x7fc59006bd10 2023-10-30T13:51:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.275+0000 7fc58d7fa640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5740664d0 0x7fc574068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.275+0000 7fc58d7fa640 1 -- 172.21.15.31:0/1311452980 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fc5840833c0 con 0x7fc59006bd10 2023-10-30T13:51:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.275+0000 7fc58ffff640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5740664d0 0x7fc574068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.275+0000 7fc58ffff640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5740664d0 0x7fc574068990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc5900c3d00 tx=0x7fc57c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.279+0000 7fc58d7fa640 1 -- 172.21.15.31:0/1311452980 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc584050470 con 0x7fc59006bd10 2023-10-30T13:51:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.475+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc59006f5f0 con 0x7fc59006bd10 2023-10-30T13:51:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.475+0000 7fc58d7fa640 1 -- 172.21.15.31:0/1311452980 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fc584053a90 con 0x7fc59006bd10 2023-10-30T13:51:58.478 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:51:58.478 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:51:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.475+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5740664d0 msgr2=0x7fc574068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.475+0000 7fc596f3e640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5740664d0 0x7fc574068990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc5900c3d00 tx=0x7fc57c009040 comp rx=0 tx=0).stop 2023-10-30T13:51:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.475+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 msgr2=0x7fc5901ee840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.475+0000 7fc596f3e640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5901ee840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc58400c4a0 tx=0x7fc58400c960 comp rx=0 tx=0).stop 2023-10-30T13:51:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 shutdown_connections 2023-10-30T13:51:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc5740664d0 0x7fc574068990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc590151c20 0x7fc590071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590151840 0x7fc5900c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 --2- 172.21.15.31:0/1311452980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc59006bd10 0x7fc5901ee840 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 >> 172.21.15.31:0/1311452980 conn(0x7fc5900cac70 msgr2=0x7fc590156e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 shutdown_connections 2023-10-30T13:51:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:51:58.479+0000 7fc596f3e640 1 -- 172.21.15.31:0/1311452980 wait complete. 2023-10-30T13:51:58.499 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:51:58.500 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 90 secs before thrashing 2023-10-30T13:52:27.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:27.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:57.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:57.569 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:27.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:27.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:28.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:28.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.650+0000 7fa3a2b3e640 1 Processor -- start 2023-10-30T13:53:28.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.650+0000 7fa3a2b3e640 1 -- start start 2023-10-30T13:53:28.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a2b3e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa39c06b760 0x7fa39c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a2b3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa39c06bb40 0x7fa39c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a2b3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a2b3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c071e00 con 0x7fa39c06bb40 2023-10-30T13:53:28.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a2b3e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c070020 con 0x7fa39c06b760 2023-10-30T13:53:28.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a2b3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c15eb90 con 0x7fa39c06c420 2023-10-30T13:53:28.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa39c06bb40 0x7fa39c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa39c06bb40 0x7fa39c157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45218/0 (socket says 172.21.15.31:45218) 2023-10-30T13:53:28.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 -- 172.21.15.31:0/2184752121 learned_addr learned my addr 172.21.15.31:0/2184752121 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:28.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a08da640 1 --2- 172.21.15.31:0/2184752121 >> v2:172.21.15.33:3300/0 conn(0x7fa39c06b760 0x7fa39c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa3a10db640 1 --2- 172.21.15.31:0/2184752121 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 -- 172.21.15.31:0/2184752121 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 msgr2=0x7fa39c157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:28.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 --2- 172.21.15.31:0/2184752121 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 -- 172.21.15.31:0/2184752121 >> v2:172.21.15.33:3300/0 conn(0x7fa39c06b760 msgr2=0x7fa39c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:28.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 --2- 172.21.15.31:0/2184752121 >> v2:172.21.15.33:3300/0 conn(0x7fa39c06b760 0x7fa39c15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.654+0000 7fa39bfff640 1 -- 172.21.15.31:0/2184752121 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c070f10 con 0x7fa39c06bb40 2023-10-30T13:53:28.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa39bfff640 1 --2- 172.21.15.31:0/2184752121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c157a20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0cb840 tx=0x7fa38801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ba7cd7a9cbacf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:28.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa39b7fe640 1 -- 172.21.15.31:0/2184752121 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3880aa610 con 0x7fa39c06bb40 2023-10-30T13:53:28.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa39b7fe640 1 -- 172.21.15.31:0/2184752121 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3880a6040 con 0x7fa39c06bb40 2023-10-30T13:53:28.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa39b7fe640 1 -- 172.21.15.31:0/2184752121 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3880aa990 con 0x7fa39c06bb40 2023-10-30T13:53:28.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/2184752121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 msgr2=0x7fa39c157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/2184752121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c157a20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0cb840 tx=0x7fa38801c4d0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/2184752121 shutdown_connections 2023-10-30T13:53:28.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/2184752121 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/2184752121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c157a20 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/2184752121 >> v2:172.21.15.33:3300/0 conn(0x7fa39c06b760 0x7fa39c15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/2184752121 >> 172.21.15.31:0/2184752121 conn(0x7fa39c0cac70 msgr2=0x7fa39c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:28.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/2184752121 shutdown_connections 2023-10-30T13:53:28.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/2184752121 wait complete. 2023-10-30T13:53:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 Processor -- start 2023-10-30T13:53:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.658+0000 7fa3a2b3e640 1 -- start start 2023-10-30T13:53:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa39c06b760 0x7fa39c158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c1594d0 con 0x7fa39c06bb40 2023-10-30T13:53:28.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c15b5b0 con 0x7fa39c06b760 2023-10-30T13:53:28.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c1f96c0 con 0x7fa39c06c420 2023-10-30T13:53:28.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45234/0 (socket says 172.21.15.31:45234) 2023-10-30T13:53:28.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 -- 172.21.15.31:0/1559100419 learned_addr learned my addr 172.21.15.31:0/1559100419 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a10db640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a08da640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa39c06b760 0x7fa39c158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 -- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 msgr2=0x7fa39c0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c0ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 -- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa39c06b760 msgr2=0x7fa39c158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa39c06b760 0x7fa39c158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 -- 172.21.15.31:0/1559100419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880094d0 con 0x7fa39c06bb40 2023-10-30T13:53:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa39bfff640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c15ed90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa38801ab80 tx=0x7fa38801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3997fa640 1 -- 172.21.15.31:0/1559100419 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3880aad80 con 0x7fa39c06bb40 2023-10-30T13:53:28.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3997fa640 1 -- 172.21.15.31:0/1559100419 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3880bd3f0 con 0x7fa39c06bb40 2023-10-30T13:53:28.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3997fa640 1 -- 172.21.15.31:0/1559100419 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3880bd780 con 0x7fa39c06bb40 2023-10-30T13:53:28.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c15b010 con 0x7fa39c06bb40 2023-10-30T13:53:28.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c1f9960 con 0x7fa39c06bb40 2023-10-30T13:53:28.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c066510 con 0x7fa39c06bb40 2023-10-30T13:53:28.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.662+0000 7fa3997fa640 1 -- 172.21.15.31:0/1559100419 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3880b33f0 con 0x7fa39c06bb40 2023-10-30T13:53:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.666+0000 7fa3997fa640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3800664a0 0x7fa380068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.666+0000 7fa3997fa640 1 -- 172.21.15.31:0/1559100419 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fa38811dfb0 con 0x7fa39c06bb40 2023-10-30T13:53:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.666+0000 7fa3a08da640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3800664a0 0x7fa380068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.666+0000 7fa3a08da640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3800664a0 0x7fa380068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa39000ba30 tx=0x7fa390008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.670+0000 7fa3997fa640 1 -- 172.21.15.31:0/1559100419 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3880eb220 con 0x7fa39c06bb40 2023-10-30T13:53:28.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.858+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa39c04e170 con 0x7fa39c06bb40 2023-10-30T13:53:28.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3997fa640 1 -- 172.21.15.31:0/1559100419 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fa3880ee840 con 0x7fa39c06bb40 2023-10-30T13:53:28.864 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:28.864 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:28.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3800664a0 msgr2=0x7fa380068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3800664a0 0x7fa380068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa39000ba30 tx=0x7fa390008040 comp rx=0 tx=0).stop 2023-10-30T13:53:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 msgr2=0x7fa39c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c15ed90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa38801ab80 tx=0x7fa38801ad10 comp rx=0 tx=0).stop 2023-10-30T13:53:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 shutdown_connections 2023-10-30T13:53:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3800664a0 0x7fa380068960 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa39c06c420 0x7fa39c0ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa39c06bb40 0x7fa39c15ed90 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 --2- 172.21.15.31:0/1559100419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa39c06b760 0x7fa39c158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 >> 172.21.15.31:0/1559100419 conn(0x7fa39c0cac70 msgr2=0x7fa39c152360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 shutdown_connections 2023-10-30T13:53:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:28.862+0000 7fa3a2b3e640 1 -- 172.21.15.31:0/1559100419 wait complete. 2023-10-30T13:53:28.885 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:53:28.886 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:53:28.886 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:53:28.911 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:53:28.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.050+0000 7f5be8d1d640 1 Processor -- start 2023-10-30T13:53:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.050+0000 7f5be8d1d640 1 -- start start 2023-10-30T13:53:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.050+0000 7f5be8d1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 0x7f5be40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.050+0000 7f5be8d1d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be40c5030 con 0x7f5be4151c20 2023-10-30T13:53:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.050+0000 7f5be8d1d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be4157830 con 0x7f5be40cb8b0 2023-10-30T13:53:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.050+0000 7f5be8d1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4157b10 con 0x7f5be4151840 2023-10-30T13:53:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f5be4151c20 0x7f5be4072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58482/0 (socket says 172.21.15.31:58482) 2023-10-30T13:53:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 -- 172.21.15.31:0/493590585 learned_addr learned my addr 172.21.15.31:0/493590585 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be259c640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 0x7f5be40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182839120 0 0) 0x7f5be40c5030 con 0x7f5be4151c20 2023-10-30T13:53:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bcc003530 con 0x7f5be4151c20 2023-10-30T13:53:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248708490 0 0) 0x7f5bcc003530 con 0x7f5be4151c20 2023-10-30T13:53:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 msgr2=0x7f5be40c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 0x7f5be40c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 >> v1:172.21.15.33:6789/0 conn(0x7f5be40cb8b0 legacy=0x7f5be406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4157df0 con 0x7f5be4151c20 2023-10-30T13:53:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f5bd8001680 con 0x7f5be4151c20 2023-10-30T13:53:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 >> v1:172.21.15.31:6789/0 conn(0x7f5be4151c20 legacy=0x7f5be4072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 0x7f5bcc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc003f30 0x7f5bcc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bcc004d10 0x7f5bcc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4157b10 con 0x7f5bcc003f30 2023-10-30T13:53:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc00bb00 con 0x7f5bcc004d10 2023-10-30T13:53:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc00bcc0 con 0x7f5be4151840 2023-10-30T13:53:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc003f30 0x7f5bcc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be259c640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 0x7f5bcc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be2d9d640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bcc004d10 0x7f5bcc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 -- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 msgr2=0x7f5bcc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5be4151840 0x7f5bcc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 -- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bcc004d10 msgr2=0x7f5bcc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bcc004d10 0x7f5bcc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.054+0000 7f5be1d9b640 1 -- 172.21.15.31:0/493590585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc003530 con 0x7f5bcc003f30 2023-10-30T13:53:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be1d9b640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc003f30 0x7f5bcc0047d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c210 tx=0x7f5bd801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592b4e846fb3e1dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:29.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80a5b20 con 0x7f5bcc003f30 2023-10-30T13:53:29.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd80a5c80 con 0x7f5bcc003f30 2023-10-30T13:53:29.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be159a640 1 -- 172.21.15.31:0/493590585 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8005b20 con 0x7f5bcc003f30 2023-10-30T13:53:29.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 -- 172.21.15.31:0/493590585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc003f30 msgr2=0x7f5bcc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 --2- 172.21.15.31:0/493590585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc003f30 0x7f5bcc0047d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c210 tx=0x7f5bd801cc20 comp rx=0 tx=0).stop 2023-10-30T13:53:29.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be2d9d640 1 -- 172.21.15.31:0/493590585 reap_dead start 2023-10-30T13:53:29.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 -- 172.21.15.31:0/493590585 shutdown_connections 2023-10-30T13:53:29.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 -- 172.21.15.31:0/493590585 >> 172.21.15.31:0/493590585 conn(0x7f5be40cac70 msgr2=0x7f5be40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:29.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 -- 172.21.15.31:0/493590585 shutdown_connections 2023-10-30T13:53:29.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 -- 172.21.15.31:0/493590585 wait complete. 2023-10-30T13:53:29.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 Processor -- start 2023-10-30T13:53:29.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.058+0000 7f5be8d1d640 1 -- start start 2023-10-30T13:53:29.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5bcc003f30 0x7f5be4148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be4071ba0 0x7f5be4149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be414a1f0 0x7f5be41f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4148fb0 con 0x7f5be414a1f0 2023-10-30T13:53:29.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4070d10 con 0x7f5be4071ba0 2023-10-30T13:53:29.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40717a0 con 0x7f5bcc003f30 2023-10-30T13:53:29.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be414a1f0 0x7f5be41f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be414a1f0 0x7f5be41f1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45248/0 (socket says 172.21.15.31:45248) 2023-10-30T13:53:29.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 -- 172.21.15.31:0/3125247373 learned_addr learned my addr 172.21.15.31:0/3125247373 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:29.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be1d9b640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be4071ba0 0x7f5be4149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be259c640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5bcc003f30 0x7f5be4148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 -- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5bcc003f30 msgr2=0x7f5be4148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5bcc003f30 0x7f5be4148a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 -- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be4071ba0 msgr2=0x7f5be4149cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be4071ba0 0x7f5be4149cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 -- 172.21.15.31:0/3125247373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be414a670 con 0x7f5be414a1f0 2023-10-30T13:53:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be2d9d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be414a1f0 0x7f5be41f1f40 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5be40c6db0 tx=0x7f5bdc00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5bd2ffd640 1 -- 172.21.15.31:0/3125247373 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003980 con 0x7f5be414a1f0 2023-10-30T13:53:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5bd2ffd640 1 -- 172.21.15.31:0/3125247373 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc003ae0 con 0x7f5be414a1f0 2023-10-30T13:53:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5bd2ffd640 1 -- 172.21.15.31:0/3125247373 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003e90 con 0x7f5be414a1f0 2023-10-30T13:53:29.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be4070f00 con 0x7f5be414a1f0 2023-10-30T13:53:29.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be40714c0 con 0x7f5be414a1f0 2023-10-30T13:53:29.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be404e170 con 0x7f5be414a1f0 2023-10-30T13:53:29.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.062+0000 7f5bd2ffd640 1 -- 172.21.15.31:0/3125247373 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5bdc0050d0 con 0x7f5be414a1f0 2023-10-30T13:53:29.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.066+0000 7f5bd2ffd640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5bc40664d0 0x7f5bc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:29.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.066+0000 7f5bd2ffd640 1 -- 172.21.15.31:0/3125247373 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f5bdc083fe0 con 0x7f5be414a1f0 2023-10-30T13:53:29.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.066+0000 7f5be259c640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5bc40664d0 0x7f5bc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:29.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.066+0000 7f5be259c640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5bc40664d0 0x7f5bc4068990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009a30 tx=0x7f5bd4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:29.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.070+0000 7f5bd2ffd640 1 -- 172.21.15.31:0/3125247373 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5bdc0510a0 con 0x7f5be414a1f0 2023-10-30T13:53:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.258+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5be414dbc0 con 0x7f5be414a1f0 2023-10-30T13:53:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5bd2ffd640 1 -- 172.21.15.31:0/3125247373 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f5bdc0546c0 con 0x7f5be414a1f0 2023-10-30T13:53:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:29.264 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:29.264 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:29.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5bc40664d0 msgr2=0x7f5bc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5bc40664d0 0x7f5bc4068990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009a30 tx=0x7f5bd4008040 comp rx=0 tx=0).stop 2023-10-30T13:53:29.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be414a1f0 msgr2=0x7f5be41f1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be414a1f0 0x7f5be41f1f40 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5be40c6db0 tx=0x7f5bdc00ec10 comp rx=0 tx=0).stop 2023-10-30T13:53:29.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 shutdown_connections 2023-10-30T13:53:29.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5bc40664d0 0x7f5bc4068990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be414a1f0 0x7f5be41f1f40 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be4071ba0 0x7f5be4149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 --2- 172.21.15.31:0/3125247373 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5bcc003f30 0x7f5be4148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 >> 172.21.15.31:0/3125247373 conn(0x7f5be40cac70 msgr2=0x7f5be414c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 shutdown_connections 2023-10-30T13:53:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:29.262+0000 7f5be8d1d640 1 -- 172.21.15.31:0/3125247373 wait complete. 2023-10-30T13:53:29.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:31.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:31.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.438+0000 7f229189c640 1 Processor -- start 2023-10-30T13:53:31.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.438+0000 7f229189c640 1 -- start start 2023-10-30T13:53:31.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f229189c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f228c0c8f00 0x7f228c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f229189c640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f228c0cbc30 0x7f228c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f229189c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0b51a0 con 0x7f228c0cf610 2023-10-30T13:53:31.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f229189c640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c15b580 con 0x7f228c0c8f00 2023-10-30T13:53:31.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f229189c640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c0b4ba0 con 0x7f228c0cbc30 2023-10-30T13:53:31.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f228affd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f228c0c8f00 0x7f228c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f228affd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f228c0c8f00 0x7f228c0cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45256/0 (socket says 172.21.15.31:45256) 2023-10-30T13:53:31.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f228affd640 1 -- 172.21.15.31:0/1215009555 learned_addr learned my addr 172.21.15.31:0/1215009555 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:31.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f2283fff640 1 --2- 172.21.15.31:0/1215009555 >> v2:172.21.15.33:3300/0 conn(0x7f228c0cbc30 0x7f228c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f228affd640 1 -- 172.21.15.31:0/1215009555 >> v2:172.21.15.33:3300/0 conn(0x7f228c0cbc30 msgr2=0x7f228c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f228affd640 1 --2- 172.21.15.31:0/1215009555 >> v2:172.21.15.33:3300/0 conn(0x7f228c0cbc30 0x7f228c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f228affd640 1 -- 172.21.15.31:0/1215009555 >> v1:172.21.15.33:6789/0 conn(0x7f228c0cf610 legacy=0x7f228c15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.442+0000 7f228affd640 1 -- 172.21.15.31:0/1215009555 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0cc150 con 0x7f228c0c8f00 2023-10-30T13:53:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f228affd640 1 --2- 172.21.15.31:0/1215009555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0c8f00 0x7f228c0cb6f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f228c08c150 tx=0x7f227c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b059fb9a05c54a99 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f228a7fc640 1 -- 172.21.15.31:0/1215009555 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0ac600 con 0x7f228c0c8f00 2023-10-30T13:53:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f228a7fc640 1 -- 172.21.15.31:0/1215009555 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c0a8050 con 0x7f228c0c8f00 2023-10-30T13:53:31.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f228a7fc640 1 -- 172.21.15.31:0/1215009555 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0ac980 con 0x7f228c0c8f00 2023-10-30T13:53:31.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 -- 172.21.15.31:0/1215009555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0c8f00 msgr2=0x7f228c0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 --2- 172.21.15.31:0/1215009555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0c8f00 0x7f228c0cb6f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f228c08c150 tx=0x7f227c01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 -- 172.21.15.31:0/1215009555 shutdown_connections 2023-10-30T13:53:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 --2- 172.21.15.31:0/1215009555 >> v2:172.21.15.33:3300/0 conn(0x7f228c0cbc30 0x7f228c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 --2- 172.21.15.31:0/1215009555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0c8f00 0x7f228c0cb6f0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 -- 172.21.15.31:0/1215009555 >> 172.21.15.31:0/1215009555 conn(0x7f228c06c0c0 msgr2=0x7f228c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:31.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 -- 172.21.15.31:0/1215009555 shutdown_connections 2023-10-30T13:53:31.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 -- 172.21.15.31:0/1215009555 wait complete. 2023-10-30T13:53:31.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 Processor -- start 2023-10-30T13:53:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.446+0000 7f229189c640 1 -- start start 2023-10-30T13:53:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f228c0c8f00 0x7f228c0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f228c0cbc30 0x7f228c0b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0cf610 0x7f228c1f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0ce300 con 0x7f228c0cf610 2023-10-30T13:53:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c1f2020 con 0x7f228c0c8f00 2023-10-30T13:53:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c154300 con 0x7f228c0cbc30 2023-10-30T13:53:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0cf610 0x7f228c1f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f228c0c8f00 0x7f228c0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0cf610 0x7f228c1f2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45260/0 (socket says 172.21.15.31:45260) 2023-10-30T13:53:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 -- 172.21.15.31:0/3645019419 learned_addr learned my addr 172.21.15.31:0/3645019419 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f2283fff640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f228c0cbc30 0x7f228c0b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 -- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f228c0cbc30 msgr2=0x7f228c0b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f228c0cbc30 0x7f228c0b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 -- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f228c0c8f00 msgr2=0x7f228c0cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f228c0c8f00 0x7f228c0cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 -- 172.21.15.31:0/3645019419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c0094d0 con 0x7f228c0cf610 2023-10-30T13:53:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f228b7fe640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0cf610 0x7f228c1f2470 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f228400c780 tx=0x7f228400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f2288ff9640 1 -- 172.21.15.31:0/3645019419 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284004b70 con 0x7f228c0cf610 2023-10-30T13:53:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f2288ff9640 1 -- 172.21.15.31:0/3645019419 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228400ce90 con 0x7f228c0cf610 2023-10-30T13:53:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f2288ff9640 1 -- 172.21.15.31:0/3645019419 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228400a850 con 0x7f228c0cf610 2023-10-30T13:53:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c1545c0 con 0x7f228c0cf610 2023-10-30T13:53:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c154b80 con 0x7f228c0cf610 2023-10-30T13:53:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c1f2190 con 0x7f228c0cf610 2023-10-30T13:53:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.450+0000 7f2288ff9640 1 -- 172.21.15.31:0/3645019419 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f228400a9b0 con 0x7f228c0cf610 2023-10-30T13:53:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.454+0000 7f2288ff9640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f226c0664d0 0x7f226c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.454+0000 7f2288ff9640 1 -- 172.21.15.31:0/3645019419 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f22840841c0 con 0x7f228c0cf610 2023-10-30T13:53:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.454+0000 7f228affd640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f226c0664d0 0x7f226c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.454+0000 7f228affd640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f226c0664d0 0x7f226c068990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f227c0ae000 tx=0x7f227c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.458+0000 7f2288ff9640 1 -- 172.21.15.31:0/3645019419 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f228c1f2190 con 0x7f228c0cf610 2023-10-30T13:53:31.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f228c1f2190 con 0x7f228c0cf610 2023-10-30T13:53:31.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f2288ff9640 1 -- 172.21.15.31:0/3645019419 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f228c1f2190 con 0x7f228c0cf610 2023-10-30T13:53:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:31.650 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:31.651 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:31.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f226c0664d0 msgr2=0x7f226c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f226c0664d0 0x7f226c068990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f227c0ae000 tx=0x7f227c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:53:31.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0cf610 msgr2=0x7f228c1f2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0cf610 0x7f228c1f2470 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f228400c780 tx=0x7f228400cc40 comp rx=0 tx=0).stop 2023-10-30T13:53:31.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 shutdown_connections 2023-10-30T13:53:31.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f226c0664d0 0x7f226c068990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c0cf610 0x7f228c1f2470 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f228c0cbc30 0x7f228c0b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 --2- 172.21.15.31:0/3645019419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f228c0c8f00 0x7f228c0cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 >> 172.21.15.31:0/3645019419 conn(0x7f228c06c0c0 msgr2=0x7f228c0c9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:31.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 shutdown_connections 2023-10-30T13:53:31.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:31.646+0000 7f229189c640 1 -- 172.21.15.31:0/3645019419 wait complete. 2023-10-30T13:53:31.670 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:33.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:33.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c83394640 1 Processor -- start 2023-10-30T13:53:33.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c83394640 1 -- start start 2023-10-30T13:53:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c83394640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c150830 0x7f2c7c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c83394640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2c7c070520 0x7f2c7c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c83394640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0c2310 con 0x7f2c7c0cb7d0 2023-10-30T13:53:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c83394640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c156710 con 0x7f2c7c070520 2023-10-30T13:53:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c83394640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0c4360 con 0x7f2c7c150830 2023-10-30T13:53:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c81931640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2c7c070520 0x7f2c7c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c8092f640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2c7c0cb7d0 0x7f2c7c0c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58484/0 (socket says 172.21.15.31:58484) 2023-10-30T13:53:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c8092f640 1 -- 172.21.15.31:0/3417249848 learned_addr learned my addr 172.21.15.31:0/3417249848 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c81130640 1 --2- 172.21.15.31:0/3417249848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c150830 0x7f2c7c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:33.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c81931640 1 -- 172.21.15.31:0/3417249848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c150830 msgr2=0x7f2c7c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:33.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c81931640 1 --2- 172.21.15.31:0/3417249848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c150830 0x7f2c7c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:33.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c81931640 1 -- 172.21.15.31:0/3417249848 >> v1:172.21.15.31:6789/0 conn(0x7f2c7c0cb7d0 legacy=0x7f2c7c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:33.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c73fff640 1 -- 172.21.15.31:0/3417249848 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419255515 0 0) 0x7f2c7c0c2310 con 0x7f2c7c0cb7d0 2023-10-30T13:53:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.822+0000 7f2c81931640 1 -- 172.21.15.31:0/3417249848 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c1569b0 con 0x7f2c7c070520 2023-10-30T13:53:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c81931640 1 --2- 172.21.15.31:0/3417249848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c070520 0x7f2c7c0709b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009d90 tx=0x7f2c74021340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59302b596e1cb674 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c73fff640 1 -- 172.21.15.31:0/3417249848 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c740b1600 con 0x7f2c7c070520 2023-10-30T13:53:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c73fff640 1 -- 172.21.15.31:0/3417249848 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c740ad050 con 0x7f2c7c070520 2023-10-30T13:53:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c73fff640 1 -- 172.21.15.31:0/3417249848 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c740b1980 con 0x7f2c7c070520 2023-10-30T13:53:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 -- 172.21.15.31:0/3417249848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c070520 msgr2=0x7f2c7c0709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 --2- 172.21.15.31:0/3417249848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c070520 0x7f2c7c0709b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009d90 tx=0x7f2c74021340 comp rx=0 tx=0).stop 2023-10-30T13:53:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 -- 172.21.15.31:0/3417249848 shutdown_connections 2023-10-30T13:53:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 --2- 172.21.15.31:0/3417249848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c070520 0x7f2c7c0709b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 --2- 172.21.15.31:0/3417249848 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c150830 0x7f2c7c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 -- 172.21.15.31:0/3417249848 >> 172.21.15.31:0/3417249848 conn(0x7f2c7c0cac70 msgr2=0x7f2c7c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 -- 172.21.15.31:0/3417249848 shutdown_connections 2023-10-30T13:53:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.826+0000 7f2c83394640 1 -- 172.21.15.31:0/3417249848 wait complete. 2023-10-30T13:53:33.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 Processor -- start 2023-10-30T13:53:33.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 -- start start 2023-10-30T13:53:33.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c7c070520 0x7f2c7c1f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:33.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c0cb7d0 0x7f2c7c14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:33.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c150830 0x7f2c7c1e4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:33.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c1440d0 con 0x7f2c7c150830 2023-10-30T13:53:33.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c1eea50 con 0x7f2c7c070520 2023-10-30T13:53:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c1f2eb0 con 0x7f2c7c0cb7d0 2023-10-30T13:53:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c150830 0x7f2c7c1e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c150830 0x7f2c7c1e4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45290/0 (socket says 172.21.15.31:45290) 2023-10-30T13:53:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 -- 172.21.15.31:0/2307700161 learned_addr learned my addr 172.21.15.31:0/2307700161 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c8092f640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c0cb7d0 0x7f2c7c14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81130640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c7c070520 0x7f2c7c1f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:33.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 -- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c0cb7d0 msgr2=0x7f2c7c14fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:33.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c0cb7d0 0x7f2c7c14fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:33.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 -- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c7c070520 msgr2=0x7f2c7c1f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:33.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c7c070520 0x7f2c7c1f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:33.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 -- 172.21.15.31:0/2307700161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c74009a70 con 0x7f2c7c150830 2023-10-30T13:53:33.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c81931640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c150830 0x7f2c7c1e4730 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2c740b4040 tx=0x7f2c7401e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:33.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c71ffb640 1 -- 172.21.15.31:0/2307700161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c740b1c50 con 0x7f2c7c150830 2023-10-30T13:53:33.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c71ffb640 1 -- 172.21.15.31:0/2307700161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c740b1db0 con 0x7f2c7c150830 2023-10-30T13:53:33.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c71ffb640 1 -- 172.21.15.31:0/2307700161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c740bc690 con 0x7f2c7c150830 2023-10-30T13:53:33.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c143270 con 0x7f2c7c150830 2023-10-30T13:53:33.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.830+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c143830 con 0x7f2c7c150830 2023-10-30T13:53:33.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.834+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c066510 con 0x7f2c7c150830 2023-10-30T13:53:33.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.834+0000 7f2c71ffb640 1 -- 172.21.15.31:0/2307700161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c740b8070 con 0x7f2c7c150830 2023-10-30T13:53:33.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.834+0000 7f2c71ffb640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c600664a0 0x7f2c60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:33.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.834+0000 7f2c71ffb640 1 -- 172.21.15.31:0/2307700161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f2c740aae40 con 0x7f2c7c150830 2023-10-30T13:53:33.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.834+0000 7f2c81130640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c600664a0 0x7f2c60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:33.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.834+0000 7f2c81130640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c600664a0 0x7f2c60068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009a50 tx=0x7f2c78008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:33.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:33.838+0000 7f2c71ffb640 1 -- 172.21.15.31:0/2307700161 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c740efe20 con 0x7f2c7c150830 2023-10-30T13:53:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c7c04e170 con 0x7f2c7c150830 2023-10-30T13:53:34.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c71ffb640 1 -- 172.21.15.31:0/2307700161 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f2c740b6020 con 0x7f2c7c150830 2023-10-30T13:53:34.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:34.037 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:34.037 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:34.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c600664a0 msgr2=0x7f2c60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:34.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c600664a0 0x7f2c60068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009a50 tx=0x7f2c78008040 comp rx=0 tx=0).stop 2023-10-30T13:53:34.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c150830 msgr2=0x7f2c7c1e4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:34.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c150830 0x7f2c7c1e4730 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2c740b4040 tx=0x7f2c7401e0e0 comp rx=0 tx=0).stop 2023-10-30T13:53:34.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 shutdown_connections 2023-10-30T13:53:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2c600664a0 0x7f2c60068960 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c7c150830 0x7f2c7c1e4730 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2c7c0cb7d0 0x7f2c7c14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 --2- 172.21.15.31:0/2307700161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c7c070520 0x7f2c7c1f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 >> 172.21.15.31:0/2307700161 conn(0x7f2c7c0cac70 msgr2=0x7f2c7c0c5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:34.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 shutdown_connections 2023-10-30T13:53:34.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:34.034+0000 7f2c83394640 1 -- 172.21.15.31:0/2307700161 wait complete. 2023-10-30T13:53:34.058 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:36.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:36.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f7078a46640 1 Processor -- start 2023-10-30T13:53:36.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f7078a46640 1 -- start start 2023-10-30T13:53:36.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f7078a46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7074150830 0x7f707406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:36.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f7078a46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f70740cb7d0 0x7f70740c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:36.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f7078a46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70740c2310 con 0x7f70740c3e40 2023-10-30T13:53:36.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f7078a46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7074156710 con 0x7f70740cb7d0 2023-10-30T13:53:36.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f7078a46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70740c4360 con 0x7f7074150830 2023-10-30T13:53:36.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f706bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f70740cb7d0 0x7f70740c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:36.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f706bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f70740cb7d0 0x7f70740c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41428/0 (socket says 172.21.15.31:41428) 2023-10-30T13:53:36.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f706bfff640 1 -- 172.21.15.31:0/3440715236 learned_addr learned my addr 172.21.15.31:0/3440715236 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:36.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f707259c640 1 --2- 172.21.15.31:0/3440715236 >> v2:172.21.15.33:3300/0 conn(0x7f7074150830 0x7f707406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:36.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f706bfff640 1 -- 172.21.15.31:0/3440715236 >> v2:172.21.15.33:3300/0 conn(0x7f7074150830 msgr2=0x7f707406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:36.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f706bfff640 1 --2- 172.21.15.31:0/3440715236 >> v2:172.21.15.33:3300/0 conn(0x7f7074150830 0x7f707406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f706bfff640 1 -- 172.21.15.31:0/3440715236 >> v1:172.21.15.33:6789/0 conn(0x7f70740c3e40 legacy=0x7f7074070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:36.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.210+0000 7f706bfff640 1 -- 172.21.15.31:0/3440715236 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70741569b0 con 0x7f70740cb7d0 2023-10-30T13:53:36.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f706bfff640 1 --2- 172.21.15.31:0/3440715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740cb7d0 0x7f70740c3900 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f707406fa40 tx=0x7f705c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e86ed943ebd02c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:36.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7071d9b640 1 -- 172.21.15.31:0/3440715236 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0a89f0 con 0x7f70740cb7d0 2023-10-30T13:53:36.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7071d9b640 1 -- 172.21.15.31:0/3440715236 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c0a8b50 con 0x7f70740cb7d0 2023-10-30T13:53:36.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7071d9b640 1 -- 172.21.15.31:0/3440715236 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0b0440 con 0x7f70740cb7d0 2023-10-30T13:53:36.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 -- 172.21.15.31:0/3440715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740cb7d0 msgr2=0x7f70740c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:36.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 --2- 172.21.15.31:0/3440715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740cb7d0 0x7f70740c3900 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f707406fa40 tx=0x7f705c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 -- 172.21.15.31:0/3440715236 shutdown_connections 2023-10-30T13:53:36.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 --2- 172.21.15.31:0/3440715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740cb7d0 0x7f70740c3900 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 --2- 172.21.15.31:0/3440715236 >> v2:172.21.15.33:3300/0 conn(0x7f7074150830 0x7f707406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 -- 172.21.15.31:0/3440715236 >> 172.21.15.31:0/3440715236 conn(0x7f70740cac70 msgr2=0x7f70740cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:36.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 -- 172.21.15.31:0/3440715236 shutdown_connections 2023-10-30T13:53:36.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.214+0000 7f7078a46640 1 -- 172.21.15.31:0/3440715236 wait complete. 2023-10-30T13:53:36.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 Processor -- start 2023-10-30T13:53:36.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 -- start start 2023-10-30T13:53:36.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740c3e40 0x7f70741561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:36.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70740cb7d0 0x7f70741f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:36.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7074150830 0x7f70741555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:36.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70741f0c60 con 0x7f70740c3e40 2023-10-30T13:53:36.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7074152eb0 con 0x7f7074150830 2023-10-30T13:53:36.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707414f160 con 0x7f70740cb7d0 2023-10-30T13:53:36.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740c3e40 0x7f70741561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:36.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740c3e40 0x7f70741561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41442/0 (socket says 172.21.15.31:41442) 2023-10-30T13:53:36.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 -- 172.21.15.31:0/4177946311 learned_addr learned my addr 172.21.15.31:0/4177946311 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:36.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f706bfff640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70740cb7d0 0x7f70741f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:36.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7072d9d640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7074150830 0x7f70741555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:36.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 -- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70740cb7d0 msgr2=0x7f70741f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:36.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70740cb7d0 0x7f70741f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 -- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7074150830 msgr2=0x7f70741555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:36.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7074150830 0x7f70741555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 -- 172.21.15.31:0/4177946311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c0094d0 con 0x7f70740c3e40 2023-10-30T13:53:36.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f707259c640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740c3e40 0x7f70741561a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f706400e870 tx=0x7f706400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:36.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f706affd640 1 -- 172.21.15.31:0/4177946311 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064009510 con 0x7f70740c3e40 2023-10-30T13:53:36.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f706affd640 1 -- 172.21.15.31:0/4177946311 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706400eec0 con 0x7f70740c3e40 2023-10-30T13:53:36.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7074153170 con 0x7f70740c3e40 2023-10-30T13:53:36.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f706affd640 1 -- 172.21.15.31:0/4177946311 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706400cb40 con 0x7f70740c3e40 2023-10-30T13:53:36.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.218+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7074153730 con 0x7f70740c3e40 2023-10-30T13:53:36.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.222+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7074153920 con 0x7f70740c3e40 2023-10-30T13:53:36.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.222+0000 7f706affd640 1 -- 172.21.15.31:0/4177946311 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f706400cca0 con 0x7f70740c3e40 2023-10-30T13:53:36.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.222+0000 7f706affd640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70540664f0 0x7f70540689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.222+0000 7f706affd640 1 -- 172.21.15.31:0/4177946311 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f7064084310 con 0x7f70740c3e40 2023-10-30T13:53:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.222+0000 7f706bfff640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70540664f0 0x7f70540689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.222+0000 7f706bfff640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70540664f0 0x7f70540689b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f705c0b2000 tx=0x7f705c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:36.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.226+0000 7f706affd640 1 -- 172.21.15.31:0/4177946311 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7064051430 con 0x7f70740c3e40 2023-10-30T13:53:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f707404e170 con 0x7f70740c3e40 2023-10-30T13:53:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f706affd640 1 -- 172.21.15.31:0/4177946311 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f7064054a50 con 0x7f70740c3e40 2023-10-30T13:53:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:36.422 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70540664f0 msgr2=0x7f70540689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70540664f0 0x7f70540689b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f705c0b2000 tx=0x7f705c008040 comp rx=0 tx=0).stop 2023-10-30T13:53:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740c3e40 msgr2=0x7f70741561a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740c3e40 0x7f70741561a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f706400e870 tx=0x7f706400ec70 comp rx=0 tx=0).stop 2023-10-30T13:53:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 shutdown_connections 2023-10-30T13:53:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f70540664f0 0x7f70540689b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7074150830 0x7f70741555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f70740cb7d0 0x7f70741f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 --2- 172.21.15.31:0/4177946311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70740c3e40 0x7f70741561a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 >> 172.21.15.31:0/4177946311 conn(0x7f70740cac70 msgr2=0x7f7074070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 shutdown_connections 2023-10-30T13:53:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:36.418+0000 7f7078a46640 1 -- 172.21.15.31:0/4177946311 wait complete. 2023-10-30T13:53:36.443 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:38.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.594+0000 7fe76e18d640 1 Processor -- start 2023-10-30T13:53:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.594+0000 7fe76e18d640 1 -- start start 2023-10-30T13:53:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe76e18d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe7680cb7d0 0x7fe7680c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe76e18d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680c2310 con 0x7fe768150830 2023-10-30T13:53:38.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe76e18d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe768156710 con 0x7fe7680c3e40 2023-10-30T13:53:38.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe76e18d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76806bc10 con 0x7fe7680cb7d0 2023-10-30T13:53:38.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe7677fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe768150830 0x7fe76806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:59226/0 (socket says 172.21.15.31:59226) 2023-10-30T13:53:38.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe7677fe640 1 -- 172.21.15.31:0/873028153 learned_addr learned my addr 172.21.15.31:0/873028153 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:38.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe766ffd640 1 --2- 172.21.15.31:0/873028153 >> v2:172.21.15.31:3300/0 conn(0x7fe7680cb7d0 0x7fe7680c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe7667fc640 1 -- 172.21.15.31:0/873028153 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762572975 0 0) 0x7fe7680c2310 con 0x7fe768150830 2023-10-30T13:53:38.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe7667fc640 1 -- 172.21.15.31:0/873028153 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe750003530 con 0x7fe768150830 2023-10-30T13:53:38.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe766ffd640 1 -- 172.21.15.31:0/873028153 >> v1:172.21.15.33:6789/0 conn(0x7fe7680c3e40 legacy=0x7fe768070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe766ffd640 1 -- 172.21.15.31:0/873028153 >> v1:172.21.15.31:6789/0 conn(0x7fe768150830 legacy=0x7fe76806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.598+0000 7fe766ffd640 1 -- 172.21.15.31:0/873028153 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680c4360 con 0x7fe7680cb7d0 2023-10-30T13:53:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe766ffd640 1 --2- 172.21.15.31:0/873028153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680cb7d0 0x7fe7680c3900 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0097c0 tx=0x7fe75c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47ef8fd26213d0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe7667fc640 1 -- 172.21.15.31:0/873028153 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0a8970 con 0x7fe7680cb7d0 2023-10-30T13:53:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe7667fc640 1 -- 172.21.15.31:0/873028153 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c0a8ad0 con 0x7fe7680cb7d0 2023-10-30T13:53:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe7667fc640 1 -- 172.21.15.31:0/873028153 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0a8e60 con 0x7fe7680cb7d0 2023-10-30T13:53:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 -- 172.21.15.31:0/873028153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680cb7d0 msgr2=0x7fe7680c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 --2- 172.21.15.31:0/873028153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680cb7d0 0x7fe7680c3900 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0097c0 tx=0x7fe75c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:53:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 -- 172.21.15.31:0/873028153 shutdown_connections 2023-10-30T13:53:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 --2- 172.21.15.31:0/873028153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680cb7d0 0x7fe7680c3900 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 -- 172.21.15.31:0/873028153 >> 172.21.15.31:0/873028153 conn(0x7fe7680cac70 msgr2=0x7fe7680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 -- 172.21.15.31:0/873028153 shutdown_connections 2023-10-30T13:53:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 -- 172.21.15.31:0/873028153 wait complete. 2023-10-30T13:53:38.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 Processor -- start 2023-10-30T13:53:38.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.602+0000 7fe76e18d640 1 -- start start 2023-10-30T13:53:38.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680c3e40 0x7fe7680c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7680cb7d0 0x7fe76815a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe768150830 0x7fe768152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681ed5a0 con 0x7fe7680c3e40 2023-10-30T13:53:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681ec6a0 con 0x7fe7680cb7d0 2023-10-30T13:53:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681ec920 con 0x7fe768150830 2023-10-30T13:53:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680c3e40 0x7fe7680c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680c3e40 0x7fe7680c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41470/0 (socket says 172.21.15.31:41470) 2023-10-30T13:53:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 -- 172.21.15.31:0/690822125 learned_addr learned my addr 172.21.15.31:0/690822125 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe766ffd640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7680cb7d0 0x7fe76815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe767fff640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe768150830 0x7fe768152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 -- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe768150830 msgr2=0x7fe768152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe768150830 0x7fe768152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 -- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7680cb7d0 msgr2=0x7fe76815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7680cb7d0 0x7fe76815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 -- 172.21.15.31:0/690822125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c009420 con 0x7fe7680c3e40 2023-10-30T13:53:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe7677fe640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680c3e40 0x7fe7680c5850 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c420 tx=0x7fe75800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe74ffff640 1 -- 172.21.15.31:0/690822125 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758005cf0 con 0x7fe7680c3e40 2023-10-30T13:53:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe74ffff640 1 -- 172.21.15.31:0/690822125 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758005e50 con 0x7fe7680c3e40 2023-10-30T13:53:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7681ecc00 con 0x7fe7680c3e40 2023-10-30T13:53:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe74ffff640 1 -- 172.21.15.31:0/690822125 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580072d0 con 0x7fe7680c3e40 2023-10-30T13:53:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7681ed1c0 con 0x7fe7680c3e40 2023-10-30T13:53:38.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768152660 con 0x7fe7680c3e40 2023-10-30T13:53:38.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.606+0000 7fe74ffff640 1 -- 172.21.15.31:0/690822125 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe758007450 con 0x7fe7680c3e40 2023-10-30T13:53:38.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.610+0000 7fe74ffff640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7540664a0 0x7fe754068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.610+0000 7fe74ffff640 1 -- 172.21.15.31:0/690822125 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fe758083880 con 0x7fe7680c3e40 2023-10-30T13:53:38.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.610+0000 7fe766ffd640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7540664a0 0x7fe754068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.610+0000 7fe766ffd640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7540664a0 0x7fe754068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0b2000 tx=0x7fe75c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:38.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.614+0000 7fe74ffff640 1 -- 172.21.15.31:0/690822125 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe758054480 con 0x7fe7680c3e40 2023-10-30T13:53:38.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.802+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe76804e170 con 0x7fe7680c3e40 2023-10-30T13:53:38.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.802+0000 7fe74ffff640 1 -- 172.21.15.31:0/690822125 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fe758017020 con 0x7fe7680c3e40 2023-10-30T13:53:38.807 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:38.807 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7540664a0 msgr2=0x7fe754068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7540664a0 0x7fe754068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0b2000 tx=0x7fe75c008040 comp rx=0 tx=0).stop 2023-10-30T13:53:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680c3e40 msgr2=0x7fe7680c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680c3e40 0x7fe7680c5850 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c420 tx=0x7fe75800c8e0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 shutdown_connections 2023-10-30T13:53:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe7540664a0 0x7fe754068960 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe768150830 0x7fe768152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7680cb7d0 0x7fe76815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 --2- 172.21.15.31:0/690822125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7680c3e40 0x7fe7680c5850 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 >> 172.21.15.31:0/690822125 conn(0x7fe7680cac70 msgr2=0x7fe7680c66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 shutdown_connections 2023-10-30T13:53:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:38.806+0000 7fe76e18d640 1 -- 172.21.15.31:0/690822125 wait complete. 2023-10-30T13:53:38.827 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:40.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.978+0000 7f1607772640 1 Processor -- start 2023-10-30T13:53:40.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1607772640 1 -- start start 2023-10-30T13:53:40.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1607772640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f1600151c20 0x7f16000c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:40.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1607772640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16000c5030 con 0x7f16000cb8b0 2023-10-30T13:53:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1607772640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1600157830 con 0x7f1600151840 2023-10-30T13:53:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1607772640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1600157b10 con 0x7f1600151c20 2023-10-30T13:53:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1605d0f640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f16000cb8b0 0x7f160006b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:59232/0 (socket says 172.21.15.31:59232) 2023-10-30T13:53:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1605d0f640 1 -- 172.21.15.31:0/4052756602 learned_addr learned my addr 172.21.15.31:0/4052756602 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1604d0d640 1 --2- 172.21.15.31:0/4052756602 >> v2:172.21.15.33:3300/0 conn(0x7f1600151c20 0x7f16000c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161444818 0 0) 0x7f16000c5030 con 0x7f16000cb8b0 2023-10-30T13:53:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec003530 con 0x7f16000cb8b0 2023-10-30T13:53:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358285267 0 0) 0x7f1600157830 con 0x7f1600151840 2023-10-30T13:53:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16000c5030 con 0x7f1600151840 2023-10-30T13:53:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1604d0d640 1 -- 172.21.15.31:0/4052756602 >> v1:172.21.15.33:6789/0 conn(0x7f1600151840 legacy=0x7f1600072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1511884311 0 0) 0x7f15ec003530 con 0x7f16000cb8b0 2023-10-30T13:53:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1604d0d640 1 -- 172.21.15.31:0/4052756602 >> v1:172.21.15.31:6789/0 conn(0x7f16000cb8b0 legacy=0x7f160006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.982+0000 7f1604d0d640 1 -- 172.21.15.31:0/4052756602 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1600157df0 con 0x7f1600151c20 2023-10-30T13:53:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1604d0d640 1 --2- 172.21.15.31:0/4052756602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 0x7f16000c49a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f15f00097c0 tx=0x7f15f001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2b5253e7a6f17c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8001a00 con 0x7f1600151c20 2023-10-30T13:53:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f00a8610 con 0x7f1600151c20 2023-10-30T13:53:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f15f7fff640 1 -- 172.21.15.31:0/4052756602 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f00a89a0 con 0x7f1600151c20 2023-10-30T13:53:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1607772640 1 -- 172.21.15.31:0/4052756602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 msgr2=0x7f16000c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1607772640 1 --2- 172.21.15.31:0/4052756602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 0x7f16000c49a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f15f00097c0 tx=0x7f15f001eb90 comp rx=0 tx=0).stop 2023-10-30T13:53:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1607772640 1 -- 172.21.15.31:0/4052756602 shutdown_connections 2023-10-30T13:53:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1607772640 1 --2- 172.21.15.31:0/4052756602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 0x7f16000c49a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1607772640 1 -- 172.21.15.31:0/4052756602 >> 172.21.15.31:0/4052756602 conn(0x7f16000cac70 msgr2=0x7f16000cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1607772640 1 -- 172.21.15.31:0/4052756602 shutdown_connections 2023-10-30T13:53:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.986+0000 7f1607772640 1 -- 172.21.15.31:0/4052756602 wait complete. 2023-10-30T13:53:40.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 Processor -- start 2023-10-30T13:53:40.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 -- start start 2023-10-30T13:53:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16000cb8b0 0x7f1600144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1600151840 0x7f16001e1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 0x7f16001e2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16001e02c0 con 0x7f1600151840 2023-10-30T13:53:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16001e0c90 con 0x7f1600151c20 2023-10-30T13:53:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160014e680 con 0x7f16000cb8b0 2023-10-30T13:53:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1600151840 0x7f16001e1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1600151840 0x7f16001e1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41482/0 (socket says 172.21.15.31:41482) 2023-10-30T13:53:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 -- 172.21.15.31:0/1547475641 learned_addr learned my addr 172.21.15.31:0/1547475641 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1605d0f640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 0x7f16001e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f160550e640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16000cb8b0 0x7f1600144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 -- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16000cb8b0 msgr2=0x7f1600144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16000cb8b0 0x7f1600144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 -- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 msgr2=0x7f16001e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 0x7f16001e2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:40.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 -- 172.21.15.31:0/1547475641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0009420 con 0x7f1600151840 2023-10-30T13:53:40.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1604d0d640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1600151840 0x7f16001e1df0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f15f00098b0 tx=0x7f15f001b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:40.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f15f5ffb640 1 -- 172.21.15.31:0/1547475641 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f00b0650 con 0x7f1600151840 2023-10-30T13:53:40.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f15f5ffb640 1 -- 172.21.15.31:0/1547475641 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f00b07b0 con 0x7f1600151840 2023-10-30T13:53:40.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1600144540 con 0x7f1600151840 2023-10-30T13:53:40.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f15f5ffb640 1 -- 172.21.15.31:0/1547475641 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f00b0b20 con 0x7f1600151840 2023-10-30T13:53:40.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.990+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16001dffe0 con 0x7f1600151840 2023-10-30T13:53:41.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.994+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16001efcc0 con 0x7f1600151840 2023-10-30T13:53:41.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.994+0000 7f15f5ffb640 1 -- 172.21.15.31:0/1547475641 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15f00b7040 con 0x7f1600151840 2023-10-30T13:53:41.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.994+0000 7f15f5ffb640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15e40664d0 0x7f15e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:41.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.994+0000 7f15f5ffb640 1 -- 172.21.15.31:0/1547475641 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f15f01215a0 con 0x7f1600151840 2023-10-30T13:53:41.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.994+0000 7f160550e640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15e40664d0 0x7f15e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:41.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.994+0000 7f160550e640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15e40664d0 0x7f15e4068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0011f0 tx=0x7f15fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:41.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:40.998+0000 7f15f5ffb640 1 -- 172.21.15.31:0/1547475641 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f16001efcc0 con 0x7f1600151840 2023-10-30T13:53:41.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.186+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16001efcc0 con 0x7f1600151840 2023-10-30T13:53:41.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.186+0000 7f15f5ffb640 1 -- 172.21.15.31:0/1547475641 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f16001efcc0 con 0x7f1600151840 2023-10-30T13:53:41.191 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:41.191 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:41.192 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:41.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15e40664d0 msgr2=0x7f15e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:41.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15e40664d0 0x7f15e4068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0011f0 tx=0x7f15fc009040 comp rx=0 tx=0).stop 2023-10-30T13:53:41.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1600151840 msgr2=0x7f16001e1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:41.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1600151840 0x7f16001e1df0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f15f00098b0 tx=0x7f15f001b930 comp rx=0 tx=0).stop 2023-10-30T13:53:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 shutdown_connections 2023-10-30T13:53:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f15e40664d0 0x7f15e4068990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1600151c20 0x7f16001e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1600151840 0x7f16001e1df0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:41.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 --2- 172.21.15.31:0/1547475641 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f16000cb8b0 0x7f1600144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:41.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 >> 172.21.15.31:0/1547475641 conn(0x7f16000cac70 msgr2=0x7f16000cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:41.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 shutdown_connections 2023-10-30T13:53:41.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:41.190+0000 7f1607772640 1 -- 172.21.15.31:0/1547475641 wait complete. 2023-10-30T13:53:41.212 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:43.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.366+0000 7ff495e2f640 1 Processor -- start 2023-10-30T13:53:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff495e2f640 1 -- start start 2023-10-30T13:53:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff495e2f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff4900c8eb0 0x7ff49015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff495e2f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff49015ac70 0x7ff49015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff495e2f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff49006aeb0 0x7ff490069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff495e2f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff490159800 con 0x7ff49015ac70 2023-10-30T13:53:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff495e2f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49015dcc0 con 0x7ff4900c8eb0 2023-10-30T13:53:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff495e2f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff490159d80 con 0x7ff49006aeb0 2023-10-30T13:53:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff49015ac70 0x7ff49015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff49015ac70 0x7ff49015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41494/0 (socket says 172.21.15.31:41494) 2023-10-30T13:53:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 -- 172.21.15.31:0/1251301687 learned_addr learned my addr 172.21.15.31:0/1251301687 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff49562e640 1 --2- 172.21.15.31:0/1251301687 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff49006aeb0 0x7ff490069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff494e2d640 1 --2- 172.21.15.31:0/1251301687 >> v2:172.21.15.33:3300/0 conn(0x7ff4900c8eb0 0x7ff49015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 -- 172.21.15.31:0/1251301687 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff49006aeb0 msgr2=0x7ff490069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 --2- 172.21.15.31:0/1251301687 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff49006aeb0 0x7ff490069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 -- 172.21.15.31:0/1251301687 >> v2:172.21.15.33:3300/0 conn(0x7ff4900c8eb0 msgr2=0x7ff49015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 --2- 172.21.15.31:0/1251301687 >> v2:172.21.15.33:3300/0 conn(0x7ff4900c8eb0 0x7ff49015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.370+0000 7ff48ffff640 1 -- 172.21.15.31:0/1251301687 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49015a750 con 0x7ff49015ac70 2023-10-30T13:53:43.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff48ffff640 1 --2- 172.21.15.31:0/1251301687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49015ac70 0x7ff49015a210 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff49006aa40 tx=0x7ff47c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c08047dd49e76e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff48f7fe640 1 -- 172.21.15.31:0/1251301687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0a89b0 con 0x7ff49015ac70 2023-10-30T13:53:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff48f7fe640 1 -- 172.21.15.31:0/1251301687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c0a8b10 con 0x7ff49015ac70 2023-10-30T13:53:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff48f7fe640 1 -- 172.21.15.31:0/1251301687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0a8ea0 con 0x7ff49015ac70 2023-10-30T13:53:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 -- 172.21.15.31:0/1251301687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49015ac70 msgr2=0x7ff49015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1251301687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49015ac70 0x7ff49015a210 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff49006aa40 tx=0x7ff47c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 -- 172.21.15.31:0/1251301687 shutdown_connections 2023-10-30T13:53:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1251301687 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff49006aeb0 0x7ff490069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1251301687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49015ac70 0x7ff49015a210 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1251301687 >> v2:172.21.15.33:3300/0 conn(0x7ff4900c8eb0 0x7ff49015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 -- 172.21.15.31:0/1251301687 >> 172.21.15.31:0/1251301687 conn(0x7ff4900cef00 msgr2=0x7ff4900cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 -- 172.21.15.31:0/1251301687 shutdown_connections 2023-10-30T13:53:43.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 -- 172.21.15.31:0/1251301687 wait complete. 2023-10-30T13:53:43.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 Processor -- start 2023-10-30T13:53:43.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.374+0000 7ff495e2f640 1 -- start start 2023-10-30T13:53:43.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49006aeb0 0x7ff4900ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:43.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff4900c8eb0 0x7ff4901e3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:43.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff49015ac70 0x7ff4901e4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:43.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff490154fc0 con 0x7ff49006aeb0 2023-10-30T13:53:43.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4901f2110 con 0x7ff49015ac70 2023-10-30T13:53:43.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4901f23f0 con 0x7ff4900c8eb0 2023-10-30T13:53:43.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49006aeb0 0x7ff4900ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:43.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49006aeb0 0x7ff4900ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41506/0 (socket says 172.21.15.31:41506) 2023-10-30T13:53:43.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 -- 172.21.15.31:0/1745411141 learned_addr learned my addr 172.21.15.31:0/1745411141 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:43.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff48ffff640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff4900c8eb0 0x7ff4901e3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:43.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff49562e640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff49015ac70 0x7ff4901e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:43.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 -- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff4900c8eb0 msgr2=0x7ff4901e3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:43.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff4900c8eb0 0x7ff4901e3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 -- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff49015ac70 msgr2=0x7ff4901e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:43.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff49015ac70 0x7ff4901e4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 -- 172.21.15.31:0/1745411141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c0094d0 con 0x7ff49006aeb0 2023-10-30T13:53:43.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff48ffff640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff4900c8eb0 0x7ff4901e3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:43.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff49562e640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff49015ac70 0x7ff4901e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:43.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff494e2d640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49006aeb0 0x7ff4900ca120 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff48400d800 tx=0x7ff48400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:43.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff48d7fa640 1 -- 172.21.15.31:0/1745411141 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48400b6c0 con 0x7ff49006aeb0 2023-10-30T13:53:43.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff48d7fa640 1 -- 172.21.15.31:0/1745411141 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48400b820 con 0x7ff49006aeb0 2023-10-30T13:53:43.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4901f26d0 con 0x7ff49006aeb0 2023-10-30T13:53:43.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff48d7fa640 1 -- 172.21.15.31:0/1745411141 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48400bb90 con 0x7ff49006aeb0 2023-10-30T13:53:43.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4901f2c90 con 0x7ff49006aeb0 2023-10-30T13:53:43.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.378+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49004e170 con 0x7ff49006aeb0 2023-10-30T13:53:43.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.382+0000 7ff48d7fa640 1 -- 172.21.15.31:0/1745411141 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff48400bcf0 con 0x7ff49006aeb0 2023-10-30T13:53:43.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.382+0000 7ff48d7fa640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff4740664a0 0x7ff474068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:43.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.382+0000 7ff48d7fa640 1 -- 172.21.15.31:0/1745411141 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7ff4840847d0 con 0x7ff49006aeb0 2023-10-30T13:53:43.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.382+0000 7ff48ffff640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff4740664a0 0x7ff474068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:43.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.382+0000 7ff48ffff640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff4740664a0 0x7ff474068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0b2000 tx=0x7ff47c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.386+0000 7ff48d7fa640 1 -- 172.21.15.31:0/1745411141 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff484051870 con 0x7ff49006aeb0 2023-10-30T13:53:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.578+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4900c6680 con 0x7ff49006aeb0 2023-10-30T13:53:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.578+0000 7ff48d7fa640 1 -- 172.21.15.31:0/1745411141 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7ff484054e90 con 0x7ff49006aeb0 2023-10-30T13:53:43.583 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:43.583 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:43.583 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:43.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff4740664a0 msgr2=0x7ff474068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:43.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff4740664a0 0x7ff474068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0b2000 tx=0x7ff47c008040 comp rx=0 tx=0).stop 2023-10-30T13:53:43.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49006aeb0 msgr2=0x7ff4900ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:43.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49006aeb0 0x7ff4900ca120 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff48400d800 tx=0x7ff48400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 shutdown_connections 2023-10-30T13:53:43.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff4740664a0 0x7ff474068960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff49015ac70 0x7ff4901e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff4900c8eb0 0x7ff4901e3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 --2- 172.21.15.31:0/1745411141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff49006aeb0 0x7ff4900ca120 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:43.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 >> 172.21.15.31:0/1745411141 conn(0x7ff4900cef00 msgr2=0x7ff4900c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:43.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 shutdown_connections 2023-10-30T13:53:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:43.582+0000 7ff495e2f640 1 -- 172.21.15.31:0/1745411141 wait complete. 2023-10-30T13:53:43.603 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:45.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:45.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760f417640 1 Processor -- start 2023-10-30T13:53:45.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760f417640 1 -- start start 2023-10-30T13:53:45.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760f417640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f76080c62b0 0x7f76081597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760f417640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760815a770 con 0x7f7608153b50 2023-10-30T13:53:45.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760f417640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080c9d40 con 0x7f76080cb180 2023-10-30T13:53:45.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760f417640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7608159cf0 con 0x7f76080c62b0 2023-10-30T13:53:45.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7608153b50 0x7f76080ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48300/0 (socket says 172.21.15.31:48300) 2023-10-30T13:53:45.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 -- 172.21.15.31:0/1955587243 learned_addr learned my addr 172.21.15.31:0/1955587243 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760dc14640 1 --2- 172.21.15.31:0/1955587243 >> v2:172.21.15.33:3300/0 conn(0x7f76080c62b0 0x7f76081597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349697140 0 0) 0x7f7604001120 con 0x7f7608153b50 2023-10-30T13:53:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f4003530 con 0x7f7608153b50 2023-10-30T13:53:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446317454 0 0) 0x7f76080c9d40 con 0x7f76080cb180 2023-10-30T13:53:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7604001120 con 0x7f76080cb180 2023-10-30T13:53:45.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1696579915 0 0) 0x7f75f4003530 con 0x7f7608153b50 2023-10-30T13:53:45.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 >> v2:172.21.15.33:3300/0 conn(0x7f76080c62b0 msgr2=0x7f76081597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:45.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 --2- 172.21.15.31:0/1955587243 >> v2:172.21.15.33:3300/0 conn(0x7f76080c62b0 0x7f76081597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 >> v1:172.21.15.33:6789/0 conn(0x7f76080cb180 legacy=0x7f760806ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:45.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7608153f30 con 0x7f7608153b50 2023-10-30T13:53:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f7604001810 con 0x7f7608153b50 2023-10-30T13:53:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 >> v1:172.21.15.31:6789/0 conn(0x7f7608153b50 legacy=0x7f76080ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f4003b60 0x7f75f4003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f40047d0 0x7f75f4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75f4005190 0x7f75f400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608159cf0 con 0x7f75f4003b60 2023-10-30T13:53:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f400bfb0 con 0x7f75f40047d0 2023-10-30T13:53:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f400c170 con 0x7f75f4005190 2023-10-30T13:53:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f4003b60 0x7f75f4003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760dc14640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f40047d0 0x7f75f4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760ec16640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75f4005190 0x7f75f400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 -- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75f4005190 msgr2=0x7f75f400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f75f4005190 0x7f75f400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 -- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f40047d0 msgr2=0x7f75f4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f40047d0 0x7f75f4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760e415640 1 -- 172.21.15.31:0/1955587243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4003530 con 0x7f75f4003b60 2023-10-30T13:53:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.758+0000 7f760ec16640 1 -- 172.21.15.31:0/1955587243 reap_dead start 2023-10-30T13:53:45.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760e415640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f4003b60 0x7f75f4003f60 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7604007fa0 tx=0x7f760401cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae0d4bab1db4768 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:45.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604005ae0 con 0x7f75f4003b60 2023-10-30T13:53:45.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7604005c60 con 0x7f75f4003b60 2023-10-30T13:53:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760d413640 1 -- 172.21.15.31:0/1955587243 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76040a5a20 con 0x7f75f4003b60 2023-10-30T13:53:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760f417640 1 -- 172.21.15.31:0/1955587243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f4003b60 msgr2=0x7f75f4003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760f417640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f4003b60 0x7f75f4003f60 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7604007fa0 tx=0x7f760401cae0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760f417640 1 -- 172.21.15.31:0/1955587243 shutdown_connections 2023-10-30T13:53:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760f417640 1 --2- 172.21.15.31:0/1955587243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f4003b60 0x7f75f4003f60 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760f417640 1 -- 172.21.15.31:0/1955587243 >> 172.21.15.31:0/1955587243 conn(0x7f7608065a80 msgr2=0x7f7608065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760f417640 1 -- 172.21.15.31:0/1955587243 shutdown_connections 2023-10-30T13:53:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.762+0000 7f760f417640 1 -- 172.21.15.31:0/1955587243 wait complete. 2023-10-30T13:53:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 Processor -- start 2023-10-30T13:53:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 -- start start 2023-10-30T13:53:45.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f4003b60 0x7f76080c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7608158a40 0x7f76080c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f760814cc40 0x7f7608159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080c5a00 con 0x7f760814cc40 2023-10-30T13:53:45.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76081585b0 con 0x7f75f4003b60 2023-10-30T13:53:45.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76080c30f0 con 0x7f7608158a40 2023-10-30T13:53:45.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f760814cc40 0x7f7608159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f760814cc40 0x7f7608159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55970/0 (socket says 172.21.15.31:55970) 2023-10-30T13:53:45.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 -- 172.21.15.31:0/1382803566 learned_addr learned my addr 172.21.15.31:0/1382803566 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:45.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760dc14640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7608158a40 0x7f76080c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760e415640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f4003b60 0x7f76080c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 -- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7608158a40 msgr2=0x7f76080c2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:45.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7608158a40 0x7f76080c2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 -- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f4003b60 msgr2=0x7f76080c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:45.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f4003b60 0x7f76080c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 -- 172.21.15.31:0/1382803566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76081576c0 con 0x7f760814cc40 2023-10-30T13:53:45.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760ec16640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f760814cc40 0x7f7608159310 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f76081573e0 tx=0x7f760000f170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:45.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f75feffd640 1 -- 172.21.15.31:0/1382803566 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760000fd00 con 0x7f760814cc40 2023-10-30T13:53:45.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f75feffd640 1 -- 172.21.15.31:0/1382803566 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760000fe80 con 0x7f760814cc40 2023-10-30T13:53:45.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f75feffd640 1 -- 172.21.15.31:0/1382803566 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600015650 con 0x7f760814cc40 2023-10-30T13:53:45.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76081579a0 con 0x7f760814cc40 2023-10-30T13:53:45.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.766+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7608157f60 con 0x7f760814cc40 2023-10-30T13:53:45.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.770+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7608067540 con 0x7f760814cc40 2023-10-30T13:53:45.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.770+0000 7f75feffd640 1 -- 172.21.15.31:0/1382803566 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76000157b0 con 0x7f760814cc40 2023-10-30T13:53:45.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.770+0000 7f75feffd640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75ec0664f0 0x7f75ec0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:45.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.770+0000 7f75feffd640 1 -- 172.21.15.31:0/1382803566 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f7600087570 con 0x7f760814cc40 2023-10-30T13:53:45.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.770+0000 7f760e415640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75ec0664f0 0x7f75ec0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:45.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.770+0000 7f760e415640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75ec0664f0 0x7f75ec0689b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f76040ab000 tx=0x7f76040b1040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:45.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.774+0000 7f75feffd640 1 -- 172.21.15.31:0/1382803566 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7600054720 con 0x7f760814cc40 2023-10-30T13:53:45.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.962+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f760804e9c0 con 0x7f760814cc40 2023-10-30T13:53:45.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.962+0000 7f75feffd640 1 -- 172.21.15.31:0/1382803566 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f7600057d40 con 0x7f760814cc40 2023-10-30T13:53:45.968 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:45.968 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:51:48.758889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5591":{"gid":5591,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":27,"addr":"172.21.15.33:6836/3928996203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":3928996203},{"type":"v1","addr":"172.21.15.33:6836","nonce":3928996203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:45.968 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2023-10-30T13:53:45.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75ec0664f0 msgr2=0x7f75ec0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:45.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75ec0664f0 0x7f75ec0689b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f76040ab000 tx=0x7f76040b1040 comp rx=0 tx=0).stop 2023-10-30T13:53:45.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f760814cc40 msgr2=0x7f7608159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:45.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f760814cc40 0x7f7608159310 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f76081573e0 tx=0x7f760000f170 comp rx=0 tx=0).stop 2023-10-30T13:53:45.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 shutdown_connections 2023-10-30T13:53:45.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f75ec0664f0 0x7f75ec0689b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f760814cc40 0x7f7608159310 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7608158a40 0x7f76080c2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 --2- 172.21.15.31:0/1382803566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f4003b60 0x7f76080c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:45.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 >> 172.21.15.31:0/1382803566 conn(0x7f7608065a80 msgr2=0x7f7608150220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:45.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 shutdown_connections 2023-10-30T13:53:45.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:45.966+0000 7f760f417640 1 -- 172.21.15.31:0/1382803566 wait complete. 2023-10-30T13:53:45.988 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:53:47.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:48.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 Processor -- start 2023-10-30T13:53:48.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 -- start start 2023-10-30T13:53:48.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea000c55d0 0x7fea0006e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fea0006eb70 0x7fea00151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fea000c59b0 0x7fea001519c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea00158720 con 0x7fea0006eb70 2023-10-30T13:53:48.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea001576f0 con 0x7fea000c59b0 2023-10-30T13:53:48.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea083c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea000c87a0 con 0x7fea000c55d0 2023-10-30T13:53:48.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea05960640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fea0006eb70 0x7fea00151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea05960640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fea0006eb70 0x7fea00151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55974/0 (socket says 172.21.15.31:55974) 2023-10-30T13:53:48.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.141+0000 7fea05960640 1 -- 172.21.15.31:0/3021677803 learned_addr learned my addr 172.21.15.31:0/3021677803 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:48.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea06161640 1 --2- 172.21.15.31:0/3021677803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea000c55d0 0x7fea0006e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea06962640 1 --2- 172.21.15.31:0/3021677803 >> v2:172.21.15.33:3300/0 conn(0x7fea000c59b0 0x7fea001519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea05960640 1 -- 172.21.15.31:0/3021677803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea000c55d0 msgr2=0x7fea0006e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:48.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea05960640 1 --2- 172.21.15.31:0/3021677803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea000c55d0 0x7fea0006e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea05960640 1 -- 172.21.15.31:0/3021677803 >> v2:172.21.15.33:3300/0 conn(0x7fea000c59b0 msgr2=0x7fea001519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:48.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea05960640 1 --2- 172.21.15.31:0/3021677803 >> v2:172.21.15.33:3300/0 conn(0x7fea000c59b0 0x7fea001519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea05960640 1 -- 172.21.15.31:0/3021677803 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea001578e0 con 0x7fea0006eb70 2023-10-30T13:53:48.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea05960640 1 --2- 172.21.15.31:0/3021677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea0006eb70 0x7fea00151480 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fea000c2c80 tx=0x7fe9f001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28bcb05054ff8f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:48.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea0515f640 1 -- 172.21.15.31:0/3021677803 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00a89b0 con 0x7fea0006eb70 2023-10-30T13:53:48.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea0515f640 1 -- 172.21.15.31:0/3021677803 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f00a8b10 con 0x7fea0006eb70 2023-10-30T13:53:48.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.145+0000 7fea0515f640 1 -- 172.21.15.31:0/3021677803 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00a8ea0 con 0x7fea0006eb70 2023-10-30T13:53:48.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- 172.21.15.31:0/3021677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea0006eb70 msgr2=0x7fea00151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 --2- 172.21.15.31:0/3021677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea0006eb70 0x7fea00151480 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fea000c2c80 tx=0x7fe9f001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- 172.21.15.31:0/3021677803 shutdown_connections 2023-10-30T13:53:48.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 --2- 172.21.15.31:0/3021677803 >> v2:172.21.15.33:3300/0 conn(0x7fea000c59b0 0x7fea001519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 --2- 172.21.15.31:0/3021677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea0006eb70 0x7fea00151480 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 --2- 172.21.15.31:0/3021677803 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea000c55d0 0x7fea0006e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- 172.21.15.31:0/3021677803 >> 172.21.15.31:0/3021677803 conn(0x7fea0006afa0 msgr2=0x7fea0006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:48.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- 172.21.15.31:0/3021677803 shutdown_connections 2023-10-30T13:53:48.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- 172.21.15.31:0/3021677803 wait complete. 2023-10-30T13:53:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 Processor -- start 2023-10-30T13:53:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- start start 2023-10-30T13:53:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea0006eb70 0x7fea001514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea000c55d0 0x7fea001588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea000c59b0 0x7fea001f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea001ee690 con 0x7fea000c55d0 2023-10-30T13:53:48.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea001ef640 con 0x7fea000c59b0 2023-10-30T13:53:48.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea083c5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea000c5180 con 0x7fea0006eb70 2023-10-30T13:53:48.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea000c55d0 0x7fea001588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea000c55d0 0x7fea001588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55986/0 (socket says 172.21.15.31:55986) 2023-10-30T13:53:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 -- 172.21.15.31:0/798355407 learned_addr learned my addr 172.21.15.31:0/798355407 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea06161640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea0006eb70 0x7fea001514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea06962640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea000c59b0 0x7fea001f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 -- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea0006eb70 msgr2=0x7fea001514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea0006eb70 0x7fea001514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 -- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea000c59b0 msgr2=0x7fea001f1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea000c59b0 0x7fea001f1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 -- 172.21.15.31:0/798355407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00094d0 con 0x7fea000c55d0 2023-10-30T13:53:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea06962640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea000c59b0 0x7fea001f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.149+0000 7fea05960640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea000c55d0 0x7fea001588b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe9f001d3a0 tx=0x7fe9f001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:48.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fe9f6ffd640 1 -- 172.21.15.31:0/798355407 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00ba600 con 0x7fea000c55d0 2023-10-30T13:53:48.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fe9f6ffd640 1 -- 172.21.15.31:0/798355407 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f00ba760 con 0x7fea000c55d0 2023-10-30T13:53:48.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea001f1ca0 con 0x7fea000c55d0 2023-10-30T13:53:48.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fe9f6ffd640 1 -- 172.21.15.31:0/798355407 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00bab10 con 0x7fea000c55d0 2023-10-30T13:53:48.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea001eeb80 con 0x7fea000c55d0 2023-10-30T13:53:48.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0015a790 con 0x7fea000c55d0 2023-10-30T13:53:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fe9f6ffd640 1 -- 172.21.15.31:0/798355407 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe9f00bac70 con 0x7fea000c55d0 2023-10-30T13:53:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fe9f6ffd640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9e40664a0 0x7fe9e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fe9f6ffd640 1 -- 172.21.15.31:0/798355407 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fe9f0122840 con 0x7fea000c55d0 2023-10-30T13:53:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.153+0000 7fea06161640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9e40664a0 0x7fe9e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.157+0000 7fea06161640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9e40664a0 0x7fe9e4068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0045f0 tx=0x7fe9fc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:48.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.157+0000 7fe9f6ffd640 1 -- 172.21.15.31:0/798355407 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea0015a790 con 0x7fea000c55d0 2023-10-30T13:53:48.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea0015a790 con 0x7fea000c55d0 2023-10-30T13:53:48.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fe9f6ffd640 1 -- 172.21.15.31:0/798355407 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fea0015a790 con 0x7fea000c55d0 2023-10-30T13:53:48.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:48.353 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:47.073149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:48.354 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 78 2023-10-30T13:53:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9e40664a0 msgr2=0x7fe9e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9e40664a0 0x7fe9e4068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0045f0 tx=0x7fe9fc008040 comp rx=0 tx=0).stop 2023-10-30T13:53:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea000c55d0 msgr2=0x7fea001588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea000c55d0 0x7fea001588b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe9f001d3a0 tx=0x7fe9f001ba90 comp rx=0 tx=0).stop 2023-10-30T13:53:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 shutdown_connections 2023-10-30T13:53:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe9e40664a0 0x7fe9e4068960 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea000c59b0 0x7fea001f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea000c55d0 0x7fea001588b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 --2- 172.21.15.31:0/798355407 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fea0006eb70 0x7fea001514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 >> 172.21.15.31:0/798355407 conn(0x7fea0006afa0 msgr2=0x7fea000cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 shutdown_connections 2023-10-30T13:53:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.349+0000 7fea083c5640 1 -- 172.21.15.31:0/798355407 wait complete. 2023-10-30T13:53:48.374 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:53:48.375 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:53:48.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.513+0000 7f563c0c9640 1 Processor -- start 2023-10-30T13:53:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.513+0000 7f563c0c9640 1 -- start start 2023-10-30T13:53:48.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f563c0c9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5634150830 0x7f563406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f563c0c9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f5634070520 0x7f5634070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f563c0c9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340c2310 con 0x7f56340cb7d0 2023-10-30T13:53:48.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f563c0c9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5634156710 con 0x7f5634150830 2023-10-30T13:53:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f563c0c9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56340c4360 con 0x7f5634070520 2023-10-30T13:53:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5639e65640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5634150830 0x7f563406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5639664640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f56340cb7d0 0x7f56340c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48314/0 (socket says 172.21.15.31:48314) 2023-10-30T13:53:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5639664640 1 -- 172.21.15.31:0/357129022 learned_addr learned my addr 172.21.15.31:0/357129022 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f563a666640 1 --2- 172.21.15.31:0/357129022 >> v2:172.21.15.33:3300/0 conn(0x7f5634070520 0x7f5634070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5639e65640 1 -- 172.21.15.31:0/357129022 >> v2:172.21.15.33:3300/0 conn(0x7f5634070520 msgr2=0x7f5634070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:48.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5639e65640 1 --2- 172.21.15.31:0/357129022 >> v2:172.21.15.33:3300/0 conn(0x7f5634070520 0x7f5634070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5639e65640 1 -- 172.21.15.31:0/357129022 >> v1:172.21.15.31:6789/0 conn(0x7f56340cb7d0 legacy=0x7f56340c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5639e65640 1 -- 172.21.15.31:0/357129022 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56341569b0 con 0x7f5634150830 2023-10-30T13:53:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.517+0000 7f5638e63640 1 -- 172.21.15.31:0/357129022 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292293808 0 0) 0x7f56340c2310 con 0x7f56340cb7d0 2023-10-30T13:53:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f5639e65640 1 --2- 172.21.15.31:0/357129022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f563406b500 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5634066e10 tx=0x7f563001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89793dc6c7d5dc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f5638e63640 1 -- 172.21.15.31:0/357129022 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56300a8950 con 0x7f5634150830 2023-10-30T13:53:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f5638e63640 1 -- 172.21.15.31:0/357129022 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56300a8ab0 con 0x7f5634150830 2023-10-30T13:53:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f5638e63640 1 -- 172.21.15.31:0/357129022 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56300a8e40 con 0x7f5634150830 2023-10-30T13:53:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 -- 172.21.15.31:0/357129022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 msgr2=0x7f563406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 --2- 172.21.15.31:0/357129022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f563406b500 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5634066e10 tx=0x7f563001ec50 comp rx=0 tx=0).stop 2023-10-30T13:53:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 -- 172.21.15.31:0/357129022 shutdown_connections 2023-10-30T13:53:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 --2- 172.21.15.31:0/357129022 >> v2:172.21.15.33:3300/0 conn(0x7f5634070520 0x7f5634070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 --2- 172.21.15.31:0/357129022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f563406b500 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 -- 172.21.15.31:0/357129022 >> 172.21.15.31:0/357129022 conn(0x7f56340cac70 msgr2=0x7f56340cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 -- 172.21.15.31:0/357129022 shutdown_connections 2023-10-30T13:53:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 -- 172.21.15.31:0/357129022 wait complete. 2023-10-30T13:53:48.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 Processor -- start 2023-10-30T13:53:48.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 -- start start 2023-10-30T13:53:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5634070520 0x7f56341599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.521+0000 7f563c0c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56340cb7d0 0x7f5634154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563c0c9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f5634158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563c0c9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563415a180 con 0x7f5634150830 2023-10-30T13:53:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563c0c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56340c9c30 con 0x7f56340cb7d0 2023-10-30T13:53:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563c0c9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563415a440 con 0x7f5634070520 2023-10-30T13:53:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f5634158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f5634158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56008/0 (socket says 172.21.15.31:56008) 2023-10-30T13:53:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 -- 172.21.15.31:0/2003729402 learned_addr learned my addr 172.21.15.31:0/2003729402 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f5639664640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56340cb7d0 0x7f5634154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f5639e65640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5634070520 0x7f56341599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 -- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5634070520 msgr2=0x7f56341599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5634070520 0x7f56341599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 -- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56340cb7d0 msgr2=0x7f5634154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56340cb7d0 0x7f5634154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 -- 172.21.15.31:0/2003729402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630009420 con 0x7f5634150830 2023-10-30T13:53:48.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563a666640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f5634158390 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f562c007e40 tx=0x7f562c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:48.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f562a7fc640 1 -- 172.21.15.31:0/2003729402 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c014620 con 0x7f5634150830 2023-10-30T13:53:48.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f562a7fc640 1 -- 172.21.15.31:0/2003729402 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c00ee90 con 0x7f5634150830 2023-10-30T13:53:48.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563415a700 con 0x7f5634150830 2023-10-30T13:53:48.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f562a7fc640 1 -- 172.21.15.31:0/2003729402 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0149e0 con 0x7f5634150830 2023-10-30T13:53:48.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56340c98c0 con 0x7f5634150830 2023-10-30T13:53:48.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56340c9430 con 0x7f5634150830 2023-10-30T13:53:48.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.525+0000 7f562a7fc640 1 -- 172.21.15.31:0/2003729402 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f562c014b40 con 0x7f5634150830 2023-10-30T13:53:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.529+0000 7f562a7fc640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f56180664d0 0x7f5618068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.529+0000 7f562a7fc640 1 -- 172.21.15.31:0/2003729402 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f562c084a70 con 0x7f5634150830 2023-10-30T13:53:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.529+0000 7f5639e65640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f56180664d0 0x7f5618068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.529+0000 7f5639e65640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f56180664d0 0x7f5618068990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f56300b2000 tx=0x7f5630008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.533+0000 7f562a7fc640 1 -- 172.21.15.31:0/2003729402 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f562c051bf0 con 0x7f5634150830 2023-10-30T13:53:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.725+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f563404e170 con 0x7f5634150830 2023-10-30T13:53:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.725+0000 7f562a7fc640 1 -- 172.21.15.31:0/2003729402 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f562c055210 con 0x7f5634150830 2023-10-30T13:53:48.732 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:48.732 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:47.073149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 78 2023-10-30T13:53:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f56180664d0 msgr2=0x7f5618068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f56180664d0 0x7f5618068990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f56300b2000 tx=0x7f5630008040 comp rx=0 tx=0).stop 2023-10-30T13:53:48.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 msgr2=0x7f5634158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:48.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f5634158390 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f562c007e40 tx=0x7f562c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 shutdown_connections 2023-10-30T13:53:48.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f56180664d0 0x7f5618068990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5634150830 0x7f5634158390 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56340cb7d0 0x7f5634154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 --2- 172.21.15.31:0/2003729402 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5634070520 0x7f56341599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 >> 172.21.15.31:0/2003729402 conn(0x7f56340cac70 msgr2=0x7f56340736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 shutdown_connections 2023-10-30T13:53:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:48.729+0000 7f563c0c9640 1 -- 172.21.15.31:0/2003729402 wait complete. 2023-10-30T13:53:48.753 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 78, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 78, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6201": { "addr": "172.21.15.31:6837/2267972067", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 2267972067, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 2267972067, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 78, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:53:47.073149+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6201 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:53:50.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:50.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce0a13e640 1 Processor -- start 2023-10-30T13:53:50.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce0a13e640 1 -- start start 2023-10-30T13:53:50.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce0a13e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce040c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:50.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce0a13e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fce040cb8b0 0x7fce0406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:50.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce0a13e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce040c5030 con 0x7fce04151840 2023-10-30T13:53:50.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce0a13e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce04157830 con 0x7fce040cb8b0 2023-10-30T13:53:50.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce0a13e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce04157b10 con 0x7fce04151c20 2023-10-30T13:53:50.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce03fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fce040cb8b0 0x7fce0406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce037fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fce04151840 0x7fce04072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48330/0 (socket says 172.21.15.31:48330) 2023-10-30T13:53:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce037fe640 1 -- 172.21.15.31:0/3825701425 learned_addr learned my addr 172.21.15.31:0/3825701425 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.905+0000 7fce02ffd640 1 --2- 172.21.15.31:0/3825701425 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce040c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce027fc640 1 -- 172.21.15.31:0/3825701425 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592767596 0 0) 0x7fce040c5030 con 0x7fce04151840 2023-10-30T13:53:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce027fc640 1 -- 172.21.15.31:0/3825701425 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdec003530 con 0x7fce04151840 2023-10-30T13:53:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce03fff640 1 -- 172.21.15.31:0/3825701425 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 msgr2=0x7fce040c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce03fff640 1 --2- 172.21.15.31:0/3825701425 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce040c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce03fff640 1 -- 172.21.15.31:0/3825701425 >> v1:172.21.15.31:6789/0 conn(0x7fce04151840 legacy=0x7fce04072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce03fff640 1 -- 172.21.15.31:0/3825701425 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce04157df0 con 0x7fce040cb8b0 2023-10-30T13:53:50.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce03fff640 1 --2- 172.21.15.31:0/3825701425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce0406b600 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009d90 tx=0x7fcdfc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8cebc20963389f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:50.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce027fc640 1 -- 172.21.15.31:0/3825701425 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0af600 con 0x7fce040cb8b0 2023-10-30T13:53:50.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce027fc640 1 -- 172.21.15.31:0/3825701425 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc0ab050 con 0x7fce040cb8b0 2023-10-30T13:53:50.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce027fc640 1 -- 172.21.15.31:0/3825701425 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0af980 con 0x7fce040cb8b0 2023-10-30T13:53:50.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce0a13e640 1 -- 172.21.15.31:0/3825701425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 msgr2=0x7fce0406b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:50.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce0a13e640 1 --2- 172.21.15.31:0/3825701425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce0406b600 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009d90 tx=0x7fcdfc01f130 comp rx=0 tx=0).stop 2023-10-30T13:53:50.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce0a13e640 1 -- 172.21.15.31:0/3825701425 shutdown_connections 2023-10-30T13:53:50.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce0a13e640 1 --2- 172.21.15.31:0/3825701425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce0406b600 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:50.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce0a13e640 1 --2- 172.21.15.31:0/3825701425 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce040c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:50.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.909+0000 7fce0a13e640 1 -- 172.21.15.31:0/3825701425 >> 172.21.15.31:0/3825701425 conn(0x7fce040cac70 msgr2=0x7fce040cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:50.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 -- 172.21.15.31:0/3825701425 shutdown_connections 2023-10-30T13:53:50.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 -- 172.21.15.31:0/3825701425 wait complete. 2023-10-30T13:53:50.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 Processor -- start 2023-10-30T13:53:50.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 -- start start 2023-10-30T13:53:50.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce041e1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:50.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce04151840 0x7fce04146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:50.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce041dd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:50.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04071e90 con 0x7fce040cb8b0 2023-10-30T13:53:50.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce040c85f0 con 0x7fce04151840 2023-10-30T13:53:50.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce0a13e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce041e18d0 con 0x7fce04151c20 2023-10-30T13:53:50.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce041e1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:50.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce041e1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56040/0 (socket says 172.21.15.31:56040) 2023-10-30T13:53:50.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 -- 172.21.15.31:0/1700504004 learned_addr learned my addr 172.21.15.31:0/1700504004 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:50.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce02ffd640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce04151840 0x7fce04146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:50.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce03fff640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce041dd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:50.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 -- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 msgr2=0x7fce041dd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:50.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce041dd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:50.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 -- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce04151840 msgr2=0x7fce04146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:50.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce04151840 0x7fce04146800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:50.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 -- 172.21.15.31:0/1700504004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc009a70 con 0x7fce040cb8b0 2023-10-30T13:53:50.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce03fff640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce041dd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:53:50.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fce037fe640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce041e1220 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c4d0 tx=0x7fcdf400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:50.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fcdebfff640 1 -- 172.21.15.31:0/1700504004 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4005cf0 con 0x7fce040cb8b0 2023-10-30T13:53:50.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fcdebfff640 1 -- 172.21.15.31:0/1700504004 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4005e50 con 0x7fce040cb8b0 2023-10-30T13:53:50.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.913+0000 7fcdebfff640 1 -- 172.21.15.31:0/1700504004 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4007270 con 0x7fce040cb8b0 2023-10-30T13:53:50.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.917+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0414bc90 con 0x7fce040cb8b0 2023-10-30T13:53:50.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.917+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0414c250 con 0x7fce040cb8b0 2023-10-30T13:53:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.917+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0404e170 con 0x7fce040cb8b0 2023-10-30T13:53:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.917+0000 7fcdebfff640 1 -- 172.21.15.31:0/1700504004 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcdf40073d0 con 0x7fce040cb8b0 2023-10-30T13:53:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.917+0000 7fcdebfff640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcdf00664f0 0x7fcdf00689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.917+0000 7fcdebfff640 1 -- 172.21.15.31:0/1700504004 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fcdf4083850 con 0x7fce040cb8b0 2023-10-30T13:53:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.917+0000 7fce02ffd640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcdf00664f0 0x7fcdf00689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.921+0000 7fce02ffd640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcdf00664f0 0x7fcdf00689b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fce041de2b0 tx=0x7fcdf8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:50.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:50.921+0000 7fcdebfff640 1 -- 172.21.15.31:0/1700504004 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcdf4050930 con 0x7fce040cb8b0 2023-10-30T13:53:51.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.109+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce0414fd20 con 0x7fce040cb8b0 2023-10-30T13:53:51.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.109+0000 7fcdebfff640 1 -- 172.21.15.31:0/1700504004 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fcdf4017090 con 0x7fce040cb8b0 2023-10-30T13:53:51.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:51.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:47.073149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:51.115 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 78 2023-10-30T13:53:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcdf00664f0 msgr2=0x7fcdf00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcdf00664f0 0x7fcdf00689b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fce041de2b0 tx=0x7fcdf8008040 comp rx=0 tx=0).stop 2023-10-30T13:53:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 msgr2=0x7fce041e1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce041e1220 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c4d0 tx=0x7fcdf400c990 comp rx=0 tx=0).stop 2023-10-30T13:53:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 shutdown_connections 2023-10-30T13:53:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcdf00664f0 0x7fcdf00689b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce04151c20 0x7fce041dd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce04151840 0x7fce04146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 --2- 172.21.15.31:0/1700504004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce040cb8b0 0x7fce041e1220 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:51.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 >> 172.21.15.31:0/1700504004 conn(0x7fce040cac70 msgr2=0x7fce04072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:51.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 shutdown_connections 2023-10-30T13:53:51.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:51.113+0000 7fce0a13e640 1 -- 172.21.15.31:0/1700504004 wait complete. 2023-10-30T13:53:51.138 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:53.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:53.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.289+0000 7fe6d1dd3640 1 Processor -- start 2023-10-30T13:53:53.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.289+0000 7fe6d1dd3640 1 -- start start 2023-10-30T13:53:53.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.289+0000 7fe6d1dd3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe6cc153c80 0x7fe6cc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:53.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.289+0000 7fe6d1dd3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe6cc0c42a0 0x7fe6cc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:53.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.289+0000 7fe6d1dd3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc159f20 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.289+0000 7fe6d1dd3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc15dec0 con 0x7fe6cc153c80 2023-10-30T13:53:53.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.289+0000 7fe6d1dd3640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc1599a0 con 0x7fe6cc0c42a0 2023-10-30T13:53:53.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6cb7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe6cc153c80 0x7fe6cc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:53.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6caffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe6cc0c4a10 0x7fe6cc15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48340/0 (socket says 172.21.15.31:48340) 2023-10-30T13:53:53.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6caffd640 1 -- 172.21.15.31:0/703905572 learned_addr learned my addr 172.21.15.31:0/703905572 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:53.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6cbfff640 1 --2- 172.21.15.31:0/703905572 >> v2:172.21.15.33:3300/0 conn(0x7fe6cc0c42a0 0x7fe6cc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:53.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6cb7fe640 1 -- 172.21.15.31:0/703905572 >> v2:172.21.15.33:3300/0 conn(0x7fe6cc0c42a0 msgr2=0x7fe6cc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:53.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6cb7fe640 1 --2- 172.21.15.31:0/703905572 >> v2:172.21.15.33:3300/0 conn(0x7fe6cc0c42a0 0x7fe6cc0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6cb7fe640 1 -- 172.21.15.31:0/703905572 >> v1:172.21.15.31:6789/0 conn(0x7fe6cc0c4a10 legacy=0x7fe6cc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:53.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6cb7fe640 1 -- 172.21.15.31:0/703905572 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc15d620 con 0x7fe6cc153c80 2023-10-30T13:53:53.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.293+0000 7fe6cbfff640 1 --2- 172.21.15.31:0/703905572 >> v2:172.21.15.33:3300/0 conn(0x7fe6cc0c42a0 0x7fe6cc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:53.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6cb7fe640 1 --2- 172.21.15.31:0/703905572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc153c80 0x7fe6cc15cf90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0b1d40 tx=0x7fe6bc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be1f0752d5c00b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:53.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6ca7fc640 1 -- 172.21.15.31:0/703905572 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0a8950 con 0x7fe6cc153c80 2023-10-30T13:53:53.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6ca7fc640 1 -- 172.21.15.31:0/703905572 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc0a8ab0 con 0x7fe6cc153c80 2023-10-30T13:53:53.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6ca7fc640 1 -- 172.21.15.31:0/703905572 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0a8e40 con 0x7fe6cc153c80 2023-10-30T13:53:53.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/703905572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc153c80 msgr2=0x7fe6cc15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:53.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/703905572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc153c80 0x7fe6cc15cf90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0b1d40 tx=0x7fe6bc01ec50 comp rx=0 tx=0).stop 2023-10-30T13:53:53.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/703905572 shutdown_connections 2023-10-30T13:53:53.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/703905572 >> v2:172.21.15.33:3300/0 conn(0x7fe6cc0c42a0 0x7fe6cc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/703905572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc153c80 0x7fe6cc15cf90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/703905572 >> 172.21.15.31:0/703905572 conn(0x7fe6cc0715c0 msgr2=0x7fe6cc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:53.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/703905572 shutdown_connections 2023-10-30T13:53:53.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.297+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/703905572 wait complete. 2023-10-30T13:53:53.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 Processor -- start 2023-10-30T13:53:53.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 -- start start 2023-10-30T13:53:53.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6cc0c42a0 0x7fe6cc15bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:53.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc0c4a10 0x7fe6cc1f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:53.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6cc153c80 0x7fe6cc156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:53.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1f2200 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc154c40 con 0x7fe6cc153c80 2023-10-30T13:53:53.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1f1680 con 0x7fe6cc0c42a0 2023-10-30T13:53:53.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc0c4a10 0x7fe6cc1f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:53.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc0c4a10 0x7fe6cc1f0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56054/0 (socket says 172.21.15.31:56054) 2023-10-30T13:53:53.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 -- 172.21.15.31:0/1509831272 learned_addr learned my addr 172.21.15.31:0/1509831272 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:53.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6cb7fe640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6cc0c42a0 0x7fe6cc15bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:53.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6cbfff640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6cc153c80 0x7fe6cc156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:53.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 -- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6cc0c42a0 msgr2=0x7fe6cc15bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:53.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6cc0c42a0 0x7fe6cc15bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 -- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6cc153c80 msgr2=0x7fe6cc156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:53.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6cc153c80 0x7fe6cc156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 -- 172.21.15.31:0/1509831272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc009420 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6cbfff640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6cc153c80 0x7fe6cc156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:53.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6caffd640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc0c4a10 0x7fe6cc1f0d80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000c4d0 tx=0x7fe6c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:53.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6b3fff640 1 -- 172.21.15.31:0/1509831272 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0005cf0 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6b3fff640 1 -- 172.21.15.31:0/1509831272 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c0005e50 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6b3fff640 1 -- 172.21.15.31:0/1509831272 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0007270 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc1f1960 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.301+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc1f1f20 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.305+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc1597d0 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.305+0000 7fe6b3fff640 1 -- 172.21.15.31:0/1509831272 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe6c00073d0 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.305+0000 7fe6b3fff640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6b80664a0 0x7fe6b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:53.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.305+0000 7fe6b3fff640 1 -- 172.21.15.31:0/1509831272 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fe6c0083670 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.305+0000 7fe6cb7fe640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6b80664a0 0x7fe6b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:53.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.305+0000 7fe6cb7fe640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6b80664a0 0x7fe6b8068960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0b2000 tx=0x7fe6bc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:53.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.309+0000 7fe6b3fff640 1 -- 172.21.15.31:0/1509831272 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe6c0050760 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.521+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6cc0cab30 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.521+0000 7fe6b3fff640 1 -- 172.21.15.31:0/1509831272 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fe6c0053d80 con 0x7fe6cc0c4a10 2023-10-30T13:53:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:53.528 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:47.073149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:53.528 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 78 2023-10-30T13:53:53.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6b80664a0 msgr2=0x7fe6b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:53.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6b80664a0 0x7fe6b8068960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0b2000 tx=0x7fe6bc008040 comp rx=0 tx=0).stop 2023-10-30T13:53:53.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc0c4a10 msgr2=0x7fe6cc1f0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:53.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc0c4a10 0x7fe6cc1f0d80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000c4d0 tx=0x7fe6c000c990 comp rx=0 tx=0).stop 2023-10-30T13:53:53.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 shutdown_connections 2023-10-30T13:53:53.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe6b80664a0 0x7fe6b8068960 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6cc153c80 0x7fe6cc156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6cc0c4a10 0x7fe6cc1f0d80 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 --2- 172.21.15.31:0/1509831272 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe6cc0c42a0 0x7fe6cc15bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:53.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 >> 172.21.15.31:0/1509831272 conn(0x7fe6cc0715c0 msgr2=0x7fe6cc0c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:53.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 shutdown_connections 2023-10-30T13:53:53.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:53.525+0000 7fe6d1dd3640 1 -- 172.21.15.31:0/1509831272 wait complete. 2023-10-30T13:53:53.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:55.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5f5217640 1 Processor -- start 2023-10-30T13:53:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5f5217640 1 -- start start 2023-10-30T13:53:55.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5f5217640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe5f0153c80 0x7fe5f015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:55.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5f5217640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe5f00c42a0 0x7fe5f00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:55.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5f5217640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f0159f20 con 0x7fe5f00c4a10 2023-10-30T13:53:55.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5f5217640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f015dec0 con 0x7fe5f00c42a0 2023-10-30T13:53:55.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5f5217640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f01599a0 con 0x7fe5f0153c80 2023-10-30T13:53:55.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5ef59e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe5f00c42a0 0x7fe5f00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:55.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5ee59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe5f00c4a10 0x7fe5f015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41646/0 (socket says 172.21.15.31:41646) 2023-10-30T13:53:55.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5ee59c640 1 -- 172.21.15.31:0/324023100 learned_addr learned my addr 172.21.15.31:0/324023100 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:53:55.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5eed9d640 1 --2- 172.21.15.31:0/324023100 >> v2:172.21.15.33:3300/0 conn(0x7fe5f0153c80 0x7fe5f015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:55.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5edd9b640 1 -- 172.21.15.31:0/324023100 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361180736 0 0) 0x7fe5f0159f20 con 0x7fe5f00c4a10 2023-10-30T13:53:55.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5edd9b640 1 -- 172.21.15.31:0/324023100 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d8003530 con 0x7fe5f00c4a10 2023-10-30T13:53:55.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5ef59e640 1 -- 172.21.15.31:0/324023100 >> v2:172.21.15.33:3300/0 conn(0x7fe5f0153c80 msgr2=0x7fe5f015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5ef59e640 1 --2- 172.21.15.31:0/324023100 >> v2:172.21.15.33:3300/0 conn(0x7fe5f0153c80 0x7fe5f015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5ef59e640 1 -- 172.21.15.31:0/324023100 >> v1:172.21.15.31:6789/0 conn(0x7fe5f00c4a10 legacy=0x7fe5f015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5ef59e640 1 -- 172.21.15.31:0/324023100 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f015d620 con 0x7fe5f00c42a0 2023-10-30T13:53:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.701+0000 7fe5eed9d640 1 --2- 172.21.15.31:0/324023100 >> v2:172.21.15.33:3300/0 conn(0x7fe5f0153c80 0x7fe5f015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:55.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5ef59e640 1 --2- 172.21.15.31:0/324023100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f00ca1e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00d00f0 tx=0x7fe5e801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e3c842ef55a69d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:55.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5edd9b640 1 -- 172.21.15.31:0/324023100 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80b0440 con 0x7fe5f00c42a0 2023-10-30T13:53:55.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5edd9b640 1 -- 172.21.15.31:0/324023100 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e80b05a0 con 0x7fe5f00c42a0 2023-10-30T13:53:55.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5edd9b640 1 -- 172.21.15.31:0/324023100 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80b0930 con 0x7fe5f00c42a0 2023-10-30T13:53:55.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 -- 172.21.15.31:0/324023100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 msgr2=0x7fe5f00ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:55.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 --2- 172.21.15.31:0/324023100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f00ca1e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00d00f0 tx=0x7fe5e801f0f0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 -- 172.21.15.31:0/324023100 shutdown_connections 2023-10-30T13:53:55.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 --2- 172.21.15.31:0/324023100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f00ca1e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 --2- 172.21.15.31:0/324023100 >> v2:172.21.15.33:3300/0 conn(0x7fe5f0153c80 0x7fe5f015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 -- 172.21.15.31:0/324023100 >> 172.21.15.31:0/324023100 conn(0x7fe5f00715c0 msgr2=0x7fe5f00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:55.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 -- 172.21.15.31:0/324023100 shutdown_connections 2023-10-30T13:53:55.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.705+0000 7fe5f5217640 1 -- 172.21.15.31:0/324023100 wait complete. 2023-10-30T13:53:55.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 Processor -- start 2023-10-30T13:53:55.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 -- start start 2023-10-30T13:53:55.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f014d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:55.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5f00c4a10 0x7fe5f014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:55.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5f0153c80 0x7fe5f014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:55.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f00cfe10 con 0x7fe5f00c42a0 2023-10-30T13:53:55.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f014fc50 con 0x7fe5f0153c80 2023-10-30T13:53:55.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5f5217640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f014ff30 con 0x7fe5f00c4a10 2023-10-30T13:53:55.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f014d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:55.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f014d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33896/0 (socket says 172.21.15.31:33896) 2023-10-30T13:53:55.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 -- 172.21.15.31:0/206387919 learned_addr learned my addr 172.21.15.31:0/206387919 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:55.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5ef59e640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5f0153c80 0x7fe5f014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:55.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5ee59c640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5f00c4a10 0x7fe5f014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 -- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5f00c4a10 msgr2=0x7fe5f014d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5f00c4a10 0x7fe5f014d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 -- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5f0153c80 msgr2=0x7fe5f014e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5f0153c80 0x7fe5f014e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5eed9d640 1 -- 172.21.15.31:0/206387919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8009a70 con 0x7fe5f00c42a0 2023-10-30T13:53:55.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.709+0000 7fe5ee59c640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5f00c4a10 0x7fe5f014d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:55.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.713+0000 7fe5eed9d640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f014d160 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000d800 tx=0x7fe5e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:55.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.713+0000 7fe5d77fe640 1 -- 172.21.15.31:0/206387919 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00096c0 con 0x7fe5f00c42a0 2023-10-30T13:53:55.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.713+0000 7fe5d77fe640 1 -- 172.21.15.31:0/206387919 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e0009820 con 0x7fe5f00c42a0 2023-10-30T13:53:55.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.713+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f014deb0 con 0x7fe5f00c42a0 2023-10-30T13:53:55.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.713+0000 7fe5d77fe640 1 -- 172.21.15.31:0/206387919 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0009bd0 con 0x7fe5f00c42a0 2023-10-30T13:53:55.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.713+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f0150370 con 0x7fe5f00c42a0 2023-10-30T13:53:55.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.713+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f014dd10 con 0x7fe5f00c42a0 2023-10-30T13:53:55.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.717+0000 7fe5d77fe640 1 -- 172.21.15.31:0/206387919 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5e0009d30 con 0x7fe5f00c42a0 2023-10-30T13:53:55.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.717+0000 7fe5d77fe640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5dc0664a0 0x7fe5dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:55.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.717+0000 7fe5d77fe640 1 -- 172.21.15.31:0/206387919 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fe5e00848a0 con 0x7fe5f00c42a0 2023-10-30T13:53:55.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.717+0000 7fe5ee59c640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5dc0664a0 0x7fe5dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:55.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.717+0000 7fe5ee59c640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5dc0664a0 0x7fe5dc068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00cdbc0 tx=0x7fe5e4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:55.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.721+0000 7fe5d77fe640 1 -- 172.21.15.31:0/206387919 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5f014dd10 con 0x7fe5f00c42a0 2023-10-30T13:53:55.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.925+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5f014dd10 con 0x7fe5f00c42a0 2023-10-30T13:53:55.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5d77fe640 1 -- 172.21.15.31:0/206387919 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fe5f014dd10 con 0x7fe5f00c42a0 2023-10-30T13:53:55.932 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:55.932 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:47.073149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:55.933 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 78 2023-10-30T13:53:55.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5dc0664a0 msgr2=0x7fe5dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:55.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5dc0664a0 0x7fe5dc068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00cdbc0 tx=0x7fe5e4009040 comp rx=0 tx=0).stop 2023-10-30T13:53:55.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 msgr2=0x7fe5f014d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f014d160 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000d800 tx=0x7fe5e000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 shutdown_connections 2023-10-30T13:53:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe5dc0664a0 0x7fe5dc068960 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5f0153c80 0x7fe5f014e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe5f00c4a10 0x7fe5f014d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 --2- 172.21.15.31:0/206387919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f00c42a0 0x7fe5f014d160 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:55.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 >> 172.21.15.31:0/206387919 conn(0x7fe5f00715c0 msgr2=0x7fe5f0158a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:55.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 shutdown_connections 2023-10-30T13:53:55.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:55.929+0000 7fe5f5217640 1 -- 172.21.15.31:0/206387919 wait complete. 2023-10-30T13:53:55.962 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:57.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.101+0000 7fb0dfdc5640 1 Processor -- start 2023-10-30T13:53:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.101+0000 7fb0dfdc5640 1 -- start start 2023-10-30T13:53:58.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dfdc5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d806e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dfdc5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb0d806dfc0 0x7fb0d8156880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dfdc5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d8156dc0 con 0x7fb0d8071f30 2023-10-30T13:53:58.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dfdc5640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d806ea70 con 0x7fb0d806dfc0 2023-10-30T13:53:58.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dfdc5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8070c40 con 0x7fb0d806dbe0 2023-10-30T13:53:58.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dd360640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb0d806dfc0 0x7fb0d8156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dd360640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb0d806dfc0 0x7fb0d8156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33904/0 (socket says 172.21.15.31:33904) 2023-10-30T13:53:58.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dd360640 1 -- 172.21.15.31:0/2556574064 learned_addr learned my addr 172.21.15.31:0/2556574064 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:58.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0ddb61640 1 --2- 172.21.15.31:0/2556574064 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dd360640 1 -- 172.21.15.31:0/2556574064 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 msgr2=0x7fb0d806e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:58.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dd360640 1 --2- 172.21.15.31:0/2556574064 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d806e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dd360640 1 -- 172.21.15.31:0/2556574064 >> v1:172.21.15.33:6789/0 conn(0x7fb0d8071f30 legacy=0x7fb0d8157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.105+0000 7fb0dd360640 1 -- 172.21.15.31:0/2556574064 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8070e90 con 0x7fb0d806dfc0 2023-10-30T13:53:58.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dd360640 1 --2- 172.21.15.31:0/2556574064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d806dfc0 0x7fb0d8156880 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009870 tx=0x7fb0c801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57cf16bcda80aa4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:58.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dcb5f640 1 -- 172.21.15.31:0/2556574064 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80ab600 con 0x7fb0d806dfc0 2023-10-30T13:53:58.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dcb5f640 1 -- 172.21.15.31:0/2556574064 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c80a7050 con 0x7fb0d806dfc0 2023-10-30T13:53:58.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dcb5f640 1 -- 172.21.15.31:0/2556574064 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80ab980 con 0x7fb0d806dfc0 2023-10-30T13:53:58.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/2556574064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d806dfc0 msgr2=0x7fb0d8156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/2556574064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d806dfc0 0x7fb0d8156880 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009870 tx=0x7fb0c801c480 comp rx=0 tx=0).stop 2023-10-30T13:53:58.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/2556574064 shutdown_connections 2023-10-30T13:53:58.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/2556574064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d806dfc0 0x7fb0d8156880 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/2556574064 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d806e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/2556574064 >> 172.21.15.31:0/2556574064 conn(0x7fb0d80c7f60 msgr2=0x7fb0d80c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/2556574064 shutdown_connections 2023-10-30T13:53:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/2556574064 wait complete. 2023-10-30T13:53:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 Processor -- start 2023-10-30T13:53:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.109+0000 7fb0dfdc5640 1 -- start start 2023-10-30T13:53:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d8149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d806dfc0 0x7fb0d81f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d8071f30 0x7fb0d80c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8155930 con 0x7fb0d8071f30 2023-10-30T13:53:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8153fd0 con 0x7fb0d806dfc0 2023-10-30T13:53:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d81542b0 con 0x7fb0d806dbe0 2023-10-30T13:53:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d8071f30 0x7fb0d80c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dd360640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d806dfc0 0x7fb0d81f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d8071f30 0x7fb0d80c7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33912/0 (socket says 172.21.15.31:33912) 2023-10-30T13:53:58.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 -- 172.21.15.31:0/3791835483 learned_addr learned my addr 172.21.15.31:0/3791835483 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:53:58.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0ddb61640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d8149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 -- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 msgr2=0x7fb0d8149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d8149360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 -- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d806dfc0 msgr2=0x7fb0d81f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d806dfc0 0x7fb0d81f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 -- 172.21.15.31:0/3791835483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c80094d0 con 0x7fb0d8071f30 2023-10-30T13:53:58.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0de362640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d8071f30 0x7fb0d80c7530 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c770 tx=0x7fb0d000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:58.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0ce7fc640 1 -- 172.21.15.31:0/3791835483 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0003b80 con 0x7fb0d8071f30 2023-10-30T13:53:58.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0ce7fc640 1 -- 172.21.15.31:0/3791835483 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0003ce0 con 0x7fb0d8071f30 2023-10-30T13:53:58.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0ce7fc640 1 -- 172.21.15.31:0/3791835483 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000aa90 con 0x7fb0d8071f30 2023-10-30T13:53:58.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d8154590 con 0x7fb0d8071f30 2023-10-30T13:53:58.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d81f2330 con 0x7fb0d8071f30 2023-10-30T13:53:58.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d81f2520 con 0x7fb0d8071f30 2023-10-30T13:53:58.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.113+0000 7fb0ce7fc640 1 -- 172.21.15.31:0/3791835483 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0d000abf0 con 0x7fb0d8071f30 2023-10-30T13:53:58.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.117+0000 7fb0ce7fc640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0bc0664a0 0x7fb0bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.117+0000 7fb0ce7fc640 1 -- 172.21.15.31:0/3791835483 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fb0d0083e90 con 0x7fb0d8071f30 2023-10-30T13:53:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.117+0000 7fb0ddb61640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0bc0664a0 0x7fb0bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.117+0000 7fb0ddb61640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0bc0664a0 0x7fb0bc068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40079c0 tx=0x7fb0d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:58.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.121+0000 7fb0ce7fc640 1 -- 172.21.15.31:0/3791835483 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0d0050f80 con 0x7fb0d8071f30 2023-10-30T13:53:58.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:58.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:58.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d8156dc0 con 0x7fb0d8071f30 2023-10-30T13:53:58.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0ce7fc640 1 -- 172.21.15.31:0/3791835483 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fb0d8156dc0 con 0x7fb0d8071f30 2023-10-30T13:53:58.317 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:53:58.317 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:47.073149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:58.318 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 78 2023-10-30T13:53:58.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0bc0664a0 msgr2=0x7fb0bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0bc0664a0 0x7fb0bc068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40079c0 tx=0x7fb0d4008040 comp rx=0 tx=0).stop 2023-10-30T13:53:58.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d8071f30 msgr2=0x7fb0d80c7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d8071f30 0x7fb0d80c7530 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c770 tx=0x7fb0d000cc30 comp rx=0 tx=0).stop 2023-10-30T13:53:58.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 shutdown_connections 2023-10-30T13:53:58.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0bc0664a0 0x7fb0bc068960 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d8071f30 0x7fb0d80c7530 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d806dfc0 0x7fb0d81f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 --2- 172.21.15.31:0/3791835483 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb0d806dbe0 0x7fb0d8149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 >> 172.21.15.31:0/3791835483 conn(0x7fb0d80c7f60 msgr2=0x7fb0d814e370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:58.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 shutdown_connections 2023-10-30T13:53:58.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:53:58.313+0000 7fb0dfdc5640 1 -- 172.21.15.31:0/3791835483 wait complete. 2023-10-30T13:53:58.338 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:00.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.489+0000 7f89ad51d640 1 Processor -- start 2023-10-30T13:54:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.489+0000 7f89ad51d640 1 -- start start 2023-10-30T13:54:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89ad51d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f89a80c4a10 0x7f89a815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89ad51d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a8159f20 con 0x7f89a8153c80 2023-10-30T13:54:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89ad51d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a815d620 con 0x7f89a80c42a0 2023-10-30T13:54:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89ad51d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a815dec0 con 0x7f89a80c4a10 2023-10-30T13:54:00.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a6ffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f89a8153c80 0x7f89a815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41652/0 (socket says 172.21.15.31:41652) 2023-10-30T13:54:00.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a6ffd640 1 -- 172.21.15.31:0/1899427353 learned_addr learned my addr 172.21.15.31:0/1899427353 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:00.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f899ffff640 1 --2- 172.21.15.31:0/1899427353 >> v2:172.21.15.33:3300/0 conn(0x7f89a80c4a10 0x7f89a815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328446264 0 0) 0x7f89a8159f20 con 0x7f89a8153c80 2023-10-30T13:54:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8994003530 con 0x7f89a8153c80 2023-10-30T13:54:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587738358 0 0) 0x7f89a815d620 con 0x7f89a80c42a0 2023-10-30T13:54:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a8159f20 con 0x7f89a80c42a0 2023-10-30T13:54:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3355350742 0 0) 0x7f8994003530 con 0x7f89a8153c80 2023-10-30T13:54:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 >> v2:172.21.15.33:3300/0 conn(0x7f89a80c4a10 msgr2=0x7f89a815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 --2- 172.21.15.31:0/1899427353 >> v2:172.21.15.33:3300/0 conn(0x7f89a80c4a10 0x7f89a815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 >> v1:172.21.15.33:6789/0 conn(0x7f89a80c42a0 legacy=0x7f89a80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a815d900 con 0x7f89a8153c80 2023-10-30T13:54:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f8998001680 con 0x7f89a8153c80 2023-10-30T13:54:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 >> v1:172.21.15.31:6789/0 conn(0x7f89a8153c80 legacy=0x7f89a815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994003b10 0x7f8994003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89940047d0 0x7f8994004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8994005190 0x7f899400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a815dec0 con 0x7f89940047d0 2023-10-30T13:54:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899400bf80 con 0x7f8994003b10 2023-10-30T13:54:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899400c140 con 0x7f8994005190 2023-10-30T13:54:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f899ffff640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89940047d0 0x7f8994004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a6ffd640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994003b10 0x7f8994003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a77fe640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8994005190 0x7f899400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f899ffff640 1 -- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8994005190 msgr2=0x7f899400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f899ffff640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8994005190 0x7f899400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f899ffff640 1 -- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994003b10 msgr2=0x7f8994003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:00.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f899ffff640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994003b10 0x7f8994003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f899ffff640 1 -- 172.21.15.31:0/1899427353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994003530 con 0x7f89940047d0 2023-10-30T13:54:00.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.493+0000 7f89a77fe640 1 -- 172.21.15.31:0/1899427353 reap_dead start 2023-10-30T13:54:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f899ffff640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89940047d0 0x7f8994004c50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f89a80c4e30 tx=0x7f899001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f45fefe5bba5ffc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89900a64c0 con 0x7f89940047d0 2023-10-30T13:54:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89900a6620 con 0x7f89940047d0 2023-10-30T13:54:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89a67fc640 1 -- 172.21.15.31:0/1899427353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89900a69d0 con 0x7f89940047d0 2023-10-30T13:54:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89ad51d640 1 -- 172.21.15.31:0/1899427353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89940047d0 msgr2=0x7f8994004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89940047d0 0x7f8994004c50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f89a80c4e30 tx=0x7f899001c9d0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89ad51d640 1 -- 172.21.15.31:0/1899427353 shutdown_connections 2023-10-30T13:54:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1899427353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89940047d0 0x7f8994004c50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89ad51d640 1 -- 172.21.15.31:0/1899427353 >> 172.21.15.31:0/1899427353 conn(0x7f89a80715c0 msgr2=0x7f89a80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89ad51d640 1 -- 172.21.15.31:0/1899427353 shutdown_connections 2023-10-30T13:54:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.497+0000 7f89ad51d640 1 -- 172.21.15.31:0/1899427353 wait complete. 2023-10-30T13:54:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 Processor -- start 2023-10-30T13:54:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 -- start start 2023-10-30T13:54:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89940047d0 0x7f89a814a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a80c7ba0 0x7f89a8151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89a80b3dd0 0x7f89a80b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a80c8610 con 0x7f89a80c7ba0 2023-10-30T13:54:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a814b1b0 con 0x7f89940047d0 2023-10-30T13:54:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a80c8050 con 0x7f89a80b3dd0 2023-10-30T13:54:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a80c7ba0 0x7f89a8151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a80c7ba0 0x7f89a8151310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33944/0 (socket says 172.21.15.31:33944) 2023-10-30T13:54:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 -- 172.21.15.31:0/1156234128 learned_addr learned my addr 172.21.15.31:0/1156234128 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:00.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89a77fe640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89a80b3dd0 0x7f89a80b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 -- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89a80b3dd0 msgr2=0x7f89a80b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89a80b3dd0 0x7f89a80b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89a6ffd640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89940047d0 0x7f89a814a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 -- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89940047d0 msgr2=0x7f89a814a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89940047d0 0x7f89a814a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 -- 172.21.15.31:0/1156234128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a80c8330 con 0x7f89a80c7ba0 2023-10-30T13:54:00.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89a6ffd640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89940047d0 0x7f89a814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:54:00.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899ffff640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a80c7ba0 0x7f89a8151310 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f899001c800 tx=0x7f8990019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899f7fe640 1 -- 172.21.15.31:0/1156234128 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89900a6a60 con 0x7f89a80c7ba0 2023-10-30T13:54:00.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899f7fe640 1 -- 172.21.15.31:0/1156234128 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89900a6bc0 con 0x7f89a80c7ba0 2023-10-30T13:54:00.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a814aaf0 con 0x7f89a80c7ba0 2023-10-30T13:54:00.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f899f7fe640 1 -- 172.21.15.31:0/1156234128 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990004030 con 0x7f89a80c7ba0 2023-10-30T13:54:00.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a8151850 con 0x7f89a80c7ba0 2023-10-30T13:54:00.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.501+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a814a950 con 0x7f89a80c7ba0 2023-10-30T13:54:00.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.505+0000 7f899f7fe640 1 -- 172.21.15.31:0/1156234128 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8990004190 con 0x7f89a80c7ba0 2023-10-30T13:54:00.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.505+0000 7f899f7fe640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89880664a0 0x7f8988068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.505+0000 7f899f7fe640 1 -- 172.21.15.31:0/1156234128 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f89900aa070 con 0x7f89a80c7ba0 2023-10-30T13:54:00.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.505+0000 7f89a6ffd640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89880664a0 0x7f8988068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.505+0000 7f89a6ffd640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89880664a0 0x7f8988068960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f89a8063a60 tx=0x7f8998001480 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.509+0000 7f899f7fe640 1 -- 172.21.15.31:0/1156234128 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f89a814a950 con 0x7f89a80c7ba0 2023-10-30T13:54:00.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.697+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89a814a950 con 0x7f89a80c7ba0 2023-10-30T13:54:00.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.697+0000 7f899f7fe640 1 -- 172.21.15.31:0/1156234128 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f89a814a950 con 0x7f89a80c7ba0 2023-10-30T13:54:00.698 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:00.699 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:57.737809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:00.699 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 79 2023-10-30T13:54:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89880664a0 msgr2=0x7f8988068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89880664a0 0x7f8988068960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f89a8063a60 tx=0x7f8998001480 comp rx=0 tx=0).stop 2023-10-30T13:54:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a80c7ba0 msgr2=0x7f89a8151310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a80c7ba0 0x7f89a8151310 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f899001c800 tx=0x7f8990019770 comp rx=0 tx=0).stop 2023-10-30T13:54:00.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 shutdown_connections 2023-10-30T13:54:00.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89880664a0 0x7f8988068960 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f89a80b3dd0 0x7f89a80b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a80c7ba0 0x7f89a8151310 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 --2- 172.21.15.31:0/1156234128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89940047d0 0x7f89a814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 >> 172.21.15.31:0/1156234128 conn(0x7f89a80715c0 msgr2=0x7f89a80c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 shutdown_connections 2023-10-30T13:54:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:00.701+0000 7f89ad51d640 1 -- 172.21.15.31:0/1156234128 wait complete. 2023-10-30T13:54:00.719 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6201": { "addr": "172.21.15.31:6837/2267972067", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 2267972067, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 2267972067, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 78, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:53:57.737809+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6201 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:54:02.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:02.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2c1405640 1 Processor -- start 2023-10-30T13:54:02.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2c1405640 1 -- start start 2023-10-30T13:54:02.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2c1405640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff2bc153c80 0x7ff2bc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2c1405640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff2bc0c42a0 0x7ff2bc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2c1405640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2bc159f20 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2c1405640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc15dec0 con 0x7ff2bc153c80 2023-10-30T13:54:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2c1405640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc1599a0 con 0x7ff2bc0c42a0 2023-10-30T13:54:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2baffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff2bc153c80 0x7ff2bc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2baffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff2bc153c80 0x7ff2bc15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33954/0 (socket says 172.21.15.31:33954) 2023-10-30T13:54:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2baffd640 1 -- 172.21.15.31:0/1946093699 learned_addr learned my addr 172.21.15.31:0/1946093699 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2bb7fe640 1 --2- 172.21.15.31:0/1946093699 >> v2:172.21.15.33:3300/0 conn(0x7ff2bc0c42a0 0x7ff2bc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2baffd640 1 -- 172.21.15.31:0/1946093699 >> v2:172.21.15.33:3300/0 conn(0x7ff2bc0c42a0 msgr2=0x7ff2bc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2baffd640 1 --2- 172.21.15.31:0/1946093699 >> v2:172.21.15.33:3300/0 conn(0x7ff2bc0c42a0 0x7ff2bc0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2baffd640 1 -- 172.21.15.31:0/1946093699 >> v1:172.21.15.33:6789/0 conn(0x7ff2bc0c4a10 legacy=0x7ff2bc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.877+0000 7ff2baffd640 1 -- 172.21.15.31:0/1946093699 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc15d620 con 0x7ff2bc153c80 2023-10-30T13:54:02.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2baffd640 1 --2- 172.21.15.31:0/1946093699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc153c80 0x7ff2bc15cf90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0b1d40 tx=0x7ff2ac01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bc87dd900653d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:02.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2ba7fc640 1 -- 172.21.15.31:0/1946093699 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac0a89b0 con 0x7ff2bc153c80 2023-10-30T13:54:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2ba7fc640 1 -- 172.21.15.31:0/1946093699 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ac0a8b10 con 0x7ff2bc153c80 2023-10-30T13:54:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2ba7fc640 1 -- 172.21.15.31:0/1946093699 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac0a8ea0 con 0x7ff2bc153c80 2023-10-30T13:54:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 -- 172.21.15.31:0/1946093699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc153c80 msgr2=0x7ff2bc15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1946093699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc153c80 0x7ff2bc15cf90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0b1d40 tx=0x7ff2ac01ecb0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 -- 172.21.15.31:0/1946093699 shutdown_connections 2023-10-30T13:54:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1946093699 >> v2:172.21.15.33:3300/0 conn(0x7ff2bc0c42a0 0x7ff2bc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1946093699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc153c80 0x7ff2bc15cf90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 -- 172.21.15.31:0/1946093699 >> 172.21.15.31:0/1946093699 conn(0x7ff2bc0715c0 msgr2=0x7ff2bc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 -- 172.21.15.31:0/1946093699 shutdown_connections 2023-10-30T13:54:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.881+0000 7ff2c1405640 1 -- 172.21.15.31:0/1946093699 wait complete. 2023-10-30T13:54:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 Processor -- start 2023-10-30T13:54:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 -- start start 2023-10-30T13:54:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2bc0c42a0 0x7ff2bc14ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc0c4a10 0x7ff2bc15c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2bc153c80 0x7ff2bc152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc15c910 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc15cb10 con 0x7ff2bc153c80 2023-10-30T13:54:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc15cdf0 con 0x7ff2bc0c42a0 2023-10-30T13:54:02.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc0c4a10 0x7ff2bc15c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc0c4a10 0x7ff2bc15c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33966/0 (socket says 172.21.15.31:33966) 2023-10-30T13:54:02.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 -- 172.21.15.31:0/1213019384 learned_addr learned my addr 172.21.15.31:0/1213019384 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:02.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2baffd640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2bc0c42a0 0x7ff2bc14ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2bb7fe640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2bc153c80 0x7ff2bc152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 -- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2bc0c42a0 msgr2=0x7ff2bc14ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2bc0c42a0 0x7ff2bc14ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 -- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2bc153c80 msgr2=0x7ff2bc152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2bc153c80 0x7ff2bc152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 -- 172.21.15.31:0/1213019384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0094d0 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2baffd640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2bc0c42a0 0x7ff2bc14ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b3fff640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc0c4a10 0x7ff2bc15c3d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400c410 tx=0x7ff2a400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b37fe640 1 -- 172.21.15.31:0/1213019384 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a4011790 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b37fe640 1 -- 172.21.15.31:0/1213019384 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a40118f0 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2b37fe640 1 -- 172.21.15.31:0/1213019384 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a4011ce0 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2bc1f27d0 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.885+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc1f2d90 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.889+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc04e170 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.889+0000 7ff2b37fe640 1 -- 172.21.15.31:0/1213019384 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff2a401c460 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.889+0000 7ff2b37fe640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff29c0664a0 0x7ff29c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.889+0000 7ff2b37fe640 1 -- 172.21.15.31:0/1213019384 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7ff2a4083fe0 con 0x7ff2bc0c4a10 2023-10-30T13:54:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.889+0000 7ff2baffd640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff29c0664a0 0x7ff29c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.889+0000 7ff2baffd640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff29c0664a0 0x7ff29c068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0b2000 tx=0x7ff2ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:02.893+0000 7ff2b37fe640 1 -- 172.21.15.31:0/1213019384 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2a4051040 con 0x7ff2bc0c4a10 2023-10-30T13:54:03.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2bc0cad40 con 0x7ff2bc0c4a10 2023-10-30T13:54:03.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2b37fe640 1 -- 172.21.15.31:0/1213019384 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5705 (secure 0 0 0) 0x7ff2a4054660 con 0x7ff2bc0c4a10 2023-10-30T13:54:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:03.085 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:53:57.737809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 79 2023-10-30T13:54:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff29c0664a0 msgr2=0x7ff29c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff29c0664a0 0x7ff29c068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0b2000 tx=0x7ff2ac008040 comp rx=0 tx=0).stop 2023-10-30T13:54:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc0c4a10 msgr2=0x7ff2bc15c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc0c4a10 0x7ff2bc15c3d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400c410 tx=0x7ff2a400c8d0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 shutdown_connections 2023-10-30T13:54:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff29c0664a0 0x7ff29c068960 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2bc153c80 0x7ff2bc152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2bc0c4a10 0x7ff2bc15c3d0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 --2- 172.21.15.31:0/1213019384 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2bc0c42a0 0x7ff2bc14ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 >> 172.21.15.31:0/1213019384 conn(0x7ff2bc0715c0 msgr2=0x7ff2bc1585d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 shutdown_connections 2023-10-30T13:54:03.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:03.085+0000 7ff2c1405640 1 -- 172.21.15.31:0/1213019384 wait complete. 2023-10-30T13:54:03.105 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:05.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 Processor -- start 2023-10-30T13:54:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 -- start start 2023-10-30T13:54:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dbe0 0x7fb06406e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb06406dfc0 0x7fb064156880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb064071f30 0x7fb064157270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06406ea70 con 0x7fb06406dfc0 2023-10-30T13:54:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb064156dc0 con 0x7fb064071f30 2023-10-30T13:54:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb069b01640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb064070c40 con 0x7fb06406dbe0 2023-10-30T13:54:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb062ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb06406dfc0 0x7fb064156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb062ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb06406dfc0 0x7fb064156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44982/0 (socket says 172.21.15.31:44982) 2023-10-30T13:54:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb062ffd640 1 -- 172.21.15.31:0/1798499971 learned_addr learned my addr 172.21.15.31:0/1798499971 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb063fff640 1 --2- 172.21.15.31:0/1798499971 >> v2:172.21.15.33:3300/0 conn(0x7fb064071f30 0x7fb064157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.261+0000 7fb0637fe640 1 --2- 172.21.15.31:0/1798499971 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dbe0 0x7fb06406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb062ffd640 1 -- 172.21.15.31:0/1798499971 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dbe0 msgr2=0x7fb06406e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb062ffd640 1 --2- 172.21.15.31:0/1798499971 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dbe0 0x7fb06406e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb062ffd640 1 -- 172.21.15.31:0/1798499971 >> v2:172.21.15.33:3300/0 conn(0x7fb064071f30 msgr2=0x7fb064157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb062ffd640 1 --2- 172.21.15.31:0/1798499971 >> v2:172.21.15.33:3300/0 conn(0x7fb064071f30 0x7fb064157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb062ffd640 1 -- 172.21.15.31:0/1798499971 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0641577b0 con 0x7fb06406dfc0 2023-10-30T13:54:05.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb062ffd640 1 --2- 172.21.15.31:0/1798499971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb06406dfc0 0x7fb064156880 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb064072de0 tx=0x7fb05801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc381ca53ee418a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:05.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb0627fc640 1 -- 172.21.15.31:0/1798499971 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0580ad680 con 0x7fb06406dfc0 2023-10-30T13:54:05.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb0627fc640 1 -- 172.21.15.31:0/1798499971 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0580ad7e0 con 0x7fb06406dfc0 2023-10-30T13:54:05.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb0627fc640 1 -- 172.21.15.31:0/1798499971 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0580adb70 con 0x7fb06406dfc0 2023-10-30T13:54:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb069b01640 1 -- 172.21.15.31:0/1798499971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb06406dfc0 msgr2=0x7fb064156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb069b01640 1 --2- 172.21.15.31:0/1798499971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb06406dfc0 0x7fb064156880 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb064072de0 tx=0x7fb05801c440 comp rx=0 tx=0).stop 2023-10-30T13:54:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb069b01640 1 -- 172.21.15.31:0/1798499971 shutdown_connections 2023-10-30T13:54:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb069b01640 1 --2- 172.21.15.31:0/1798499971 >> v2:172.21.15.33:3300/0 conn(0x7fb064071f30 0x7fb064157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb069b01640 1 --2- 172.21.15.31:0/1798499971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb06406dfc0 0x7fb064156880 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb069b01640 1 --2- 172.21.15.31:0/1798499971 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dbe0 0x7fb06406e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.265+0000 7fb069b01640 1 -- 172.21.15.31:0/1798499971 >> 172.21.15.31:0/1798499971 conn(0x7fb0640c7f60 msgr2=0x7fb0640c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 -- 172.21.15.31:0/1798499971 shutdown_connections 2023-10-30T13:54:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 -- 172.21.15.31:0/1798499971 wait complete. 2023-10-30T13:54:05.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 Processor -- start 2023-10-30T13:54:05.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 -- start start 2023-10-30T13:54:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06406dbe0 0x7fb064155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dfc0 0x7fb0640ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb064071f30 0x7fb0640ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0640c60d0 con 0x7fb064071f30 2023-10-30T13:54:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06414bd40 con 0x7fb06406dbe0 2023-10-30T13:54:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb069b01640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06413eb00 con 0x7fb06406dfc0 2023-10-30T13:54:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb064071f30 0x7fb0640ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb064071f30 0x7fb0640ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44994/0 (socket says 172.21.15.31:44994) 2023-10-30T13:54:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 -- 172.21.15.31:0/3267570285 learned_addr learned my addr 172.21.15.31:0/3267570285 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb0637fe640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06406dbe0 0x7fb064155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb062ffd640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dfc0 0x7fb0640ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 -- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dfc0 msgr2=0x7fb0640ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dfc0 0x7fb0640ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 -- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06406dbe0 msgr2=0x7fb064155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06406dbe0 0x7fb064155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 -- 172.21.15.31:0/3267570285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0580094d0 con 0x7fb064071f30 2023-10-30T13:54:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.269+0000 7fb063fff640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb064071f30 0x7fb0640ca580 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00efc0 tx=0x7fb05c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb04bfff640 1 -- 172.21.15.31:0/3267570285 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c014480 con 0x7fb064071f30 2023-10-30T13:54:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb04bfff640 1 -- 172.21.15.31:0/3267570285 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c00f040 con 0x7fb064071f30 2023-10-30T13:54:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb04bfff640 1 -- 172.21.15.31:0/3267570285 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c014820 con 0x7fb064071f30 2023-10-30T13:54:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0641f3920 con 0x7fb064071f30 2023-10-30T13:54:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06414c2c0 con 0x7fb064071f30 2023-10-30T13:54:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0640c6240 con 0x7fb064071f30 2023-10-30T13:54:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb04bfff640 1 -- 172.21.15.31:0/3267570285 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb05c014980 con 0x7fb064071f30 2023-10-30T13:54:05.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.273+0000 7fb04bfff640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0500664a0 0x7fb050068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:05.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.277+0000 7fb04bfff640 1 -- 172.21.15.31:0/3267570285 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fb05c084ca0 con 0x7fb064071f30 2023-10-30T13:54:05.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.277+0000 7fb0637fe640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0500664a0 0x7fb050068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:05.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.277+0000 7fb0637fe640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0500664a0 0x7fb050068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb054009ac0 tx=0x7fb054009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:05.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.281+0000 7fb04bfff640 1 -- 172.21.15.31:0/3267570285 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb05c051d90 con 0x7fb064071f30 2023-10-30T13:54:05.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.469+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb064071420 con 0x7fb064071f30 2023-10-30T13:54:05.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.469+0000 7fb04bfff640 1 -- 172.21.15.31:0/3267570285 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fb05c018020 con 0x7fb064071f30 2023-10-30T13:54:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:05.471 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:03.375885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:05.472 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 80 2023-10-30T13:54:05.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0500664a0 msgr2=0x7fb050068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:05.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0500664a0 0x7fb050068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb054009ac0 tx=0x7fb054009190 comp rx=0 tx=0).stop 2023-10-30T13:54:05.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb064071f30 msgr2=0x7fb0640ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:05.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb064071f30 0x7fb0640ca580 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00efc0 tx=0x7fb05c00c490 comp rx=0 tx=0).stop 2023-10-30T13:54:05.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 shutdown_connections 2023-10-30T13:54:05.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb0500664a0 0x7fb050068960 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb064071f30 0x7fb0640ca580 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb06406dfc0 0x7fb0640ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 --2- 172.21.15.31:0/3267570285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06406dbe0 0x7fb064155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:05.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 >> 172.21.15.31:0/3267570285 conn(0x7fb0640c7f60 msgr2=0x7fb06414a450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:05.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 shutdown_connections 2023-10-30T13:54:05.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:05.473+0000 7fb069b01640 1 -- 172.21.15.31:0/3267570285 wait complete. 2023-10-30T13:54:05.493 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:07.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:07.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.649+0000 7ffbfa805640 1 Processor -- start 2023-10-30T13:54:07.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.649+0000 7ffbfa805640 1 -- start start 2023-10-30T13:54:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbfa805640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffbf4153aa0 0x7ffbf415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbfa805640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf40701e0 0x7ffbf40ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbfa805640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf40c7b00 con 0x7ffbf40cd420 2023-10-30T13:54:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbfa805640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf4070a50 con 0x7ffbf4153aa0 2023-10-30T13:54:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbfa805640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf415dd80 con 0x7ffbf40701e0 2023-10-30T13:54:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ffbf4153aa0 0x7ffbf415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf8da2640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7ffbf40cd420 0x7ffbf406b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54020/0 (socket says 172.21.15.31:54020) 2023-10-30T13:54:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf8da2640 1 -- 172.21.15.31:0/3032830215 learned_addr learned my addr 172.21.15.31:0/3032830215 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf37fe640 1 --2- 172.21.15.31:0/3032830215 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf40701e0 0x7ffbf40ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf3fff640 1 -- 172.21.15.31:0/3032830215 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf40701e0 msgr2=0x7ffbf40ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf3fff640 1 --2- 172.21.15.31:0/3032830215 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf40701e0 0x7ffbf40ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf3fff640 1 -- 172.21.15.31:0/3032830215 >> v1:172.21.15.31:6789/0 conn(0x7ffbf40cd420 legacy=0x7ffbf406b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:07.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf3fff640 1 -- 172.21.15.31:0/3032830215 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf40c2050 con 0x7ffbf4153aa0 2023-10-30T13:54:07.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf2ffd640 1 -- 172.21.15.31:0/3032830215 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028374981 0 0) 0x7ffbf40c7b00 con 0x7ffbf40cd420 2023-10-30T13:54:07.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.653+0000 7ffbf37fe640 1 --2- 172.21.15.31:0/3032830215 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf40701e0 0x7ffbf40ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:07.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbf3fff640 1 --2- 172.21.15.31:0/3032830215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf4153aa0 0x7ffbf415d840 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40097c0 tx=0x7ffbe401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b744d1e0314d5593 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:07.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbf2ffd640 1 -- 172.21.15.31:0/3032830215 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe40a8970 con 0x7ffbf4153aa0 2023-10-30T13:54:07.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbf2ffd640 1 -- 172.21.15.31:0/3032830215 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe40a8ad0 con 0x7ffbf4153aa0 2023-10-30T13:54:07.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbf2ffd640 1 -- 172.21.15.31:0/3032830215 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe40a8e60 con 0x7ffbf4153aa0 2023-10-30T13:54:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 -- 172.21.15.31:0/3032830215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf4153aa0 msgr2=0x7ffbf415d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3032830215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf4153aa0 0x7ffbf415d840 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40097c0 tx=0x7ffbe401ec70 comp rx=0 tx=0).stop 2023-10-30T13:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 -- 172.21.15.31:0/3032830215 shutdown_connections 2023-10-30T13:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3032830215 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf40701e0 0x7ffbf40ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3032830215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf4153aa0 0x7ffbf415d840 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 -- 172.21.15.31:0/3032830215 >> 172.21.15.31:0/3032830215 conn(0x7ffbf4071fb0 msgr2=0x7ffbf40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 -- 172.21.15.31:0/3032830215 shutdown_connections 2023-10-30T13:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 -- 172.21.15.31:0/3032830215 wait complete. 2023-10-30T13:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 Processor -- start 2023-10-30T13:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.657+0000 7ffbfa805640 1 -- start start 2023-10-30T13:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbf40701e0 0x7ffbf4159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf40cd420 0x7ffbf4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf4153aa0 0x7ffbf415d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf4158e60 con 0x7ffbf40cd420 2023-10-30T13:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf4159500 con 0x7ffbf40701e0 2023-10-30T13:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf40cce20 con 0x7ffbf4153aa0 2023-10-30T13:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf40cd420 0x7ffbf4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf40cd420 0x7ffbf4159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45012/0 (socket says 172.21.15.31:45012) 2023-10-30T13:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 -- 172.21.15.31:0/3748401801 learned_addr learned my addr 172.21.15.31:0/3748401801 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf3fff640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbf40701e0 0x7ffbf4159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf8da2640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf4153aa0 0x7ffbf415d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 -- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf4153aa0 msgr2=0x7ffbf415d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf4153aa0 0x7ffbf415d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 -- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbf40701e0 msgr2=0x7ffbf4159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbf40701e0 0x7ffbf4159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 -- 172.21.15.31:0/3748401801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe4009420 con 0x7ffbf40cd420 2023-10-30T13:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf37fe640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf40cd420 0x7ffbf4159ed0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800e840 tx=0x7ffbe800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf0ff9640 1 -- 172.21.15.31:0/3748401801 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe800c8d0 con 0x7ffbf40cd420 2023-10-30T13:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf0ff9640 1 -- 172.21.15.31:0/3748401801 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe800ca30 con 0x7ffbf40cd420 2023-10-30T13:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbf0ff9640 1 -- 172.21.15.31:0/3748401801 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe800cdc0 con 0x7ffbf40cd420 2023-10-30T13:54:07.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf41588a0 con 0x7ffbf40cd420 2023-10-30T13:54:07.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf4159190 con 0x7ffbf40cd420 2023-10-30T13:54:07.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.661+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf4158ff0 con 0x7ffbf40cd420 2023-10-30T13:54:07.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.665+0000 7ffbf0ff9640 1 -- 172.21.15.31:0/3748401801 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffbe8013410 con 0x7ffbf40cd420 2023-10-30T13:54:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.665+0000 7ffbf0ff9640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbe00664d0 0x7ffbe0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.665+0000 7ffbf0ff9640 1 -- 172.21.15.31:0/3748401801 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7ffbe8084050 con 0x7ffbf40cd420 2023-10-30T13:54:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.665+0000 7ffbf3fff640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbe00664d0 0x7ffbe0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.665+0000 7ffbf3fff640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbe00664d0 0x7ffbe0068990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40b2000 tx=0x7ffbe4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.669+0000 7ffbf0ff9640 1 -- 172.21.15.31:0/3748401801 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffbf4158ff0 con 0x7ffbf40cd420 2023-10-30T13:54:07.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.861+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbf4158ff0 con 0x7ffbf40cd420 2023-10-30T13:54:07.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.861+0000 7ffbf0ff9640 1 -- 172.21.15.31:0/3748401801 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5702 (secure 0 0 0) 0x7ffbf4158ff0 con 0x7ffbf40cd420 2023-10-30T13:54:07.862 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:07.862 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:03.375885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:07.863 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 80 2023-10-30T13:54:07.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.861+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbe00664d0 msgr2=0x7ffbe0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:07.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.861+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbe00664d0 0x7ffbe0068990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40b2000 tx=0x7ffbe4008040 comp rx=0 tx=0).stop 2023-10-30T13:54:07.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.861+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf40cd420 msgr2=0x7ffbf4159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:07.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.861+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf40cd420 0x7ffbf4159ed0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800e840 tx=0x7ffbe800ed00 comp rx=0 tx=0).stop 2023-10-30T13:54:07.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 shutdown_connections 2023-10-30T13:54:07.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ffbe00664d0 0x7ffbe0068990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ffbf4153aa0 0x7ffbf415d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbf40cd420 0x7ffbf4159ed0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 --2- 172.21.15.31:0/3748401801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbf40701e0 0x7ffbf4159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:07.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 >> 172.21.15.31:0/3748401801 conn(0x7ffbf4071fb0 msgr2=0x7ffbf40c7540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:07.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 shutdown_connections 2023-10-30T13:54:07.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:07.865+0000 7ffbfa805640 1 -- 172.21.15.31:0/3748401801 wait complete. 2023-10-30T13:54:07.884 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:09.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:10.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.037+0000 7fb4313f8640 1 Processor -- start 2023-10-30T13:54:10.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.037+0000 7fb4313f8640 1 -- start start 2023-10-30T13:54:10.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb4313f8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 0x7fb42c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb4313f8640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c0c2310 con 0x7fb42c150830 2023-10-30T13:54:10.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb4313f8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c156710 con 0x7fb42c0cb7d0 2023-10-30T13:54:10.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb4313f8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c06bc10 con 0x7fb42c070520 2023-10-30T13:54:10.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb42c150830 0x7fb42c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54034/0 (socket says 172.21.15.31:54034) 2023-10-30T13:54:10.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42affd640 1 -- 172.21.15.31:0/2368636441 learned_addr learned my addr 172.21.15.31:0/2368636441 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:10.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42b7fe640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 0x7fb42c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186917767 0 0) 0x7fb42c0c2310 con 0x7fb42c150830 2023-10-30T13:54:10.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb418003530 con 0x7fb42c150830 2023-10-30T13:54:10.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 991894988 0 0) 0x7fb418003530 con 0x7fb42c150830 2023-10-30T13:54:10.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 msgr2=0x7fb42c0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:10.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 0x7fb42c0709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 >> v1:172.21.15.33:6789/0 conn(0x7fb42c0cb7d0 legacy=0x7fb42c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:10.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0c4360 con 0x7fb42c150830 2023-10-30T13:54:10.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fb41c001680 con 0x7fb42c150830 2023-10-30T13:54:10.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 >> v1:172.21.15.31:6789/0 conn(0x7fb42c150830 legacy=0x7fb42c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:10.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 0x7fb4180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb418003f30 0x7fb4180047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418004d10 0x7fb418005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c06bc10 con 0x7fb418003f30 2023-10-30T13:54:10.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41800bb00 con 0x7fb418004d10 2023-10-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41800ded0 con 0x7fb42c070520 2023-10-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb423fff640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb418003f30 0x7fb4180047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42affd640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 0x7fb4180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb42b7fe640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418004d10 0x7fb418005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb423fff640 1 -- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 msgr2=0x7fb4180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:10.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb423fff640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb42c070520 0x7fb4180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb423fff640 1 -- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418004d10 msgr2=0x7fb418005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:10.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb423fff640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418004d10 0x7fb418005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.041+0000 7fb423fff640 1 -- 172.21.15.31:0/2368636441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb418003530 con 0x7fb418003f30 2023-10-30T13:54:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb423fff640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb418003f30 0x7fb4180047d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb414001780 tx=0x7fb41401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ccc8ae4be5c59b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4140050d0 con 0x7fb418003f30 2023-10-30T13:54:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4140a7050 con 0x7fb418003f30 2023-10-30T13:54:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb42a7fc640 1 -- 172.21.15.31:0/2368636441 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb414005490 con 0x7fb418003f30 2023-10-30T13:54:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb4313f8640 1 -- 172.21.15.31:0/2368636441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb418003f30 msgr2=0x7fb4180047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb4313f8640 1 --2- 172.21.15.31:0/2368636441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb418003f30 0x7fb4180047d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb414001780 tx=0x7fb41401c400 comp rx=0 tx=0).stop 2023-10-30T13:54:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb42b7fe640 1 -- 172.21.15.31:0/2368636441 reap_dead start 2023-10-30T13:54:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb4313f8640 1 -- 172.21.15.31:0/2368636441 shutdown_connections 2023-10-30T13:54:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb4313f8640 1 -- 172.21.15.31:0/2368636441 >> 172.21.15.31:0/2368636441 conn(0x7fb42c0cac70 msgr2=0x7fb42c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb4313f8640 1 -- 172.21.15.31:0/2368636441 shutdown_connections 2023-10-30T13:54:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.045+0000 7fb4313f8640 1 -- 172.21.15.31:0/2368636441 wait complete. 2023-10-30T13:54:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 Processor -- start 2023-10-30T13:54:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 -- start start 2023-10-30T13:54:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb418003f30 0x7fb42c0c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb42c0c66f0 0x7fb42c14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42c14af20 0x7fb42c1473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c14a2e0 con 0x7fb42c14af20 2023-10-30T13:54:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c14a5c0 con 0x7fb42c0c66f0 2023-10-30T13:54:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c152090 con 0x7fb418003f30 2023-10-30T13:54:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42c14af20 0x7fb42c1473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42c14af20 0x7fb42c1473d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45034/0 (socket says 172.21.15.31:45034) 2023-10-30T13:54:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 -- 172.21.15.31:0/1307062898 learned_addr learned my addr 172.21.15.31:0/1307062898 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb423fff640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb42c0c66f0 0x7fb42c14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42affd640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb418003f30 0x7fb42c0c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 -- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb418003f30 msgr2=0x7fb42c0c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb418003f30 0x7fb42c0c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 -- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb42c0c66f0 msgr2=0x7fb42c14a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb42c0c66f0 0x7fb42c14a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 -- 172.21.15.31:0/1307062898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0ca210 con 0x7fb42c14af20 2023-10-30T13:54:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb42b7fe640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42c14af20 0x7fb42c1473d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb42c1541f0 tx=0x7fb4240115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4237fe640 1 -- 172.21.15.31:0/1307062898 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400e070 con 0x7fb42c14af20 2023-10-30T13:54:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4237fe640 1 -- 172.21.15.31:0/1307062898 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb424011e40 con 0x7fb42c14af20 2023-10-30T13:54:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c0ca480 con 0x7fb42c14af20 2023-10-30T13:54:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4237fe640 1 -- 172.21.15.31:0/1307062898 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4240039a0 con 0x7fb42c14af20 2023-10-30T13:54:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c1ea470 con 0x7fb42c14af20 2023-10-30T13:54:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.049+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c14a700 con 0x7fb42c14af20 2023-10-30T13:54:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.053+0000 7fb4237fe640 1 -- 172.21.15.31:0/1307062898 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb424003b00 con 0x7fb42c14af20 2023-10-30T13:54:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.053+0000 7fb4237fe640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb40c0664a0 0x7fb40c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.053+0000 7fb4237fe640 1 -- 172.21.15.31:0/1307062898 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fb4240848c0 con 0x7fb42c14af20 2023-10-30T13:54:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.053+0000 7fb42affd640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb40c0664a0 0x7fb40c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.053+0000 7fb42affd640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb40c0664a0 0x7fb40c068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb41c007d30 tx=0x7fb41c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.057+0000 7fb4237fe640 1 -- 172.21.15.31:0/1307062898 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb42c14a700 con 0x7fb42c14af20 2023-10-30T13:54:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.249+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb42c14a700 con 0x7fb42c14af20 2023-10-30T13:54:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.249+0000 7fb4237fe640 1 -- 172.21.15.31:0/1307062898 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5708 (secure 0 0 0) 0x7fb42c14a700 con 0x7fb42c14af20 2023-10-30T13:54:10.251 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:10.251 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:08.485154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:10.251 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 81 2023-10-30T13:54:10.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb40c0664a0 msgr2=0x7fb40c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:10.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb40c0664a0 0x7fb40c068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb41c007d30 tx=0x7fb41c009040 comp rx=0 tx=0).stop 2023-10-30T13:54:10.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42c14af20 msgr2=0x7fb42c1473d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:10.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42c14af20 0x7fb42c1473d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb42c1541f0 tx=0x7fb4240115a0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 shutdown_connections 2023-10-30T13:54:10.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb40c0664a0 0x7fb40c068960 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42c14af20 0x7fb42c1473d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb42c0c66f0 0x7fb42c14a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 --2- 172.21.15.31:0/1307062898 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb418003f30 0x7fb42c0c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:10.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 >> 172.21.15.31:0/1307062898 conn(0x7fb42c0cac70 msgr2=0x7fb42c0c36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:10.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 shutdown_connections 2023-10-30T13:54:10.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:10.253+0000 7fb4313f8640 1 -- 172.21.15.31:0/1307062898 wait complete. 2023-10-30T13:54:10.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:12.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:12.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f36846640 1 Processor -- start 2023-10-30T13:54:12.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f36846640 1 -- start start 2023-10-30T13:54:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f36846640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f30153c80 0x7f7f3015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f36846640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7f300c4a10 0x7f7f300c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f36846640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f30159f20 con 0x7f7f3015a3b0 2023-10-30T13:54:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f36846640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3015dec0 con 0x7f7f300c4a10 2023-10-30T13:54:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f36846640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f301599a0 con 0x7f7f30153c80 2023-10-30T13:54:12.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f34de3640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7f3015a3b0 0x7f7f300ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54046/0 (socket says 172.21.15.31:54046) 2023-10-30T13:54:12.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f34de3640 1 -- 172.21.15.31:0/1165744126 learned_addr learned my addr 172.21.15.31:0/1165744126 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:12.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2f7fe640 1 --2- 172.21.15.31:0/1165744126 >> v2:172.21.15.31:3300/0 conn(0x7f7f300c4a10 0x7f7f300c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:12.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2ffff640 1 --2- 172.21.15.31:0/1165744126 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f30153c80 0x7f7f3015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:12.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2effd640 1 -- 172.21.15.31:0/1165744126 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974599320 0 0) 0x7f7f30159f20 con 0x7f7f3015a3b0 2023-10-30T13:54:12.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2f7fe640 1 -- 172.21.15.31:0/1165744126 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f30153c80 msgr2=0x7f7f3015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:12.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2f7fe640 1 --2- 172.21.15.31:0/1165744126 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f30153c80 0x7f7f3015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2f7fe640 1 -- 172.21.15.31:0/1165744126 >> v1:172.21.15.31:6789/0 conn(0x7f7f3015a3b0 legacy=0x7f7f300ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:12.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2f7fe640 1 -- 172.21.15.31:0/1165744126 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f300c47e0 con 0x7f7f300c4a10 2023-10-30T13:54:12.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.429+0000 7f7f2ffff640 1 --2- 172.21.15.31:0/1165744126 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f30153c80 0x7f7f3015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f2f7fe640 1 --2- 172.21.15.31:0/1165744126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f300c4a10 0x7f7f300c42a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7f240097c0 tx=0x7f7f2401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fb865cee77cfc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f2effd640 1 -- 172.21.15.31:0/1165744126 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f240a8970 con 0x7f7f300c4a10 2023-10-30T13:54:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f2effd640 1 -- 172.21.15.31:0/1165744126 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f240a8ad0 con 0x7f7f300c4a10 2023-10-30T13:54:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f2effd640 1 -- 172.21.15.31:0/1165744126 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f240a8e60 con 0x7f7f300c4a10 2023-10-30T13:54:12.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 -- 172.21.15.31:0/1165744126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f300c4a10 msgr2=0x7f7f300c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:12.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 --2- 172.21.15.31:0/1165744126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f300c4a10 0x7f7f300c42a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7f240097c0 tx=0x7f7f2401ec70 comp rx=0 tx=0).stop 2023-10-30T13:54:12.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 -- 172.21.15.31:0/1165744126 shutdown_connections 2023-10-30T13:54:12.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 --2- 172.21.15.31:0/1165744126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f300c4a10 0x7f7f300c42a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 --2- 172.21.15.31:0/1165744126 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f30153c80 0x7f7f3015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 -- 172.21.15.31:0/1165744126 >> 172.21.15.31:0/1165744126 conn(0x7f7f300715c0 msgr2=0x7f7f300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:12.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 -- 172.21.15.31:0/1165744126 shutdown_connections 2023-10-30T13:54:12.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.433+0000 7f7f36846640 1 -- 172.21.15.31:0/1165744126 wait complete. 2023-10-30T13:54:12.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 Processor -- start 2023-10-30T13:54:12.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 -- start start 2023-10-30T13:54:12.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f300c4a10 0x7f7f300b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:12.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f30153c80 0x7f7f3015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:12.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f3015a3b0 0x7f7f300ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:12.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f300b5130 con 0x7f7f30153c80 2023-10-30T13:54:12.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3015c9c0 con 0x7f7f300c4a10 2023-10-30T13:54:12.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f30157680 con 0x7f7f3015a3b0 2023-10-30T13:54:12.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f30153c80 0x7f7f3015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:12.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f30153c80 0x7f7f3015ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45052/0 (socket says 172.21.15.31:45052) 2023-10-30T13:54:12.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 -- 172.21.15.31:0/2362936874 learned_addr learned my addr 172.21.15.31:0/2362936874 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:12.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2ffff640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f300c4a10 0x7f7f300b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f34de3640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f3015a3b0 0x7f7f300ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 -- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f3015a3b0 msgr2=0x7f7f300ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f3015a3b0 0x7f7f300ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 -- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f300c4a10 msgr2=0x7f7f300b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f300c4a10 0x7f7f300b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 -- 172.21.15.31:0/2362936874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f24009420 con 0x7f7f30153c80 2023-10-30T13:54:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2f7fe640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f30153c80 0x7f7f3015ab60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f240b3040 tx=0x7f7f2401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:12.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2cff9640 1 -- 172.21.15.31:0/2362936874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f240b9630 con 0x7f7f30153c80 2023-10-30T13:54:12.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2cff9640 1 -- 172.21.15.31:0/2362936874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f240b97b0 con 0x7f7f30153c80 2023-10-30T13:54:12.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f2cff9640 1 -- 172.21.15.31:0/2362936874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f240b9b60 con 0x7f7f30153c80 2023-10-30T13:54:12.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f300b3b00 con 0x7f7f30153c80 2023-10-30T13:54:12.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.437+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f30157940 con 0x7f7f30153c80 2023-10-30T13:54:12.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.441+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3004e170 con 0x7f7f30153c80 2023-10-30T13:54:12.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.441+0000 7f7f2cff9640 1 -- 172.21.15.31:0/2362936874 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f240b7040 con 0x7f7f30153c80 2023-10-30T13:54:12.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.441+0000 7f7f2cff9640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f1c0664d0 0x7f7f1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:12.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.441+0000 7f7f2cff9640 1 -- 172.21.15.31:0/2362936874 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f7f24122260 con 0x7f7f30153c80 2023-10-30T13:54:12.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.441+0000 7f7f2ffff640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f1c0664d0 0x7f7f1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:12.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.441+0000 7f7f2ffff640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f1c0664d0 0x7f7f1c068990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000c900 tx=0x7f7f20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.445+0000 7f7f2cff9640 1 -- 172.21.15.31:0/2362936874 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f240ef330 con 0x7f7f30153c80 2023-10-30T13:54:12.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.633+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f3015cb00 con 0x7f7f30153c80 2023-10-30T13:54:12.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.633+0000 7f7f2cff9640 1 -- 172.21.15.31:0/2362936874 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5708 (secure 0 0 0) 0x7f7f240f2950 con 0x7f7f30153c80 2023-10-30T13:54:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:08.485154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:12.636 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 81 2023-10-30T13:54:12.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f1c0664d0 msgr2=0x7f7f1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:12.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f1c0664d0 0x7f7f1c068990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000c900 tx=0x7f7f20008040 comp rx=0 tx=0).stop 2023-10-30T13:54:12.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f30153c80 msgr2=0x7f7f3015ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:12.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f30153c80 0x7f7f3015ab60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f240b3040 tx=0x7f7f2401ba10 comp rx=0 tx=0).stop 2023-10-30T13:54:12.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 shutdown_connections 2023-10-30T13:54:12.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f1c0664d0 0x7f7f1c068990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f3015a3b0 0x7f7f300ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f30153c80 0x7f7f3015ab60 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 --2- 172.21.15.31:0/2362936874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f300c4a10 0x7f7f300b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 >> 172.21.15.31:0/2362936874 conn(0x7f7f300715c0 msgr2=0x7f7f30148a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 shutdown_connections 2023-10-30T13:54:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:12.637+0000 7f7f36846640 1 -- 172.21.15.31:0/2362936874 wait complete. 2023-10-30T13:54:12.657 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 81, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6201": { "addr": "172.21.15.31:6837/2267972067", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 2267972067, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 2267972067, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 78, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:54:08.485154+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6201 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:54:14.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:14.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.805+0000 7fe227ec9640 1 Processor -- start 2023-10-30T13:54:14.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.805+0000 7fe227ec9640 1 -- start start 2023-10-30T13:54:14.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.805+0000 7fe227ec9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe220159980 0x7fe220159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.805+0000 7fe227ec9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2200c5f80 con 0x7fe22006c420 2023-10-30T13:54:14.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe227ec9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2200c5690 con 0x7fe220153bb0 2023-10-30T13:54:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe227ec9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22015a2d0 con 0x7fe220159980 2023-10-30T13:54:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225464640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fe22006c420 0x7fe2200c6c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:60484/0 (socket says 172.21.15.31:60484) 2023-10-30T13:54:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225464640 1 -- 172.21.15.31:0/283330096 learned_addr learned my addr 172.21.15.31:0/283330096 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe226466640 1 --2- 172.21.15.31:0/283330096 >> v2:172.21.15.33:3300/0 conn(0x7fe220159980 0x7fe220159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865645880 0 0) 0x7fe2200c5f80 con 0x7fe22006c420 2023-10-30T13:54:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c003530 con 0x7fe22006c420 2023-10-30T13:54:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626356047 0 0) 0x7fe20c003530 con 0x7fe22006c420 2023-10-30T13:54:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 >> v2:172.21.15.33:3300/0 conn(0x7fe220159980 msgr2=0x7fe220159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 --2- 172.21.15.31:0/283330096 >> v2:172.21.15.33:3300/0 conn(0x7fe220159980 0x7fe220159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 >> v1:172.21.15.33:6789/0 conn(0x7fe220153bb0 legacy=0x7fe2200c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22015a580 con 0x7fe22006c420 2023-10-30T13:54:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fe210001680 con 0x7fe22006c420 2023-10-30T13:54:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 >> v1:172.21.15.31:6789/0 conn(0x7fe22006c420 legacy=0x7fe2200c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe20c003980 0x7fe20c003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe20c004380 0x7fe20c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c005050 0x7fe20c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22015a2d0 con 0x7fe20c003980 2023-10-30T13:54:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c00bf00 con 0x7fe20c005050 2023-10-30T13:54:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c00c0c0 con 0x7fe20c004380 2023-10-30T13:54:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225c65640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe20c003980 0x7fe20c003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe226466640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c005050 0x7fe20c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225464640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe20c004380 0x7fe20c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225c65640 1 -- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe20c004380 msgr2=0x7fe20c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225c65640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe20c004380 0x7fe20c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:14.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225c65640 1 -- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c005050 msgr2=0x7fe20c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:14.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225c65640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c005050 0x7fe20c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:14.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe225c65640 1 -- 172.21.15.31:0/283330096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c003530 con 0x7fe20c003980 2023-10-30T13:54:14.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe226466640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c005050 0x7fe20c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:14.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.809+0000 7fe226466640 1 -- 172.21.15.31:0/283330096 reap_dead start 2023-10-30T13:54:14.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe225c65640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe20c003980 0x7fe20c003d60 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe21c001860 tx=0x7fe21c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb66fe4adfe8b019 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:14.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c0a8870 con 0x7fe20c003980 2023-10-30T13:54:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c0a89d0 con 0x7fe20c003980 2023-10-30T13:54:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe224c63640 1 -- 172.21.15.31:0/283330096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c0a8dc0 con 0x7fe20c003980 2023-10-30T13:54:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 -- 172.21.15.31:0/283330096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe20c003980 msgr2=0x7fe20c003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe20c003980 0x7fe20c003d60 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe21c001860 tx=0x7fe21c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:54:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 -- 172.21.15.31:0/283330096 shutdown_connections 2023-10-30T13:54:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 --2- 172.21.15.31:0/283330096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe20c003980 0x7fe20c003d60 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 -- 172.21.15.31:0/283330096 >> 172.21.15.31:0/283330096 conn(0x7fe2200cac70 msgr2=0x7fe2200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 -- 172.21.15.31:0/283330096 shutdown_connections 2023-10-30T13:54:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 -- 172.21.15.31:0/283330096 wait complete. 2023-10-30T13:54:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 Processor -- start 2023-10-30T13:54:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.813+0000 7fe227ec9640 1 -- start start 2023-10-30T13:54:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c003980 0x7fe2200c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2201e3a30 0x7fe2200c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2201515d0 0x7fe22014d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200c2040 con 0x7fe2201515d0 2023-10-30T13:54:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2201ef380 con 0x7fe20c003980 2023-10-30T13:54:14.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200c6aa0 con 0x7fe2201e3a30 2023-10-30T13:54:14.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2201515d0 0x7fe22014d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2201515d0 0x7fe22014d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33656/0 (socket says 172.21.15.31:33656) 2023-10-30T13:54:14.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 -- 172.21.15.31:0/2595907725 learned_addr learned my addr 172.21.15.31:0/2595907725 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:14.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe225464640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2201e3a30 0x7fe2200c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe225c65640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c003980 0x7fe2200c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 -- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2201e3a30 msgr2=0x7fe2200c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:14.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2201e3a30 0x7fe2200c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:14.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 -- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c003980 msgr2=0x7fe2200c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:14.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c003980 0x7fe2200c81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:14.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 -- 172.21.15.31:0/2595907725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2201eea50 con 0x7fe2201515d0 2023-10-30T13:54:14.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe226466640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2201515d0 0x7fe22014d430 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe22006ff30 tx=0x7fe218010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:14.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe2167fc640 1 -- 172.21.15.31:0/2595907725 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21801b070 con 0x7fe2201515d0 2023-10-30T13:54:14.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe2167fc640 1 -- 172.21.15.31:0/2595907725 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21800f040 con 0x7fe2201515d0 2023-10-30T13:54:14.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe2167fc640 1 -- 172.21.15.31:0/2595907725 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21800e050 con 0x7fe2201515d0 2023-10-30T13:54:14.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2201eed30 con 0x7fe2201515d0 2023-10-30T13:54:14.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2201ee700 con 0x7fe2201515d0 2023-10-30T13:54:14.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2201ef530 con 0x7fe2201515d0 2023-10-30T13:54:14.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.817+0000 7fe2167fc640 1 -- 172.21.15.31:0/2595907725 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe218015440 con 0x7fe2201515d0 2023-10-30T13:54:14.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.821+0000 7fe2167fc640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe2040664a0 0x7fe204068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:14.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.821+0000 7fe2167fc640 1 -- 172.21.15.31:0/2595907725 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fe2180844e0 con 0x7fe2201515d0 2023-10-30T13:54:14.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.821+0000 7fe225c65640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe2040664a0 0x7fe204068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:14.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.821+0000 7fe225c65640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe2040664a0 0x7fe204068960 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0b2000 tx=0x7fe21c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:14.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:14.825+0000 7fe2167fc640 1 -- 172.21.15.31:0/2595907725 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2201ef530 con 0x7fe2201515d0 2023-10-30T13:54:15.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.013+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe220072260 con 0x7fe2201515d0 2023-10-30T13:54:15.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.013+0000 7fe2167fc640 1 -- 172.21.15.31:0/2595907725 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5708 (secure 0 0 0) 0x7fe218051520 con 0x7fe2201515d0 2023-10-30T13:54:15.015 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:15.015 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:08.485154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 81 2023-10-30T13:54:15.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe2040664a0 msgr2=0x7fe204068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:15.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe2040664a0 0x7fe204068960 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0b2000 tx=0x7fe21c009040 comp rx=0 tx=0).stop 2023-10-30T13:54:15.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2201515d0 msgr2=0x7fe22014d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:15.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2201515d0 0x7fe22014d430 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe22006ff30 tx=0x7fe218010590 comp rx=0 tx=0).stop 2023-10-30T13:54:15.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 shutdown_connections 2023-10-30T13:54:15.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe2040664a0 0x7fe204068960 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:15.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2201515d0 0x7fe22014d430 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:15.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe2201e3a30 0x7fe2200c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:15.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 --2- 172.21.15.31:0/2595907725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe20c003980 0x7fe2200c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:15.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 >> 172.21.15.31:0/2595907725 conn(0x7fe2200cac70 msgr2=0x7fe22014f860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:15.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 shutdown_connections 2023-10-30T13:54:15.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:15.017+0000 7fe227ec9640 1 -- 172.21.15.31:0/2595907725 wait complete. 2023-10-30T13:54:15.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:17.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.201+0000 7f5f50ff5640 1 Processor -- start 2023-10-30T13:54:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.201+0000 7f5f50ff5640 1 -- start start 2023-10-30T13:54:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f50ff5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f5f4c0c5c10 0x7f5f4c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f50ff5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f50ff5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c15cd50 con 0x7f5f4c159760 2023-10-30T13:54:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f50ff5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c15d0a0 con 0x7f5f4c0c5c10 2023-10-30T13:54:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f50ff5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0ca000 con 0x7f5f4c069e90 2023-10-30T13:54:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f43fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f5f4c159760 0x7f5f4c15d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:60486/0 (socket says 172.21.15.31:60486) 2023-10-30T13:54:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f43fff640 1 -- 172.21.15.31:0/1562936413 learned_addr learned my addr 172.21.15.31:0/1562936413 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f4ad9d640 1 --2- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f4a59c640 1 --2- 172.21.15.31:0/1562936413 >> v2:172.21.15.33:3300/0 conn(0x7f5f4c0c5c10 0x7f5f4c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:17.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f49d9b640 1 -- 172.21.15.31:0/1562936413 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840312659 0 0) 0x7f5f4c15cd50 con 0x7f5f4c159760 2023-10-30T13:54:17.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f49d9b640 1 -- 172.21.15.31:0/1562936413 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f38003530 con 0x7f5f4c159760 2023-10-30T13:54:17.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f4a59c640 1 -- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 msgr2=0x7f5f4c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f4a59c640 1 --2- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f4a59c640 1 -- 172.21.15.31:0/1562936413 >> v1:172.21.15.31:6789/0 conn(0x7f5f4c159760 legacy=0x7f5f4c15d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f49d9b640 1 -- 172.21.15.31:0/1562936413 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3597534006 0 0) 0x7f5f38003530 con 0x7f5f4c159760 2023-10-30T13:54:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.205+0000 7f5f4a59c640 1 -- 172.21.15.31:0/1562936413 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c159cf0 con 0x7f5f4c0c5c10 2023-10-30T13:54:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f4a59c640 1 --2- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 0x7f5f4c06eef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0097c0 tx=0x7f5f3c01c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ecc784208a52bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f49d9b640 1 -- 172.21.15.31:0/1562936413 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34001680 con 0x7f5f4c0c5c10 2023-10-30T13:54:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f49d9b640 1 -- 172.21.15.31:0/1562936413 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c0a7050 con 0x7f5f4c0c5c10 2023-10-30T13:54:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f49d9b640 1 -- 172.21.15.31:0/1562936413 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c0ab620 con 0x7f5f4c0c5c10 2023-10-30T13:54:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 -- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 msgr2=0x7f5f4c06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:17.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 0x7f5f4c06eef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0097c0 tx=0x7f5f3c01c370 comp rx=0 tx=0).stop 2023-10-30T13:54:17.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 -- 172.21.15.31:0/1562936413 shutdown_connections 2023-10-30T13:54:17.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/1562936413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 0x7f5f4c06eef0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 -- 172.21.15.31:0/1562936413 >> 172.21.15.31:0/1562936413 conn(0x7f5f4c06afa0 msgr2=0x7f5f4c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 -- 172.21.15.31:0/1562936413 shutdown_connections 2023-10-30T13:54:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 -- 172.21.15.31:0/1562936413 wait complete. 2023-10-30T13:54:17.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 Processor -- start 2023-10-30T13:54:17.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.209+0000 7f5f50ff5640 1 -- start start 2023-10-30T13:54:17.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c1f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:17.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 0x7f5f4c0cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:17.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c159760 0x7f5f4c15bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:17.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c154050 con 0x7f5f4c159760 2023-10-30T13:54:17.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cb250 con 0x7f5f4c0c5c10 2023-10-30T13:54:17.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cb810 con 0x7f5f4c069e90 2023-10-30T13:54:17.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c159760 0x7f5f4c15bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:17.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c159760 0x7f5f4c15bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33658/0 (socket says 172.21.15.31:33658) 2023-10-30T13:54:17.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 -- 172.21.15.31:0/3395460892 learned_addr learned my addr 172.21.15.31:0/3395460892 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:17.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f43fff640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 0x7f5f4c0cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:17.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4a59c640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c1f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:17.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 -- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 msgr2=0x7f5f4c1f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:17.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c1f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 -- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 msgr2=0x7f5f4c0cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:17.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 0x7f5f4c0cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 -- 172.21.15.31:0/3395460892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c009420 con 0x7f5f4c159760 2023-10-30T13:54:17.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f4ad9d640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c159760 0x7f5f4c15bc30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400ec40 tx=0x7f5f4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:17.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f42ffd640 1 -- 172.21.15.31:0/3395460892 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4400ccc0 con 0x7f5f4c159760 2023-10-30T13:54:17.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f42ffd640 1 -- 172.21.15.31:0/3395460892 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4400f040 con 0x7f5f4c159760 2023-10-30T13:54:17.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f42ffd640 1 -- 172.21.15.31:0/3395460892 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44014480 con 0x7f5f4c159760 2023-10-30T13:54:17.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c151a60 con 0x7f5f4c159760 2023-10-30T13:54:17.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c154580 con 0x7f5f4c159760 2023-10-30T13:54:17.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c0cb9c0 con 0x7f5f4c159760 2023-10-30T13:54:17.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.213+0000 7f5f42ffd640 1 -- 172.21.15.31:0/3395460892 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f440145e0 con 0x7f5f4c159760 2023-10-30T13:54:17.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.217+0000 7f5f42ffd640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5f2c0664a0 0x7f5f2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:17.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.217+0000 7f5f42ffd640 1 -- 172.21.15.31:0/3395460892 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f5f44084c40 con 0x7f5f4c159760 2023-10-30T13:54:17.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.217+0000 7f5f4a59c640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5f2c0664a0 0x7f5f2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:17.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.217+0000 7f5f4a59c640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5f2c0664a0 0x7f5f2c068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0ad000 tx=0x7f5f3c0a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:17.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.221+0000 7f5f42ffd640 1 -- 172.21.15.31:0/3395460892 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f4c0cb9c0 con 0x7f5f4c159760 2023-10-30T13:54:17.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f4c0cb9c0 con 0x7f5f4c159760 2023-10-30T13:54:17.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f42ffd640 1 -- 172.21.15.31:0/3395460892 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f5f4c0cb9c0 con 0x7f5f4c159760 2023-10-30T13:54:17.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:17.410 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:17.411 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2023-10-30T13:54:17.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5f2c0664a0 msgr2=0x7f5f2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:17.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5f2c0664a0 0x7f5f2c068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0ad000 tx=0x7f5f3c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:54:17.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c159760 msgr2=0x7f5f4c15bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:17.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c159760 0x7f5f4c15bc30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400ec40 tx=0x7f5f4400c490 comp rx=0 tx=0).stop 2023-10-30T13:54:17.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 shutdown_connections 2023-10-30T13:54:17.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5f2c0664a0 0x7f5f2c068960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c159760 0x7f5f4c15bc30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f4c0c5c10 0x7f5f4c0cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 --2- 172.21.15.31:0/3395460892 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5f4c069e90 0x7f5f4c1f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:17.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 >> 172.21.15.31:0/3395460892 conn(0x7f5f4c06afa0 msgr2=0x7f5f4c070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:17.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 shutdown_connections 2023-10-30T13:54:17.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:17.409+0000 7f5f50ff5640 1 -- 172.21.15.31:0/3395460892 wait complete. 2023-10-30T13:54:17.432 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5591 2023-10-30T13:54:17.432 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.d 2023-10-30T13:54:24.350 INFO:tasks.ceph.mds.a.smithi031.stderr:2023-10-30T13:54:24.349+0000 7fb2464fb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:54:24.433 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:54:24.433 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:54:24.433 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:54:24.436 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:54:24.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:24.517 INFO:tasks.ceph.mds.d.smithi033.stdout:starting mds.d at 2023-10-30T13:54:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.593+0000 7fc368de4640 1 Processor -- start 2023-10-30T13:54:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.593+0000 7fc368de4640 1 -- start start 2023-10-30T13:54:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc368de4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc36415cd70 0x7fc364069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc368de4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36415dcc0 con 0x7fc3640c8eb0 2023-10-30T13:54:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc368de4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc364159d80 con 0x7fc36415ac70 2023-10-30T13:54:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc368de4640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc364159800 con 0x7fc36415cd70 2023-10-30T13:54:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc3637fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc3640c8eb0 0x7fc36415a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:60494/0 (socket says 172.21.15.31:60494) 2023-10-30T13:54:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc3637fe640 1 -- 172.21.15.31:0/2565543281 learned_addr learned my addr 172.21.15.31:0/2565543281 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc363fff640 1 --2- 172.21.15.31:0/2565543281 >> v2:172.21.15.33:3300/0 conn(0x7fc36415cd70 0x7fc364069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc3627fc640 1 -- 172.21.15.31:0/2565543281 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756905810 0 0) 0x7fc354001120 con 0x7fc3640c8eb0 2023-10-30T13:54:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc3627fc640 1 -- 172.21.15.31:0/2565543281 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c003530 con 0x7fc3640c8eb0 2023-10-30T13:54:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc363fff640 1 -- 172.21.15.31:0/2565543281 >> v1:172.21.15.33:6789/0 conn(0x7fc36415ac70 legacy=0x7fc36406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc363fff640 1 -- 172.21.15.31:0/2565543281 >> v1:172.21.15.31:6789/0 conn(0x7fc3640c8eb0 legacy=0x7fc36415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.597+0000 7fc363fff640 1 -- 172.21.15.31:0/2565543281 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36415d1c0 con 0x7fc36415cd70 2023-10-30T13:54:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc363fff640 1 --2- 172.21.15.31:0/2565543281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 0x7fc364069e90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc35c009d90 tx=0x7fc35c01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd0c5185734ab6ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:24.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc3627fc640 1 -- 172.21.15.31:0/2565543281 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0af600 con 0x7fc36415cd70 2023-10-30T13:54:24.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc3627fc640 1 -- 172.21.15.31:0/2565543281 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c0ab050 con 0x7fc36415cd70 2023-10-30T13:54:24.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc3627fc640 1 -- 172.21.15.31:0/2565543281 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0af980 con 0x7fc36415cd70 2023-10-30T13:54:24.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc368de4640 1 -- 172.21.15.31:0/2565543281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 msgr2=0x7fc364069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc368de4640 1 --2- 172.21.15.31:0/2565543281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 0x7fc364069e90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc35c009d90 tx=0x7fc35c01f130 comp rx=0 tx=0).stop 2023-10-30T13:54:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc368de4640 1 -- 172.21.15.31:0/2565543281 shutdown_connections 2023-10-30T13:54:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc368de4640 1 --2- 172.21.15.31:0/2565543281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 0x7fc364069e90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc368de4640 1 -- 172.21.15.31:0/2565543281 >> 172.21.15.31:0/2565543281 conn(0x7fc3640cef00 msgr2=0x7fc3640cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc368de4640 1 -- 172.21.15.31:0/2565543281 shutdown_connections 2023-10-30T13:54:24.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.601+0000 7fc368de4640 1 -- 172.21.15.31:0/2565543281 wait complete. 2023-10-30T13:54:24.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 Processor -- start 2023-10-30T13:54:24.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 -- start start 2023-10-30T13:54:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3640c8eb0 0x7fc36406a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc36415ac70 0x7fc3641eee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 0x7fc3640cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3641f23b0 con 0x7fc3640c8eb0 2023-10-30T13:54:24.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640c7240 con 0x7fc36415cd70 2023-10-30T13:54:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36414f0a0 con 0x7fc36415ac70 2023-10-30T13:54:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3640c8eb0 0x7fc36406a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3640c8eb0 0x7fc36406a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33660/0 (socket says 172.21.15.31:33660) 2023-10-30T13:54:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 -- 172.21.15.31:0/1763043810 learned_addr learned my addr 172.21.15.31:0/1763043810 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc363fff640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 0x7fc3640cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:24.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc362ffd640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc36415ac70 0x7fc3641eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:24.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 -- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc36415ac70 msgr2=0x7fc3641eee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:24.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc36415ac70 0x7fc3641eee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 -- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 msgr2=0x7fc3640cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 0x7fc3640cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 -- 172.21.15.31:0/1763043810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c009a70 con 0x7fc3640c8eb0 2023-10-30T13:54:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc3637fe640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3640c8eb0 0x7fc36406a0f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc35400c500 tx=0x7fc35400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc34bfff640 1 -- 172.21.15.31:0/1763043810 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354005cf0 con 0x7fc3640c8eb0 2023-10-30T13:54:24.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc34bfff640 1 -- 172.21.15.31:0/1763043810 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc354005e50 con 0x7fc3640c8eb0 2023-10-30T13:54:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36414f380 con 0x7fc3640c8eb0 2023-10-30T13:54:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc34bfff640 1 -- 172.21.15.31:0/1763043810 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354007310 con 0x7fc3640c8eb0 2023-10-30T13:54:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.605+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3641f28e0 con 0x7fc3640c8eb0 2023-10-30T13:54:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.609+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36404e170 con 0x7fc3640c8eb0 2023-10-30T13:54:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.609+0000 7fc34bfff640 1 -- 172.21.15.31:0/1763043810 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc354007470 con 0x7fc3640c8eb0 2023-10-30T13:54:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.609+0000 7fc34bfff640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3500664a0 0x7fc350068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.609+0000 7fc34bfff640 1 -- 172.21.15.31:0/1763043810 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fc3540837f0 con 0x7fc3640c8eb0 2023-10-30T13:54:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.609+0000 7fc362ffd640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3500664a0 0x7fc350068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.613+0000 7fc362ffd640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3500664a0 0x7fc350068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc36415b8b0 tx=0x7fc358009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.613+0000 7fc34bfff640 1 -- 172.21.15.31:0/1763043810 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc354050860 con 0x7fc3640c8eb0 2023-10-30T13:54:24.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc36415dcc0 con 0x7fc3640c8eb0 2023-10-30T13:54:24.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc34bfff640 1 -- 172.21.15.31:0/1763043810 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fc36415dcc0 con 0x7fc3640c8eb0 2023-10-30T13:54:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:24.830 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:24.830 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2023-10-30T13:54:24.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3500664a0 msgr2=0x7fc350068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:24.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3500664a0 0x7fc350068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc36415b8b0 tx=0x7fc358009040 comp rx=0 tx=0).stop 2023-10-30T13:54:24.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3640c8eb0 msgr2=0x7fc36406a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:24.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3640c8eb0 0x7fc36406a0f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc35400c500 tx=0x7fc35400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 shutdown_connections 2023-10-30T13:54:24.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc3500664a0 0x7fc350068960 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc36415cd70 0x7fc3640cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc36415ac70 0x7fc3641eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 --2- 172.21.15.31:0/1763043810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3640c8eb0 0x7fc36406a0f0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.829+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 >> 172.21.15.31:0/1763043810 conn(0x7fc3640cef00 msgr2=0x7fc3640c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.833+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 shutdown_connections 2023-10-30T13:54:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:24.833+0000 7fc368de4640 1 -- 172.21.15.31:0/1763043810 wait complete. 2023-10-30T13:54:24.851 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:54:26.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:27.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.017+0000 7f7722091640 1 Processor -- start 2023-10-30T13:54:27.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.017+0000 7f7722091640 1 -- start start 2023-10-30T13:54:27.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f7722091640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f771c0c4a10 0x7f771c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f7722091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c42a0 0x7f771c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f7722091640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c159f20 con 0x7f771c153c80 2023-10-30T13:54:27.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f7722091640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c15dec0 con 0x7f771c0c4a10 2023-10-30T13:54:27.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f7722091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1599a0 con 0x7f771c0c42a0 2023-10-30T13:54:27.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771bfff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c42a0 0x7f771c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771affd640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f771c0c4a10 0x7f771c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771b7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f771c153c80 0x7f771c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:37564/0 (socket says 172.21.15.31:37564) 2023-10-30T13:54:27.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771b7fe640 1 -- 172.21.15.31:0/3072012488 learned_addr learned my addr 172.21.15.31:0/3072012488 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:54:27.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771bfff640 1 --2- 172.21.15.31:0/3072012488 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c42a0 0x7f771c0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:59364/0 (socket says 172.21.15.31:59364) 2023-10-30T13:54:27.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771a7fc640 1 -- 172.21.15.31:0/3072012488 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720809246 0 0) 0x7f771c159f20 con 0x7f771c153c80 2023-10-30T13:54:27.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771a7fc640 1 -- 172.21.15.31:0/3072012488 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704003530 con 0x7f771c153c80 2023-10-30T13:54:27.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771bfff640 1 -- 172.21.15.31:0/3072012488 >> v2:172.21.15.33:3300/0 conn(0x7f771c0c4a10 msgr2=0x7f771c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:27.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771bfff640 1 --2- 172.21.15.31:0/3072012488 >> v2:172.21.15.33:3300/0 conn(0x7f771c0c4a10 0x7f771c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771bfff640 1 -- 172.21.15.31:0/3072012488 >> v1:172.21.15.33:6789/0 conn(0x7f771c153c80 legacy=0x7f771c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.021+0000 7f771bfff640 1 -- 172.21.15.31:0/3072012488 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c15d620 con 0x7f771c0c42a0 2023-10-30T13:54:27.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f771bfff640 1 --2- 172.21.15.31:0/3072012488 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c42a0 0x7f771c0ca1e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7714009d90 tx=0x7f771401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd6bb9c201355b47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:27.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f771a7fc640 1 -- 172.21.15.31:0/3072012488 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140af600 con 0x7f771c0c42a0 2023-10-30T13:54:27.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f771a7fc640 1 -- 172.21.15.31:0/3072012488 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77140ab050 con 0x7f771c0c42a0 2023-10-30T13:54:27.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f771a7fc640 1 -- 172.21.15.31:0/3072012488 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140af960 con 0x7f771c0c42a0 2023-10-30T13:54:27.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 -- 172.21.15.31:0/3072012488 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c42a0 msgr2=0x7f771c0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 --2- 172.21.15.31:0/3072012488 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c42a0 0x7f771c0ca1e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7714009d90 tx=0x7f771401f130 comp rx=0 tx=0).stop 2023-10-30T13:54:27.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 -- 172.21.15.31:0/3072012488 shutdown_connections 2023-10-30T13:54:27.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 --2- 172.21.15.31:0/3072012488 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c42a0 0x7f771c0ca1e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 --2- 172.21.15.31:0/3072012488 >> v2:172.21.15.33:3300/0 conn(0x7f771c0c4a10 0x7f771c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 -- 172.21.15.31:0/3072012488 >> 172.21.15.31:0/3072012488 conn(0x7f771c0715c0 msgr2=0x7f771c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:27.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 -- 172.21.15.31:0/3072012488 shutdown_connections 2023-10-30T13:54:27.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.025+0000 7f7722091640 1 -- 172.21.15.31:0/3072012488 wait complete. 2023-10-30T13:54:27.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 Processor -- start 2023-10-30T13:54:27.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 -- start start 2023-10-30T13:54:27.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f771c0c42a0 0x7f771c0c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c4a10 0x7f771c149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f771c153c80 0x7f771c14a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c15b050 con 0x7f771c153c80 2023-10-30T13:54:27.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c15b330 con 0x7f771c0c42a0 2023-10-30T13:54:27.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c15b610 con 0x7f771c0c4a10 2023-10-30T13:54:27.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f771c153c80 0x7f771c14a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f771c0c42a0 0x7f771c0c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f771c153c80 0x7f771c14a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45302/0 (socket says 172.21.15.31:45302) 2023-10-30T13:54:27.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 -- 172.21.15.31:0/3683238931 learned_addr learned my addr 172.21.15.31:0/3683238931 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:54:27.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771affd640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c4a10 0x7f771c149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 -- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c4a10 msgr2=0x7f771c149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c4a10 0x7f771c149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 -- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f771c0c42a0 msgr2=0x7f771c0c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f771c0c42a0 0x7f771c0c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 -- 172.21.15.31:0/3683238931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7714009a70 con 0x7f771c153c80 2023-10-30T13:54:27.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f771bfff640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f771c153c80 0x7f771c14a050 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f771401cd60 tx=0x7f771401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:27.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7703fff640 1 -- 172.21.15.31:0/3683238931 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714007000 con 0x7f771c153c80 2023-10-30T13:54:27.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7703fff640 1 -- 172.21.15.31:0/3683238931 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7714009450 con 0x7f771c153c80 2023-10-30T13:54:27.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c15b8f0 con 0x7f771c153c80 2023-10-30T13:54:27.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7703fff640 1 -- 172.21.15.31:0/3683238931 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140ba420 con 0x7f771c153c80 2023-10-30T13:54:27.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.029+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c0b2510 con 0x7f771c153c80 2023-10-30T13:54:27.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.033+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c04e170 con 0x7f771c153c80 2023-10-30T13:54:27.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.033+0000 7f7703fff640 1 -- 172.21.15.31:0/3683238931 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f77140ba580 con 0x7f771c153c80 2023-10-30T13:54:27.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.033+0000 7f7703fff640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f77080664a0 0x7f7708068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.033+0000 7f7703fff640 1 -- 172.21.15.31:0/3683238931 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f7714122d40 con 0x7f771c153c80 2023-10-30T13:54:27.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.033+0000 7f771b7fe640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f77080664a0 0x7f7708068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.037+0000 7f771b7fe640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f77080664a0 0x7f7708068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f770c0011f0 tx=0x7f770c007d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:27.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.041+0000 7f7703fff640 1 -- 172.21.15.31:0/3683238931 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f77140efec0 con 0x7f771c153c80 2023-10-30T13:54:27.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f771c0cad40 con 0x7f771c153c80 2023-10-30T13:54:27.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7703fff640 1 -- 172.21.15.31:0/3683238931 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f77140b4020 con 0x7f771c153c80 2023-10-30T13:54:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:54:27.234 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:27.234 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:54:27.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f77080664a0 msgr2=0x7f7708068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f77080664a0 0x7f7708068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f770c0011f0 tx=0x7f770c007d40 comp rx=0 tx=0).stop 2023-10-30T13:54:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f771c153c80 msgr2=0x7f771c14a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f771c153c80 0x7f771c14a050 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f771401cd60 tx=0x7f771401bed0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 shutdown_connections 2023-10-30T13:54:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f77080664a0 0x7f7708068960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f771c153c80 0x7f771c14a050 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f771c0c4a10 0x7f771c149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 --2- 172.21.15.31:0/3683238931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f771c0c42a0 0x7f771c0c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 >> 172.21.15.31:0/3683238931 conn(0x7f771c0715c0 msgr2=0x7f771c1585e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 shutdown_connections 2023-10-30T13:54:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:54:27.233+0000 7f7722091640 1 -- 172.21.15.31:0/3683238931 wait complete. 2023-10-30T13:54:27.255 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:54:27.255 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 78 secs before thrashing 2023-10-30T13:54:28.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:28.195 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:42.411 INFO:tasks.workunit.client.0.smithi031.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2023-10-30T13:54:42.416 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into 'ceph'... 2023-10-30T13:54:58.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:58.252 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:28.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:28.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:45.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.403+0000 7fd9d6f1d640 1 Processor -- start 2023-10-30T13:55:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.403+0000 7fd9d6f1d640 1 -- start start 2023-10-30T13:55:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d6f1d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd9d0153c80 0x7fd9d015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d6f1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d6f1d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d0159f20 con 0x7fd9d00c4a10 2023-10-30T13:55:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d6f1d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d015dec0 con 0x7fd9d0153c80 2023-10-30T13:55:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d6f1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d01599a0 con 0x7fd9d00c42a0 2023-10-30T13:55:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9cffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fd9d00c4a10 0x7fd9d015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52304/0 (socket says 172.21.15.31:52304) 2023-10-30T13:55:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9cffff640 1 -- 172.21.15.31:0/3315357066 learned_addr learned my addr 172.21.15.31:0/3315357066 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:55:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d4cb9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd9d0153c80 0x7fd9d015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d54ba640 1 --2- 172.21.15.31:0/3315357066 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d4cb9640 1 -- 172.21.15.31:0/3315357066 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 msgr2=0x7fd9d00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d4cb9640 1 --2- 172.21.15.31:0/3315357066 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d4cb9640 1 -- 172.21.15.31:0/3315357066 >> v1:172.21.15.31:6789/0 conn(0x7fd9d00c4a10 legacy=0x7fd9d015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d4cb9640 1 -- 172.21.15.31:0/3315357066 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d015d620 con 0x7fd9d0153c80 2023-10-30T13:55:45.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9d54ba640 1 --2- 172.21.15.31:0/3315357066 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:45.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.407+0000 7fd9cf7fe640 1 -- 172.21.15.31:0/3315357066 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757378567 0 0) 0x7fd9d0159f20 con 0x7fd9d00c4a10 2023-10-30T13:55:45.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d4cb9640 1 --2- 172.21.15.31:0/3315357066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d015cf90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097c0 tx=0x7fd9c401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85de51d2f624f4da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9cf7fe640 1 -- 172.21.15.31:0/3315357066 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40a8970 con 0x7fd9d0153c80 2023-10-30T13:55:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9cf7fe640 1 -- 172.21.15.31:0/3315357066 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c40a8ad0 con 0x7fd9d0153c80 2023-10-30T13:55:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9cf7fe640 1 -- 172.21.15.31:0/3315357066 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40a8e60 con 0x7fd9d0153c80 2023-10-30T13:55:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/3315357066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 msgr2=0x7fd9d015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/3315357066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d015cf90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097c0 tx=0x7fd9c401ec70 comp rx=0 tx=0).stop 2023-10-30T13:55:45.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/3315357066 shutdown_connections 2023-10-30T13:55:45.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/3315357066 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/3315357066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d015cf90 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/3315357066 >> 172.21.15.31:0/3315357066 conn(0x7fd9d00715c0 msgr2=0x7fd9d00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:45.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/3315357066 shutdown_connections 2023-10-30T13:55:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/3315357066 wait complete. 2023-10-30T13:55:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 Processor -- start 2023-10-30T13:55:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- start start 2023-10-30T13:55:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9d00c4a10 0x7fd9d0145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d01460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d01eee40 con 0x7fd9d0153c80 2023-10-30T13:55:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d01efc60 con 0x7fd9d00c4a10 2023-10-30T13:55:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.411+0000 7fd9d6f1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d01ef050 con 0x7fd9d00c42a0 2023-10-30T13:55:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d01460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d01460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47270/0 (socket says 172.21.15.31:47270) 2023-10-30T13:55:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 -- 172.21.15.31:0/51730204 learned_addr learned my addr 172.21.15.31:0/51730204 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d4cb9640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9cffff640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9d00c4a10 0x7fd9d0145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 -- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 msgr2=0x7fd9d015c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d015c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 -- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9d00c4a10 msgr2=0x7fd9d0145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9d00c4a10 0x7fd9d0145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 -- 172.21.15.31:0/51730204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c4009420 con 0x7fd9d0153c80 2023-10-30T13:55:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9cffff640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9d00c4a10 0x7fd9d0145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d54ba640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d01460a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8007ef0 tx=0x7fd9c800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9cd7fa640 1 -- 172.21.15.31:0/51730204 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8003ba0 con 0x7fd9d0153c80 2023-10-30T13:55:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9cd7fa640 1 -- 172.21.15.31:0/51730204 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c800f040 con 0x7fd9d0153c80 2023-10-30T13:55:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d01ef330 con 0x7fd9d0153c80 2023-10-30T13:55:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9cd7fa640 1 -- 172.21.15.31:0/51730204 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c800de80 con 0x7fd9d0153c80 2023-10-30T13:55:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d01ef8f0 con 0x7fd9d0153c80 2023-10-30T13:55:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d004e170 con 0x7fd9d0153c80 2023-10-30T13:55:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.415+0000 7fd9cd7fa640 1 -- 172.21.15.31:0/51730204 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9c8004030 con 0x7fd9d0153c80 2023-10-30T13:55:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.419+0000 7fd9cd7fa640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9b40664d0 0x7fd9b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.419+0000 7fd9cd7fa640 1 -- 172.21.15.31:0/51730204 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fd9c8084390 con 0x7fd9d0153c80 2023-10-30T13:55:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.419+0000 7fd9d4cb9640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9b40664d0 0x7fd9b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.419+0000 7fd9d4cb9640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9b40664d0 0x7fd9b4068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40b2000 tx=0x7fd9c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.423+0000 7fd9cd7fa640 1 -- 172.21.15.31:0/51730204 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd9c80515b0 con 0x7fd9d0153c80 2023-10-30T13:55:45.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9d00cb360 con 0x7fd9d0153c80 2023-10-30T13:55:45.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9cd7fa640 1 -- 172.21.15.31:0/51730204 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fd9c8054bd0 con 0x7fd9d0153c80 2023-10-30T13:55:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:55:45.618 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:55:45.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9b40664d0 msgr2=0x7fd9b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9b40664d0 0x7fd9b4068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40b2000 tx=0x7fd9c4008040 comp rx=0 tx=0).stop 2023-10-30T13:55:45.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 msgr2=0x7fd9d01460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d01460a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8007ef0 tx=0x7fd9c800d4a0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 shutdown_connections 2023-10-30T13:55:45.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9b40664d0 0x7fd9b4068990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9d0153c80 0x7fd9d01460a0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9d00c4a10 0x7fd9d0145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 --2- 172.21.15.31:0/51730204 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd9d00c42a0 0x7fd9d015c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 >> 172.21.15.31:0/51730204 conn(0x7fd9d00715c0 msgr2=0x7fd9d006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 shutdown_connections 2023-10-30T13:55:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.615+0000 7fd9d6f1d640 1 -- 172.21.15.31:0/51730204 wait complete. 2023-10-30T13:55:45.638 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:55:45.639 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:55:45.639 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:55:45.661 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:55:45.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:45.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.843+0000 7f8d06365640 1 Processor -- start 2023-10-30T13:55:45.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.843+0000 7f8d06365640 1 -- start start 2023-10-30T13:55:45.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8d06365640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8d00153c80 0x7f8d0015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8d06365640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8d000c4a10 0x7f8d000c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8d06365640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d00159f20 con 0x7f8d0015a3b0 2023-10-30T13:55:45.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8d06365640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0015dec0 con 0x7f8d00153c80 2023-10-30T13:55:45.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8d06365640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d001599a0 con 0x7f8d000c4a10 2023-10-30T13:55:45.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8d00153c80 0x7f8d0015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8d00153c80 0x7f8d0015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47274/0 (socket says 172.21.15.31:47274) 2023-10-30T13:55:45.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 -- 172.21.15.31:0/2050074815 learned_addr learned my addr 172.21.15.31:0/2050074815 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/2050074815 >> v2:172.21.15.33:3300/0 conn(0x7f8d000c4a10 0x7f8d000c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 -- 172.21.15.31:0/2050074815 >> v2:172.21.15.33:3300/0 conn(0x7f8d000c4a10 msgr2=0x7f8d000c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 --2- 172.21.15.31:0/2050074815 >> v2:172.21.15.33:3300/0 conn(0x7f8d000c4a10 0x7f8d000c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 -- 172.21.15.31:0/2050074815 >> v1:172.21.15.33:6789/0 conn(0x7f8d0015a3b0 legacy=0x7f8d000ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 -- 172.21.15.31:0/2050074815 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d000c47e0 con 0x7f8d00153c80 2023-10-30T13:55:45.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfffff640 1 --2- 172.21.15.31:0/2050074815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d0015cf90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8d0006be50 tx=0x7f8cf001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac6440e12263688 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:45.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.847+0000 7f8cfeffd640 1 -- 172.21.15.31:0/2050074815 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf00a89f0 con 0x7f8d00153c80 2023-10-30T13:55:45.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8cfeffd640 1 -- 172.21.15.31:0/2050074815 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf00a8b50 con 0x7f8d00153c80 2023-10-30T13:55:45.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8cfeffd640 1 -- 172.21.15.31:0/2050074815 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf00b0440 con 0x7f8d00153c80 2023-10-30T13:55:45.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- 172.21.15.31:0/2050074815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 msgr2=0x7f8d0015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 --2- 172.21.15.31:0/2050074815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d0015cf90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8d0006be50 tx=0x7f8cf001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- 172.21.15.31:0/2050074815 shutdown_connections 2023-10-30T13:55:45.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 --2- 172.21.15.31:0/2050074815 >> v2:172.21.15.33:3300/0 conn(0x7f8d000c4a10 0x7f8d000c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 --2- 172.21.15.31:0/2050074815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d0015cf90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- 172.21.15.31:0/2050074815 >> 172.21.15.31:0/2050074815 conn(0x7f8d000715c0 msgr2=0x7f8d000719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:45.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- 172.21.15.31:0/2050074815 shutdown_connections 2023-10-30T13:55:45.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- 172.21.15.31:0/2050074815 wait complete. 2023-10-30T13:55:45.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 Processor -- start 2023-10-30T13:55:45.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- start start 2023-10-30T13:55:45.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d000c4a10 0x7f8d00152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d001f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d0015a3b0 0x7f8d00145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d001531f0 con 0x7f8d00153c80 2023-10-30T13:55:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0015ab60 con 0x7f8d0015a3b0 2023-10-30T13:55:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d06365640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0015ae00 con 0x7f8d000c4a10 2023-10-30T13:55:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8cff7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d001f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8cff7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d001f23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47290/0 (socket says 172.21.15.31:47290) 2023-10-30T13:55:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8cff7fe640 1 -- 172.21.15.31:0/1718716191 learned_addr learned my addr 172.21.15.31:0/1718716191 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8cfffff640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d000c4a10 0x7f8d00152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.851+0000 7f8d04902640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d0015a3b0 0x7f8d00145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cff7fe640 1 -- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d000c4a10 msgr2=0x7f8d00152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d000c4a10 0x7f8d00152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cff7fe640 1 -- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d0015a3b0 msgr2=0x7f8d00145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:45.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d0015a3b0 0x7f8d00145a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:45.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cff7fe640 1 -- 172.21.15.31:0/1718716191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf00094d0 con 0x7f8d00153c80 2023-10-30T13:55:45.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d001f23e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400d7d0 tx=0x7f8cf400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:45.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cfcff9640 1 -- 172.21.15.31:0/1718716191 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400b6c0 con 0x7f8d00153c80 2023-10-30T13:55:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cfcff9640 1 -- 172.21.15.31:0/1718716191 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf400b820 con 0x7f8d00153c80 2023-10-30T13:55:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d001f2cb0 con 0x7f8d00153c80 2023-10-30T13:55:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cfcff9640 1 -- 172.21.15.31:0/1718716191 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400bbd0 con 0x7f8d00153c80 2023-10-30T13:55:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0015c820 con 0x7f8d00153c80 2023-10-30T13:55:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0004e170 con 0x7f8d00153c80 2023-10-30T13:55:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cfcff9640 1 -- 172.21.15.31:0/1718716191 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8cf400bd30 con 0x7f8d00153c80 2023-10-30T13:55:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cfcff9640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8cec0664a0 0x7f8cec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cfcff9640 1 -- 172.21.15.31:0/1718716191 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f8cf40848b0 con 0x7f8d00153c80 2023-10-30T13:55:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.855+0000 7f8cfffff640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8cec0664a0 0x7f8cec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.859+0000 7f8cfffff640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8cec0664a0 0x7f8cec068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00b2000 tx=0x7f8cf0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:45.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:45.859+0000 7f8cfcff9640 1 -- 172.21.15.31:0/1718716191 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8cf4051870 con 0x7f8d00153c80 2023-10-30T13:55:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.047+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d000cad40 con 0x7f8d00153c80 2023-10-30T13:55:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.047+0000 7f8cfcff9640 1 -- 172.21.15.31:0/1718716191 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f8cf4054e90 con 0x7f8d00153c80 2023-10-30T13:55:46.052 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:55:46.052 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:55:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8cec0664a0 msgr2=0x7f8cec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8cec0664a0 0x7f8cec068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00b2000 tx=0x7f8cf0008040 comp rx=0 tx=0).stop 2023-10-30T13:55:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 msgr2=0x7f8d001f23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d001f23e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400d7d0 tx=0x7f8cf400dc90 comp rx=0 tx=0).stop 2023-10-30T13:55:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 shutdown_connections 2023-10-30T13:55:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8cec0664a0 0x7f8cec068960 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d0015a3b0 0x7f8d00145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d00153c80 0x7f8d001f23e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 --2- 172.21.15.31:0/1718716191 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d000c4a10 0x7f8d00152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 >> 172.21.15.31:0/1718716191 conn(0x7f8d000715c0 msgr2=0x7f8d000caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 shutdown_connections 2023-10-30T13:55:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:46.051+0000 7f8d06365640 1 -- 172.21.15.31:0/1718716191 wait complete. 2023-10-30T13:55:46.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:55:48.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:48.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f8e8ea640 1 Processor -- start 2023-10-30T13:55:48.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f8e8ea640 1 -- start start 2023-10-30T13:55:48.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f8e8ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806c0a0 0x7f8f88072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:48.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f8e8ea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8806b500 con 0x7f8f8806fdf0 2023-10-30T13:55:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f8e8ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8814e410 con 0x7f8f8814e030 2023-10-30T13:55:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f8e8ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881582e0 con 0x7f8f8806c0a0 2023-10-30T13:55:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f877fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8f8806fdf0 0x7f8f8806b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52310/0 (socket says 172.21.15.31:52310) 2023-10-30T13:55:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f877fe640 1 -- 172.21.15.31:0/2881202054 learned_addr learned my addr 172.21.15.31:0/2881202054 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:55:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.227+0000 7f8f8ce87640 1 --2- 172.21.15.31:0/2881202054 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806c0a0 0x7f8f88072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f86ffd640 1 -- 172.21.15.31:0/2881202054 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987543761 0 0) 0x7f8f8806b500 con 0x7f8f8806fdf0 2023-10-30T13:55:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f86ffd640 1 -- 172.21.15.31:0/2881202054 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f70003530 con 0x7f8f8806fdf0 2023-10-30T13:55:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f8ce87640 1 -- 172.21.15.31:0/2881202054 >> v1:172.21.15.33:6789/0 conn(0x7f8f8814e030 legacy=0x7f8f880707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f86ffd640 1 -- 172.21.15.31:0/2881202054 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1815233781 0 0) 0x7f8f70003530 con 0x7f8f8806fdf0 2023-10-30T13:55:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f8ce87640 1 -- 172.21.15.31:0/2881202054 >> v1:172.21.15.31:6789/0 conn(0x7f8f8806fdf0 legacy=0x7f8f8806b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f8ce87640 1 -- 172.21.15.31:0/2881202054 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f880c5820 con 0x7f8f8806c0a0 2023-10-30T13:55:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f8ce87640 1 --2- 172.21.15.31:0/2881202054 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806c0a0 0x7f8f88072890 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009dc0 tx=0x7f8f8001c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5d303108994228f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f86ffd640 1 -- 172.21.15.31:0/2881202054 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c001680 con 0x7f8f8806c0a0 2023-10-30T13:55:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f86ffd640 1 -- 172.21.15.31:0/2881202054 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f800b0400 con 0x7f8f8806c0a0 2023-10-30T13:55:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.231+0000 7f8f86ffd640 1 -- 172.21.15.31:0/2881202054 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f800b0790 con 0x7f8f8806c0a0 2023-10-30T13:55:48.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/2881202054 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806c0a0 msgr2=0x7f8f88072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/2881202054 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806c0a0 0x7f8f88072890 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009dc0 tx=0x7f8f8001c740 comp rx=0 tx=0).stop 2023-10-30T13:55:48.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/2881202054 shutdown_connections 2023-10-30T13:55:48.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/2881202054 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806c0a0 0x7f8f88072890 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/2881202054 >> 172.21.15.31:0/2881202054 conn(0x7f8f88069e90 msgr2=0x7f8f8806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:48.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/2881202054 shutdown_connections 2023-10-30T13:55:48.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/2881202054 wait complete. 2023-10-30T13:55:48.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 Processor -- start 2023-10-30T13:55:48.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- start start 2023-10-30T13:55:48.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8806c0a0 0x7f8f880af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:48.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806fdf0 0x7f8f88152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:48.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8814e030 0x7f8f88156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:48.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881f2cb0 con 0x7f8f8814e030 2023-10-30T13:55:48.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f88157480 con 0x7f8f8806c0a0 2023-10-30T13:55:48.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8e8ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f88156120 con 0x7f8f8806fdf0 2023-10-30T13:55:48.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8ce87640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8814e030 0x7f8f88156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8ce87640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8814e030 0x7f8f88156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47294/0 (socket says 172.21.15.31:47294) 2023-10-30T13:55:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8ce87640 1 -- 172.21.15.31:0/69630598 learned_addr learned my addr 172.21.15.31:0/69630598 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f87fff640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8806c0a0 0x7f8f880af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f877fe640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806fdf0 0x7f8f88152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8ce87640 1 -- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806fdf0 msgr2=0x7f8f88152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8ce87640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806fdf0 0x7f8f88152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8ce87640 1 -- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8806c0a0 msgr2=0x7f8f880af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.235+0000 7f8f8ce87640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8806c0a0 0x7f8f880af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f8ce87640 1 -- 172.21.15.31:0/69630598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80009a70 con 0x7f8f8814e030 2023-10-30T13:55:48.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f8ce87640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8814e030 0x7f8f88156f40 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8f8001ccb0 tx=0x7f8f800194e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f84ff9640 1 -- 172.21.15.31:0/69630598 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f80003ba0 con 0x7f8f8814e030 2023-10-30T13:55:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f84ff9640 1 -- 172.21.15.31:0/69630598 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f800b6040 con 0x7f8f8814e030 2023-10-30T13:55:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f881564a0 con 0x7f8f8814e030 2023-10-30T13:55:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f84ff9640 1 -- 172.21.15.31:0/69630598 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f80009450 con 0x7f8f8814e030 2023-10-30T13:55:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f881f29d0 con 0x7f8f8814e030 2023-10-30T13:55:48.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f881562e0 con 0x7f8f8814e030 2023-10-30T13:55:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f84ff9640 1 -- 172.21.15.31:0/69630598 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f80004030 con 0x7f8f8814e030 2023-10-30T13:55:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f84ff9640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8f740664a0 0x7f8f74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f84ff9640 1 -- 172.21.15.31:0/69630598 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f8f8011ee20 con 0x7f8f8814e030 2023-10-30T13:55:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.239+0000 7f8f87fff640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8f740664a0 0x7f8f74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.243+0000 7f8f87fff640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8f740664a0 0x7f8f74068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8f78000c10 tx=0x7f8f78009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:48.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.243+0000 7f8f84ff9640 1 -- 172.21.15.31:0/69630598 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f800ebfa0 con 0x7f8f8814e030 2023-10-30T13:55:48.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f8804e170 con 0x7f8f8814e030 2023-10-30T13:55:48.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f84ff9640 1 -- 172.21.15.31:0/69630598 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f8f800bf020 con 0x7f8f8814e030 2023-10-30T13:55:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:55:48.438 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:48.438 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:55:48.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8f740664a0 msgr2=0x7f8f74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8f740664a0 0x7f8f74068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8f78000c10 tx=0x7f8f78009040 comp rx=0 tx=0).stop 2023-10-30T13:55:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8814e030 msgr2=0x7f8f88156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8814e030 0x7f8f88156f40 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8f8001ccb0 tx=0x7f8f800194e0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 shutdown_connections 2023-10-30T13:55:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8f740664a0 0x7f8f74068960 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8814e030 0x7f8f88156f40 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8f8806fdf0 0x7f8f88152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 --2- 172.21.15.31:0/69630598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8806c0a0 0x7f8f880af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 >> 172.21.15.31:0/69630598 conn(0x7f8f88069e90 msgr2=0x7f8f8806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 shutdown_connections 2023-10-30T13:55:48.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:48.435+0000 7f8f8e8ea640 1 -- 172.21.15.31:0/69630598 wait complete. 2023-10-30T13:55:48.460 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:55:50.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:50.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.611+0000 7fec2751d640 1 Processor -- start 2023-10-30T13:55:50.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.611+0000 7fec2751d640 1 -- start start 2023-10-30T13:55:50.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec2751d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006bb40 0x7fec20157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:50.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec2751d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fec2006c420 0x7fec20157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:50.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec2751d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec20071e00 con 0x7fec2006b760 2023-10-30T13:55:50.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec2751d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec20070020 con 0x7fec2006c420 2023-10-30T13:55:50.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec2751d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec20070f10 con 0x7fec2006bb40 2023-10-30T13:55:50.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec25aba640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fec2006c420 0x7fec20157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:50.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec24ab8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006bb40 0x7fec20157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:50.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec25aba640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fec2006c420 0x7fec20157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:48806/0 (socket says 172.21.15.31:48806) 2023-10-30T13:55:50.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec25aba640 1 -- 172.21.15.31:0/747104635 learned_addr learned my addr 172.21.15.31:0/747104635 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:50.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec252b9640 1 --1- 172.21.15.31:0/747104635 >> v1:172.21.15.33:6789/0 conn(0x7fec2006b760 0x7fec2015e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:41316/0 (socket says 172.21.15.31:41316) 2023-10-30T13:55:50.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec25aba640 1 -- 172.21.15.31:0/747104635 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006bb40 msgr2=0x7fec20157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:50.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec25aba640 1 --2- 172.21.15.31:0/747104635 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006bb40 0x7fec20157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec25aba640 1 -- 172.21.15.31:0/747104635 >> v1:172.21.15.33:6789/0 conn(0x7fec2006b760 legacy=0x7fec2015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.615+0000 7fec25aba640 1 -- 172.21.15.31:0/747104635 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec200711f0 con 0x7fec2006c420 2023-10-30T13:55:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec25aba640 1 --2- 172.21.15.31:0/747104635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006c420 0x7fec20157f60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fec18009e50 tx=0x7fec1801c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d93081d49a9195 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec17fff640 1 -- 172.21.15.31:0/747104635 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec180a58a0 con 0x7fec2006c420 2023-10-30T13:55:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec17fff640 1 -- 172.21.15.31:0/747104635 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec180a5a00 con 0x7fec2006c420 2023-10-30T13:55:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec17fff640 1 -- 172.21.15.31:0/747104635 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec180a5d90 con 0x7fec2006c420 2023-10-30T13:55:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 -- 172.21.15.31:0/747104635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006c420 msgr2=0x7fec20157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 --2- 172.21.15.31:0/747104635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006c420 0x7fec20157f60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fec18009e50 tx=0x7fec1801c9e0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 -- 172.21.15.31:0/747104635 shutdown_connections 2023-10-30T13:55:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 --2- 172.21.15.31:0/747104635 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006bb40 0x7fec20157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 --2- 172.21.15.31:0/747104635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006c420 0x7fec20157f60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 -- 172.21.15.31:0/747104635 >> 172.21.15.31:0/747104635 conn(0x7fec200cac70 msgr2=0x7fec200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 -- 172.21.15.31:0/747104635 shutdown_connections 2023-10-30T13:55:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 -- 172.21.15.31:0/747104635 wait complete. 2023-10-30T13:55:50.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 Processor -- start 2023-10-30T13:55:50.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.619+0000 7fec2751d640 1 -- start start 2023-10-30T13:55:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2006b760 0x7fec20160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006bb40 0x7fec20160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006c420 0x7fec201f9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec201fa120 con 0x7fec2006b760 2023-10-30T13:55:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015f5f0 con 0x7fec2006bb40 2023-10-30T13:55:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015fb70 con 0x7fec2006c420 2023-10-30T13:55:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2006b760 0x7fec20160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2006b760 0x7fec20160000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47306/0 (socket says 172.21.15.31:47306) 2023-10-30T13:55:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 -- 172.21.15.31:0/1962821090 learned_addr learned my addr 172.21.15.31:0/1962821090 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec25aba640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006c420 0x7fec201f9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec24ab8640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006bb40 0x7fec20160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 -- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006c420 msgr2=0x7fec201f9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006c420 0x7fec201f9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 -- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006bb40 msgr2=0x7fec20160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006bb40 0x7fec20160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 -- 172.21.15.31:0/1962821090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec18009b30 con 0x7fec2006b760 2023-10-30T13:55:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec252b9640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2006b760 0x7fec20160000 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c4c0 tx=0x7fec1c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec15ffb640 1 -- 172.21.15.31:0/1962821090 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c007480 con 0x7fec2006b760 2023-10-30T13:55:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec15ffb640 1 -- 172.21.15.31:0/1962821090 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c0075e0 con 0x7fec2006b760 2023-10-30T13:55:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec15ffb640 1 -- 172.21.15.31:0/1962821090 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c004a10 con 0x7fec2006b760 2023-10-30T13:55:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec201effa0 con 0x7fec2006b760 2023-10-30T13:55:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2015f800 con 0x7fec2006b760 2023-10-30T13:55:50.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2004e170 con 0x7fec2006b760 2023-10-30T13:55:50.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.623+0000 7fec15ffb640 1 -- 172.21.15.31:0/1962821090 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec1c004b70 con 0x7fec2006b760 2023-10-30T13:55:50.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.627+0000 7fec15ffb640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fec040664a0 0x7fec04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:50.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.627+0000 7fec15ffb640 1 -- 172.21.15.31:0/1962821090 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fec1c0833f0 con 0x7fec2006b760 2023-10-30T13:55:50.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.627+0000 7fec24ab8640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fec040664a0 0x7fec04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:50.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.627+0000 7fec24ab8640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fec040664a0 0x7fec04068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fec201f3860 tx=0x7fec10008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:50.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.631+0000 7fec15ffb640 1 -- 172.21.15.31:0/1962821090 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec1c0504c0 con 0x7fec2006b760 2023-10-30T13:55:50.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec2015be10 con 0x7fec2006b760 2023-10-30T13:55:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec15ffb640 1 -- 172.21.15.31:0/1962821090 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fec1c053ae0 con 0x7fec2006b760 2023-10-30T13:55:50.822 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:55:50.823 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:55:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fec040664a0 msgr2=0x7fec04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fec040664a0 0x7fec04068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fec201f3860 tx=0x7fec10008040 comp rx=0 tx=0).stop 2023-10-30T13:55:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2006b760 msgr2=0x7fec20160000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2006b760 0x7fec20160000 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c4c0 tx=0x7fec1c00c980 comp rx=0 tx=0).stop 2023-10-30T13:55:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 shutdown_connections 2023-10-30T13:55:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fec040664a0 0x7fec04068960 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fec2006c420 0x7fec201f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec2006bb40 0x7fec20160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 --2- 172.21.15.31:0/1962821090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2006b760 0x7fec20160000 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 >> 172.21.15.31:0/1962821090 conn(0x7fec200cac70 msgr2=0x7fec2015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 shutdown_connections 2023-10-30T13:55:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:50.819+0000 7fec2751d640 1 -- 172.21.15.31:0/1962821090 wait complete. 2023-10-30T13:55:50.843 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:55:52.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:52.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.995+0000 7f4e0a9d3640 1 Processor -- start 2023-10-30T13:55:52.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.995+0000 7f4e0a9d3640 1 -- start start 2023-10-30T13:55:53.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e0a9d3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4e040c4a10 0x7f4e0415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e0a9d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c42a0 0x7f4e040ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e0a9d3640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e04159f20 con 0x7f4e04153c80 2023-10-30T13:55:53.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e0a9d3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0415dec0 con 0x7f4e040c4a10 2023-10-30T13:55:53.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e0a9d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e041599a0 con 0x7f4e040c42a0 2023-10-30T13:55:53.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e037fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4e040c4a10 0x7f4e0415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e03fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f4e04153c80 0x7f4e0415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:52314/0 (socket says 172.21.15.31:52314) 2023-10-30T13:55:53.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e03fff640 1 -- 172.21.15.31:0/2923815858 learned_addr learned my addr 172.21.15.31:0/2923815858 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:55:53.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e08f70640 1 --2- 172.21.15.31:0/2923815858 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c42a0 0x7f4e040ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e037fe640 1 -- 172.21.15.31:0/2923815858 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c42a0 msgr2=0x7f4e040ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:53.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e037fe640 1 --2- 172.21.15.31:0/2923815858 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c42a0 0x7f4e040ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e037fe640 1 -- 172.21.15.31:0/2923815858 >> v1:172.21.15.31:6789/0 conn(0x7f4e04153c80 legacy=0x7f4e0415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e037fe640 1 -- 172.21.15.31:0/2923815858 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0415d620 con 0x7f4e040c4a10 2023-10-30T13:55:53.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:52.999+0000 7f4e02ffd640 1 -- 172.21.15.31:0/2923815858 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481791670 0 0) 0x7f4e04159f20 con 0x7f4e04153c80 2023-10-30T13:55:53.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e037fe640 1 --2- 172.21.15.31:0/2923815858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e040c4a10 0x7f4e0415cf90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf7d224062eee96 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:53.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e02ffd640 1 -- 172.21.15.31:0/2923815858 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df80a8970 con 0x7f4e040c4a10 2023-10-30T13:55:53.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e02ffd640 1 -- 172.21.15.31:0/2923815858 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df80a8ad0 con 0x7f4e040c4a10 2023-10-30T13:55:53.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e02ffd640 1 -- 172.21.15.31:0/2923815858 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df80a8e60 con 0x7f4e040c4a10 2023-10-30T13:55:53.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/2923815858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e040c4a10 msgr2=0x7f4e0415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/2923815858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e040c4a10 0x7f4e0415cf90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df801ec70 comp rx=0 tx=0).stop 2023-10-30T13:55:53.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/2923815858 shutdown_connections 2023-10-30T13:55:53.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/2923815858 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c42a0 0x7f4e040ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/2923815858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e040c4a10 0x7f4e0415cf90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/2923815858 >> 172.21.15.31:0/2923815858 conn(0x7f4e040715c0 msgr2=0x7f4e040719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:53.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/2923815858 shutdown_connections 2023-10-30T13:55:53.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/2923815858 wait complete. 2023-10-30T13:55:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 Processor -- start 2023-10-30T13:55:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- start start 2023-10-30T13:55:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e040c42a0 0x7f4e0414a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c4a10 0x7f4e040ce910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e04153c80 0x7f4e0415c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0414b4a0 con 0x7f4e04153c80 2023-10-30T13:55:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0414b780 con 0x7f4e040c42a0 2023-10-30T13:55:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.003+0000 7f4e0a9d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0414ba60 con 0x7f4e040c4a10 2023-10-30T13:55:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e04153c80 0x7f4e0415c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e04153c80 0x7f4e0415c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47330/0 (socket says 172.21.15.31:47330) 2023-10-30T13:55:53.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 -- 172.21.15.31:0/812167460 learned_addr learned my addr 172.21.15.31:0/812167460 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:53.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e03fff640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e040c42a0 0x7f4e0414a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e037fe640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c4a10 0x7f4e040ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 -- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c4a10 msgr2=0x7f4e040ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c4a10 0x7f4e040ce910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 -- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e040c42a0 msgr2=0x7f4e0414a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e040c42a0 0x7f4e0414a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 -- 172.21.15.31:0/812167460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df8009420 con 0x7f4e04153c80 2023-10-30T13:55:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e08f70640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e04153c80 0x7f4e0415c660 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc007e40 tx=0x7f4dfc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e00ff9640 1 -- 172.21.15.31:0/812167460 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc014620 con 0x7f4e04153c80 2023-10-30T13:55:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e00ff9640 1 -- 172.21.15.31:0/812167460 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc00ee90 con 0x7f4e04153c80 2023-10-30T13:55:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e00ff9640 1 -- 172.21.15.31:0/812167460 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0149e0 con 0x7f4e04153c80 2023-10-30T13:55:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0414bd40 con 0x7f4e04153c80 2023-10-30T13:55:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e040ce410 con 0x7f4e04153c80 2023-10-30T13:55:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e040ce5d0 con 0x7f4e04153c80 2023-10-30T13:55:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.007+0000 7f4e00ff9640 1 -- 172.21.15.31:0/812167460 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4dfc014b40 con 0x7f4e04153c80 2023-10-30T13:55:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.011+0000 7f4e00ff9640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4df00664a0 0x7f4df0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.011+0000 7f4e00ff9640 1 -- 172.21.15.31:0/812167460 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f4dfc084b60 con 0x7f4e04153c80 2023-10-30T13:55:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.011+0000 7f4e03fff640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4df00664a0 0x7f4df0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.011+0000 7f4e03fff640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4df00664a0 0x7f4df0068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4df4000f70 tx=0x7f4df4007df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.015+0000 7f4e00ff9640 1 -- 172.21.15.31:0/812167460 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4dfc051c20 con 0x7f4e04153c80 2023-10-30T13:55:53.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e0404e170 con 0x7f4e04153c80 2023-10-30T13:55:53.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e00ff9640 1 -- 172.21.15.31:0/812167460 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f4dfc055240 con 0x7f4e04153c80 2023-10-30T13:55:53.210 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:55:53.210 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:53.211 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:55:53.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4df00664a0 msgr2=0x7f4df0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4df00664a0 0x7f4df0068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4df4000f70 tx=0x7f4df4007df0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e04153c80 msgr2=0x7f4e0415c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e04153c80 0x7f4e0415c660 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc007e40 tx=0x7f4dfc00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 shutdown_connections 2023-10-30T13:55:53.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4df00664a0 0x7f4df0068960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e04153c80 0x7f4e0415c660 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e040c4a10 0x7f4e040ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 --2- 172.21.15.31:0/812167460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e040c42a0 0x7f4e0414a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 >> 172.21.15.31:0/812167460 conn(0x7f4e040715c0 msgr2=0x7f4e040cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.207+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 shutdown_connections 2023-10-30T13:55:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:53.211+0000 7f4e0a9d3640 1 -- 172.21.15.31:0/812167460 wait complete. 2023-10-30T13:55:53.232 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:55:55.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:55.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 Processor -- start 2023-10-30T13:55:55.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 -- start start 2023-10-30T13:55:55.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc894153c80 0x7fc89415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:55.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc8940c4a10 0x7fc8940c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:55.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc89415a3b0 0x7fc8940ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:55.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89415dec0 con 0x7fc8940c4a10 2023-10-30T13:55:55.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8941599a0 con 0x7fc894153c80 2023-10-30T13:55:55.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89bf9d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc894159f20 con 0x7fc89415a3b0 2023-10-30T13:55:55.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc8940c4a10 0x7fc8940c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:55.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc8940c4a10 0x7fc8940c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47348/0 (socket says 172.21.15.31:47348) 2023-10-30T13:55:55.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 -- 172.21.15.31:0/2609739521 learned_addr learned my addr 172.21.15.31:0/2609739521 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:55.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899d39640 1 --2- 172.21.15.31:0/2609739521 >> v2:172.21.15.33:3300/0 conn(0x7fc894153c80 0x7fc89415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:55.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc89a53a640 1 --2- 172.21.15.31:0/2609739521 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc89415a3b0 0x7fc8940ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:55.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 -- 172.21.15.31:0/2609739521 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc89415a3b0 msgr2=0x7fc8940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:55.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 --2- 172.21.15.31:0/2609739521 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc89415a3b0 0x7fc8940ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 -- 172.21.15.31:0/2609739521 >> v2:172.21.15.33:3300/0 conn(0x7fc894153c80 msgr2=0x7fc89415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 --2- 172.21.15.31:0/2609739521 >> v2:172.21.15.33:3300/0 conn(0x7fc894153c80 0x7fc89415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.387+0000 7fc899538640 1 -- 172.21.15.31:0/2609739521 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8940c47e0 con 0x7fc8940c4a10 2023-10-30T13:55:55.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc899538640 1 --2- 172.21.15.31:0/2609739521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8940c4a10 0x7fc8940c42a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc8940d00f0 tx=0x7fc88401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1338fea7b8c9e9ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:55.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc898d37640 1 -- 172.21.15.31:0/2609739521 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8840a89b0 con 0x7fc8940c4a10 2023-10-30T13:55:55.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc898d37640 1 -- 172.21.15.31:0/2609739521 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8840a8b10 con 0x7fc8940c4a10 2023-10-30T13:55:55.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc898d37640 1 -- 172.21.15.31:0/2609739521 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8840a8ea0 con 0x7fc8940c4a10 2023-10-30T13:55:55.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 -- 172.21.15.31:0/2609739521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8940c4a10 msgr2=0x7fc8940c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:55.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/2609739521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8940c4a10 0x7fc8940c42a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc8940d00f0 tx=0x7fc88401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 -- 172.21.15.31:0/2609739521 shutdown_connections 2023-10-30T13:55:55.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/2609739521 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc89415a3b0 0x7fc8940ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/2609739521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8940c4a10 0x7fc8940c42a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/2609739521 >> v2:172.21.15.33:3300/0 conn(0x7fc894153c80 0x7fc89415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 -- 172.21.15.31:0/2609739521 >> 172.21.15.31:0/2609739521 conn(0x7fc8940715c0 msgr2=0x7fc8940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:55.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 -- 172.21.15.31:0/2609739521 shutdown_connections 2023-10-30T13:55:55.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.391+0000 7fc89bf9d640 1 -- 172.21.15.31:0/2609739521 wait complete. 2023-10-30T13:55:55.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 Processor -- start 2023-10-30T13:55:55.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 -- start start 2023-10-30T13:55:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8940c4a10 0x7fc8941efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc894153c80 0x7fc8940c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc89415a3b0 0x7fc8940cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc894156d40 con 0x7fc89415a3b0 2023-10-30T13:55:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8941f00a0 con 0x7fc8940c4a10 2023-10-30T13:55:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8940ce660 con 0x7fc894153c80 2023-10-30T13:55:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc89415a3b0 0x7fc8940cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc89415a3b0 0x7fc8940cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47362/0 (socket says 172.21.15.31:47362) 2023-10-30T13:55:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 -- 172.21.15.31:0/1657694947 learned_addr learned my addr 172.21.15.31:0/1657694947 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc899538640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc894153c80 0x7fc8940c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc899d39640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8940c4a10 0x7fc8941efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 -- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc894153c80 msgr2=0x7fc8940c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc894153c80 0x7fc8940c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 -- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8940c4a10 msgr2=0x7fc8941efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8940c4a10 0x7fc8941efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 -- 172.21.15.31:0/1657694947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840094d0 con 0x7fc89415a3b0 2023-10-30T13:55:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89a53a640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc89415a3b0 0x7fc8940cd8e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc88c007e60 tx=0x7fc88c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc88a7fc640 1 -- 172.21.15.31:0/1657694947 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c014420 con 0x7fc89415a3b0 2023-10-30T13:55:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc88a7fc640 1 -- 172.21.15.31:0/1657694947 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c0145a0 con 0x7fc89415a3b0 2023-10-30T13:55:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc88a7fc640 1 -- 172.21.15.31:0/1657694947 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c014950 con 0x7fc89415a3b0 2023-10-30T13:55:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8940ce940 con 0x7fc89415a3b0 2023-10-30T13:55:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.395+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8940cef00 con 0x7fc89415a3b0 2023-10-30T13:55:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.399+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8940c62c0 con 0x7fc89415a3b0 2023-10-30T13:55:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.399+0000 7fc88a7fc640 1 -- 172.21.15.31:0/1657694947 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc88c014ab0 con 0x7fc89415a3b0 2023-10-30T13:55:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.399+0000 7fc88a7fc640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664d0 0x7fc878068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.399+0000 7fc88a7fc640 1 -- 172.21.15.31:0/1657694947 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fc88c0849a0 con 0x7fc89415a3b0 2023-10-30T13:55:55.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.399+0000 7fc899d39640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664d0 0x7fc878068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:55.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.399+0000 7fc899d39640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664d0 0x7fc878068990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc890009ae0 tx=0x7fc890009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:55.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.403+0000 7fc88a7fc640 1 -- 172.21.15.31:0/1657694947 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc88c051b30 con 0x7fc89415a3b0 2023-10-30T13:55:55.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.591+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc89404e170 con 0x7fc89415a3b0 2023-10-30T13:55:55.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.591+0000 7fc88a7fc640 1 -- 172.21.15.31:0/1657694947 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fc88c055150 con 0x7fc89415a3b0 2023-10-30T13:55:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:55:55.596 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:55.597 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:55:55.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664d0 msgr2=0x7fc878068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:55.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664d0 0x7fc878068990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc890009ae0 tx=0x7fc890009190 comp rx=0 tx=0).stop 2023-10-30T13:55:55.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc89415a3b0 msgr2=0x7fc8940cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc89415a3b0 0x7fc8940cd8e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc88c007e60 tx=0x7fc88c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 shutdown_connections 2023-10-30T13:55:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8780664d0 0x7fc878068990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc89415a3b0 0x7fc8940cd8e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc894153c80 0x7fc8940c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 --2- 172.21.15.31:0/1657694947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8940c4a10 0x7fc8941efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 >> 172.21.15.31:0/1657694947 conn(0x7fc8940715c0 msgr2=0x7fc8940c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 shutdown_connections 2023-10-30T13:55:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:55.595+0000 7fc89bf9d640 1 -- 172.21.15.31:0/1657694947 wait complete. 2023-10-30T13:55:55.617 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:55:57.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde73091640 1 Processor -- start 2023-10-30T13:55:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde73091640 1 -- start start 2023-10-30T13:55:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde73091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 0x7fde6c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde73091640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c159f20 con 0x7fde6c0c4a10 2023-10-30T13:55:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde73091640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c15d620 con 0x7fde6c0c42a0 2023-10-30T13:55:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde73091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c15dec0 con 0x7fde6c153c80 2023-10-30T13:55:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fde6c0c4a10 0x7fde6c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48690/0 (socket says 172.21.15.31:48690) 2023-10-30T13:55:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6bfff640 1 -- 172.21.15.31:0/3491407219 learned_addr learned my addr 172.21.15.31:0/3491407219 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:55:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde70e2d640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 0x7fde6c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174065624 0 0) 0x7fde6c159f20 con 0x7fde6c0c4a10 2023-10-30T13:55:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde5c003530 con 0x7fde6c0c4a10 2023-10-30T13:55:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755652371 0 0) 0x7fde5c003530 con 0x7fde6c0c4a10 2023-10-30T13:55:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 msgr2=0x7fde6c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 0x7fde6c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 >> v1:172.21.15.33:6789/0 conn(0x7fde6c0c42a0 legacy=0x7fde6c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c15d900 con 0x7fde6c0c4a10 2023-10-30T13:55:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fde58001680 con 0x7fde6c0c4a10 2023-10-30T13:55:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 >> v1:172.21.15.31:6789/0 conn(0x7fde6c0c4a10 legacy=0x7fde6c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c0039b0 0x7fde5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde5c004360 0x7fde5c004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.771+0000 7fde6b7fe640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 0x7fde5c005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c15dec0 con 0x7fde5c0039b0 2023-10-30T13:55:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c00b990 con 0x7fde5c004360 2023-10-30T13:55:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c00bb50 con 0x7fde6c153c80 2023-10-30T13:55:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde70e2d640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c0039b0 0x7fde5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde6bfff640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde5c004360 0x7fde5c004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde7162e640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 0x7fde5c005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde70e2d640 1 -- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 msgr2=0x7fde5c005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde70e2d640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde6c153c80 0x7fde5c005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde70e2d640 1 -- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde5c004360 msgr2=0x7fde5c004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde70e2d640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde5c004360 0x7fde5c004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde70e2d640 1 -- 172.21.15.31:0/3491407219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c003530 con 0x7fde5c0039b0 2023-10-30T13:55:57.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde6bfff640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde5c004360 0x7fde5c004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.775+0000 7fde70e2d640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c0039b0 0x7fde5c003d90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fde60009ae0 tx=0x7fde6001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40db522cfa73cf7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde600a5640 con 0x7fde5c0039b0 2023-10-30T13:55:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde600a57a0 con 0x7fde5c0039b0 2023-10-30T13:55:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde6b7fe640 1 -- 172.21.15.31:0/3491407219 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde600a5b50 con 0x7fde5c0039b0 2023-10-30T13:55:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- 172.21.15.31:0/3491407219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c0039b0 msgr2=0x7fde5c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 --2- 172.21.15.31:0/3491407219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c0039b0 0x7fde5c003d90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fde60009ae0 tx=0x7fde6001ca50 comp rx=0 tx=0).stop 2023-10-30T13:55:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde7162e640 1 -- 172.21.15.31:0/3491407219 reap_dead start 2023-10-30T13:55:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- 172.21.15.31:0/3491407219 shutdown_connections 2023-10-30T13:55:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- 172.21.15.31:0/3491407219 >> 172.21.15.31:0/3491407219 conn(0x7fde6c0715c0 msgr2=0x7fde6c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- 172.21.15.31:0/3491407219 shutdown_connections 2023-10-30T13:55:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- 172.21.15.31:0/3491407219 wait complete. 2023-10-30T13:55:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 Processor -- start 2023-10-30T13:55:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- start start 2023-10-30T13:55:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde5c0039b0 0x7fde6c0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde6c147590 0x7fde6c159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde6c15cdf0 0x7fde6c15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c148ee0 con 0x7fde6c147590 2023-10-30T13:55:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c1482f0 con 0x7fde6c15cdf0 2023-10-30T13:55:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde73091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c15c970 con 0x7fde5c0039b0 2023-10-30T13:55:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.779+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde6c147590 0x7fde6c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde70e2d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde5c0039b0 0x7fde6c0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde6c147590 0x7fde6c159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47386/0 (socket says 172.21.15.31:47386) 2023-10-30T13:55:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 -- 172.21.15.31:0/2920027951 learned_addr learned my addr 172.21.15.31:0/2920027951 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:55:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde7162e640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde6c15cdf0 0x7fde6c15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 -- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde5c0039b0 msgr2=0x7fde6c0b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde5c0039b0 0x7fde6c0b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 -- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde6c15cdf0 msgr2=0x7fde6c15ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde6c15cdf0 0x7fde6c15ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 -- 172.21.15.31:0/2920027951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c1485b0 con 0x7fde6c147590 2023-10-30T13:55:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde6bfff640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde6c147590 0x7fde6c159060 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fde5800e880 tx=0x7fde5800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde697fa640 1 -- 172.21.15.31:0/2920027951 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5800c8d0 con 0x7fde6c147590 2023-10-30T13:55:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde697fa640 1 -- 172.21.15.31:0/2920027951 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5800ca30 con 0x7fde6c147590 2023-10-30T13:55:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde697fa640 1 -- 172.21.15.31:0/2920027951 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5800cdc0 con 0x7fde6c147590 2023-10-30T13:55:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c148890 con 0x7fde6c147590 2023-10-30T13:55:57.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c0ce1c0 con 0x7fde6c147590 2023-10-30T13:55:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c149050 con 0x7fde6c147590 2023-10-30T13:55:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde697fa640 1 -- 172.21.15.31:0/2920027951 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde5801d460 con 0x7fde6c147590 2023-10-30T13:55:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.783+0000 7fde697fa640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fde500664d0 0x7fde50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.787+0000 7fde697fa640 1 -- 172.21.15.31:0/2920027951 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fde58084240 con 0x7fde6c147590 2023-10-30T13:55:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.787+0000 7fde70e2d640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fde500664d0 0x7fde50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.787+0000 7fde70e2d640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fde500664d0 0x7fde50068990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fde60009ae0 tx=0x7fde600046d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.791+0000 7fde697fa640 1 -- 172.21.15.31:0/2920027951 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde580513c0 con 0x7fde6c147590 2023-10-30T13:55:57.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.979+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde6c04e170 con 0x7fde6c147590 2023-10-30T13:55:57.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.979+0000 7fde697fa640 1 -- 172.21.15.31:0/2920027951 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fde58059e60 con 0x7fde6c147590 2023-10-30T13:55:57.983 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:55:57.983 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:57.984 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:55:57.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fde500664d0 msgr2=0x7fde50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:57.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fde500664d0 0x7fde50068990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fde60009ae0 tx=0x7fde600046d0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde6c147590 msgr2=0x7fde6c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:57.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde6c147590 0x7fde6c159060 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fde5800e880 tx=0x7fde5800ed40 comp rx=0 tx=0).stop 2023-10-30T13:55:57.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 shutdown_connections 2023-10-30T13:55:57.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fde500664d0 0x7fde50068990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde6c15cdf0 0x7fde6c15ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde6c147590 0x7fde6c159060 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 --2- 172.21.15.31:0/2920027951 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fde5c0039b0 0x7fde6c0b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:57.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 >> 172.21.15.31:0/2920027951 conn(0x7fde6c0715c0 msgr2=0x7fde6c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:57.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 shutdown_connections 2023-10-30T13:55:57.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:55:57.983+0000 7fde73091640 1 -- 172.21.15.31:0/2920027951 wait complete. 2023-10-30T13:55:58.005 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:55:59.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:59.051 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:00.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:00.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.155+0000 7f42afb65640 1 Processor -- start 2023-10-30T13:56:00.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42afb65640 1 -- start start 2023-10-30T13:56:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42afb65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 0x7f42a80c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42afb65640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a815cda0 con 0x7f42a80c5ba0 2023-10-30T13:56:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42afb65640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a806f490 con 0x7f42a80c9b40 2023-10-30T13:56:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42afb65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a815d990 con 0x7f42a80c5f80 2023-10-30T13:56:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad901640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f42a80c5ba0 0x7f42a8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48696/0 (socket says 172.21.15.31:48696) 2023-10-30T13:56:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad901640 1 -- 172.21.15.31:0/490252431 learned_addr learned my addr 172.21.15.31:0/490252431 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:00.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad100640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 0x7f42a80c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427696680 0 0) 0x7f42a815cda0 con 0x7f42a80c5ba0 2023-10-30T13:56:00.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4294003530 con 0x7f42a80c5ba0 2023-10-30T13:56:00.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903788774 0 0) 0x7f4294003530 con 0x7f42a80c5ba0 2023-10-30T13:56:00.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 msgr2=0x7f42a80c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:00.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 0x7f42a80c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 >> v1:172.21.15.33:6789/0 conn(0x7f42a80c9b40 legacy=0x7f42a81597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a815cff0 con 0x7f42a80c5ba0 2023-10-30T13:56:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f42a4001680 con 0x7f42a80c5ba0 2023-10-30T13:56:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 >> v1:172.21.15.31:6789/0 conn(0x7f42a80c5ba0 legacy=0x7f42a8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 0x7f42940039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4294003ef0 0x7f42940047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294004d10 0x7f4294005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a815d990 con 0x7f4294003ef0 2023-10-30T13:56:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429400baa0 con 0x7f4294004d10 2023-10-30T13:56:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429400bc60 con 0x7f42a80c5f80 2023-10-30T13:56:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad100640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4294003ef0 0x7f42940047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad901640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 0x7f42940039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ae102640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294004d10 0x7f4294005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad100640 1 -- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 msgr2=0x7f42940039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad100640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80c5f80 0x7f42940039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad100640 1 -- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294004d10 msgr2=0x7f4294005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad100640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294004d10 0x7f4294005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ad100640 1 -- 172.21.15.31:0/490252431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294003530 con 0x7f4294003ef0 2023-10-30T13:56:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.159+0000 7f42ae102640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294004d10 0x7f4294005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:00.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42ad100640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4294003ef0 0x7f42940047d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4298009ae0 tx=0x7f429801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750071ac0c4d9ffa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:00.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42980a6430 con 0x7f4294003ef0 2023-10-30T13:56:00.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42980a6590 con 0x7f4294003ef0 2023-10-30T13:56:00.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42ac8ff640 1 -- 172.21.15.31:0/490252431 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42980a6940 con 0x7f4294003ef0 2023-10-30T13:56:00.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42afb65640 1 -- 172.21.15.31:0/490252431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4294003ef0 msgr2=0x7f42940047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42afb65640 1 --2- 172.21.15.31:0/490252431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4294003ef0 0x7f42940047d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4298009ae0 tx=0x7f429801ca50 comp rx=0 tx=0).stop 2023-10-30T13:56:00.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42ae102640 1 -- 172.21.15.31:0/490252431 reap_dead start 2023-10-30T13:56:00.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42afb65640 1 -- 172.21.15.31:0/490252431 shutdown_connections 2023-10-30T13:56:00.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42afb65640 1 -- 172.21.15.31:0/490252431 >> 172.21.15.31:0/490252431 conn(0x7f42a806afa0 msgr2=0x7f42a806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:00.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42afb65640 1 -- 172.21.15.31:0/490252431 shutdown_connections 2023-10-30T13:56:00.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.163+0000 7f42afb65640 1 -- 172.21.15.31:0/490252431 wait complete. 2023-10-30T13:56:00.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 Processor -- start 2023-10-30T13:56:00.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 -- start start 2023-10-30T13:56:00.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294003ef0 0x7f42a81f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80ad8b0 0x7f42a80c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42a80ae6c0 0x7f42a80cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80c2c80 con 0x7f42a80ae6c0 2023-10-30T13:56:00.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80addd0 con 0x7f4294003ef0 2023-10-30T13:56:00.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80c1210 con 0x7f42a80ad8b0 2023-10-30T13:56:00.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42a80ae6c0 0x7f42a80cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42a80ae6c0 0x7f42a80cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47400/0 (socket says 172.21.15.31:47400) 2023-10-30T13:56:00.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 -- 172.21.15.31:0/2801664487 learned_addr learned my addr 172.21.15.31:0/2801664487 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:00.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ad901640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294003ef0 0x7f42a81f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ad100640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80ad8b0 0x7f42a80c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 -- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80ad8b0 msgr2=0x7f42a80c16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80ad8b0 0x7f42a80c16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 -- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294003ef0 msgr2=0x7f42a81f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294003ef0 0x7f42a81f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 -- 172.21.15.31:0/2801664487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a80c23e0 con 0x7f42a80ae6c0 2023-10-30T13:56:00.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ad100640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80ad8b0 0x7f42a80c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:00.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42ae102640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42a80ae6c0 0x7f42a80cbf50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f42a806d630 tx=0x7f42a000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:00.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f429e7fc640 1 -- 172.21.15.31:0/2801664487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0017070 con 0x7f42a80ae6c0 2023-10-30T13:56:00.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f429e7fc640 1 -- 172.21.15.31:0/2801664487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0006b60 con 0x7f42a80ae6c0 2023-10-30T13:56:00.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a80c26c0 con 0x7f42a80ae6c0 2023-10-30T13:56:00.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f429e7fc640 1 -- 172.21.15.31:0/2801664487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0006cf0 con 0x7f42a80ae6c0 2023-10-30T13:56:00.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.167+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a80c0ea0 con 0x7f42a80ae6c0 2023-10-30T13:56:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.171+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a80c1060 con 0x7f42a80ae6c0 2023-10-30T13:56:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.171+0000 7f429e7fc640 1 -- 172.21.15.31:0/2801664487 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42a0003980 con 0x7f42a80ae6c0 2023-10-30T13:56:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.171+0000 7f429e7fc640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f428c0664a0 0x7f428c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.171+0000 7f429e7fc640 1 -- 172.21.15.31:0/2801664487 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f42a0083930 con 0x7f42a80ae6c0 2023-10-30T13:56:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.171+0000 7f42ad901640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f428c0664a0 0x7f428c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.171+0000 7f42ad901640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f428c0664a0 0x7f428c068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f42a4007c60 tx=0x7f42a4000f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.175+0000 7f429e7fc640 1 -- 172.21.15.31:0/2801664487 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42a0050a00 con 0x7f42a80ae6c0 2023-10-30T13:56:00.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.367+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42a80c0d00 con 0x7f42a80ae6c0 2023-10-30T13:56:00.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.367+0000 7f429e7fc640 1 -- 172.21.15.31:0/2801664487 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f42a80c0d00 con 0x7f42a80ae6c0 2023-10-30T13:56:00.372 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:00.372 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5813,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:54:14.228778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6201},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6201":{"gid":6201,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":37,"addr":"172.21.15.31:6837/2267972067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2267972067},{"type":"v1","addr":"172.21.15.31:6837","nonce":2267972067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2023-10-30T13:56:00.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f428c0664a0 msgr2=0x7f428c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f428c0664a0 0x7f428c068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f42a4007c60 tx=0x7f42a4000f90 comp rx=0 tx=0).stop 2023-10-30T13:56:00.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42a80ae6c0 msgr2=0x7f42a80cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42a80ae6c0 0x7f42a80cbf50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f42a806d630 tx=0x7f42a000eb90 comp rx=0 tx=0).stop 2023-10-30T13:56:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 shutdown_connections 2023-10-30T13:56:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f428c0664a0 0x7f428c068960 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42a80ae6c0 0x7f42a80cbf50 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f42a80ad8b0 0x7f42a80c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 --2- 172.21.15.31:0/2801664487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4294003ef0 0x7f42a81f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 >> 172.21.15.31:0/2801664487 conn(0x7f42a806afa0 msgr2=0x7f42a80ca1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:00.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 shutdown_connections 2023-10-30T13:56:00.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:00.371+0000 7f42afb65640 1 -- 172.21.15.31:0/2801664487 wait complete. 2023-10-30T13:56:00.393 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:56:02.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70f59e640 1 Processor -- start 2023-10-30T13:56:02.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70f59e640 1 -- start start 2023-10-30T13:56:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70f59e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 0x7fb7101597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70f59e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71015a770 con 0x7fb7100cb180 2023-10-30T13:56:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70f59e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7100c9d40 con 0x7fb7100c62b0 2023-10-30T13:56:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70f59e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb710159cf0 con 0x7fb710153b50 2023-10-30T13:56:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70ed9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb7100cb180 0x7fb71006ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48712/0 (socket says 172.21.15.31:48712) 2023-10-30T13:56:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70ed9d640 1 -- 172.21.15.31:0/3524428308 learned_addr learned my addr 172.21.15.31:0/3524428308 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70e59c640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 0x7fb7101597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019272110 0 0) 0x7fb7080014a0 con 0x7fb7100cb180 2023-10-30T13:56:02.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6fc003530 con 0x7fb7100cb180 2023-10-30T13:56:02.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706348979 0 0) 0x7fb7100c9d40 con 0x7fb7100c62b0 2023-10-30T13:56:02.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7080014a0 con 0x7fb7100c62b0 2023-10-30T13:56:02.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4240616565 0 0) 0x7fb6fc003530 con 0x7fb7100cb180 2023-10-30T13:56:02.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 msgr2=0x7fb7101597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:02.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 0x7fb7101597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 >> v1:172.21.15.33:6789/0 conn(0x7fb7100c62b0 legacy=0x7fb7100ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb710153f30 con 0x7fb7100cb180 2023-10-30T13:56:02.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fb708001b90 con 0x7fb7100cb180 2023-10-30T13:56:02.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 >> v1:172.21.15.31:6789/0 conn(0x7fb7100cb180 legacy=0x7fb71006ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 0x7fb6fc003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6fc0040f0 0x7fb6fc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc004d10 0x7fb6fc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb710159cf0 con 0x7fb6fc0040f0 2023-10-30T13:56:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc00bad0 con 0x7fb6fc004d10 2023-10-30T13:56:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc00bc90 con 0x7fb710153b50 2023-10-30T13:56:02.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb707fff640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6fc0040f0 0x7fb6fc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70ed9d640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc004d10 0x7fb6fc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb70e59c640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 0x7fb6fc003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb707fff640 1 -- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 msgr2=0x7fb6fc003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:02.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb707fff640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb710153b50 0x7fb6fc003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb707fff640 1 -- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc004d10 msgr2=0x7fb6fc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:02.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb707fff640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc004d10 0x7fb6fc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.547+0000 7fb707fff640 1 -- 172.21.15.31:0/3524428308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc003530 con 0x7fb6fc0040f0 2023-10-30T13:56:02.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb707fff640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6fc0040f0 0x7fb6fc0047d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb7100c66d0 tx=0x7fb6f801ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d4c97ef0a3a77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:02.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f80052d0 con 0x7fb6fc0040f0 2023-10-30T13:56:02.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f8005450 con 0x7fb6fc0040f0 2023-10-30T13:56:02.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70dd9b640 1 -- 172.21.15.31:0/3524428308 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f80a8850 con 0x7fb6fc0040f0 2023-10-30T13:56:02.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70f59e640 1 -- 172.21.15.31:0/3524428308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6fc0040f0 msgr2=0x7fb6fc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70f59e640 1 --2- 172.21.15.31:0/3524428308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6fc0040f0 0x7fb6fc0047d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb7100c66d0 tx=0x7fb6f801ebf0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70ed9d640 1 -- 172.21.15.31:0/3524428308 reap_dead start 2023-10-30T13:56:02.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70f59e640 1 -- 172.21.15.31:0/3524428308 shutdown_connections 2023-10-30T13:56:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70f59e640 1 -- 172.21.15.31:0/3524428308 >> 172.21.15.31:0/3524428308 conn(0x7fb710065a80 msgr2=0x7fb710065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70f59e640 1 -- 172.21.15.31:0/3524428308 shutdown_connections 2023-10-30T13:56:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.551+0000 7fb70f59e640 1 -- 172.21.15.31:0/3524428308 wait complete. 2023-10-30T13:56:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 Processor -- start 2023-10-30T13:56:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 -- start start 2023-10-30T13:56:02.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc0040f0 0x7fb7101566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7100c2b40 0x7fb7100cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb7100c5590 0x7fb7101f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7100c42a0 con 0x7fb7100c2b40 2023-10-30T13:56:02.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7100c5100 con 0x7fb6fc0040f0 2023-10-30T13:56:02.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7100c44f0 con 0x7fb7100c5590 2023-10-30T13:56:02.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7100c2b40 0x7fb7100cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7100c2b40 0x7fb7100cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47416/0 (socket says 172.21.15.31:47416) 2023-10-30T13:56:02.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 -- 172.21.15.31:0/423514954 learned_addr learned my addr 172.21.15.31:0/423514954 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:02.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70e59c640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc0040f0 0x7fb7101566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70ed9d640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb7100c5590 0x7fb7101f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 -- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb7100c5590 msgr2=0x7fb7101f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb7100c5590 0x7fb7101f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 -- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc0040f0 msgr2=0x7fb7101566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc0040f0 0x7fb7101566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 -- 172.21.15.31:0/423514954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7100c47d0 con 0x7fb7100c2b40 2023-10-30T13:56:02.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70e59c640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc0040f0 0x7fb7101566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:02.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb707fff640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7100c2b40 0x7fb7100cdc20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8005040 tx=0x7fb6fc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:02.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb706ffd640 1 -- 172.21.15.31:0/423514954 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f80a8d60 con 0x7fb7100c2b40 2023-10-30T13:56:02.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb706ffd640 1 -- 172.21.15.31:0/423514954 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f80050d0 con 0x7fb7100c2b40 2023-10-30T13:56:02.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb706ffd640 1 -- 172.21.15.31:0/423514954 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8003600 con 0x7fb7100c2b40 2023-10-30T13:56:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7100c4ab0 con 0x7fb7100c2b40 2023-10-30T13:56:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.555+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb710156c10 con 0x7fb7100c2b40 2023-10-30T13:56:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.559+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb710156c10 con 0x7fb7100c2b40 2023-10-30T13:56:02.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.559+0000 7fb706ffd640 1 -- 172.21.15.31:0/423514954 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb6f8003760 con 0x7fb7100c2b40 2023-10-30T13:56:02.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.559+0000 7fb706ffd640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6f00664a0 0x7fb6f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.559+0000 7fb706ffd640 1 -- 172.21.15.31:0/423514954 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb6f81216e0 con 0x7fb7100c2b40 2023-10-30T13:56:02.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.559+0000 7fb70e59c640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6f00664a0 0x7fb6f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.559+0000 7fb70e59c640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6f00664a0 0x7fb6f0068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb700009b00 tx=0x7fb700008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:02.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.563+0000 7fb706ffd640 1 -- 172.21.15.31:0/423514954 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6f80ee770 con 0x7fb7100c2b40 2023-10-30T13:56:02.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.751+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb710156c10 con 0x7fb7100c2b40 2023-10-30T13:56:02.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb706ffd640 1 -- 172.21.15.31:0/423514954 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fb6f80f1d90 con 0x7fb7100c2b40 2023-10-30T13:56:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:02.757 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:02.130169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:02.758 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2023-10-30T13:56:02.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6f00664a0 msgr2=0x7fb6f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6f00664a0 0x7fb6f0068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb700009b00 tx=0x7fb700008040 comp rx=0 tx=0).stop 2023-10-30T13:56:02.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7100c2b40 msgr2=0x7fb7100cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7100c2b40 0x7fb7100cdc20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8005040 tx=0x7fb6fc005800 comp rx=0 tx=0).stop 2023-10-30T13:56:02.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 shutdown_connections 2023-10-30T13:56:02.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb6f00664a0 0x7fb6f0068960 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb7100c5590 0x7fb7101f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7100c2b40 0x7fb7100cdc20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 --2- 172.21.15.31:0/423514954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6fc0040f0 0x7fb7101566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 >> 172.21.15.31:0/423514954 conn(0x7fb710065a80 msgr2=0x7fb71014e660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:02.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 shutdown_connections 2023-10-30T13:56:02.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.755+0000 7fb70f59e640 1 -- 172.21.15.31:0/423514954 wait complete. 2023-10-30T13:56:02.778 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:56:02.778 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:56:02.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:02.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efddabc4640 1 Processor -- start 2023-10-30T13:56:02.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efddabc4640 1 -- start start 2023-10-30T13:56:02.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efddabc4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7efdd4153c80 0x7efdd415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efddabc4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7efdd40c42a0 0x7efdd40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efddabc4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd4159f20 con 0x7efdd40c4a10 2023-10-30T13:56:02.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efddabc4640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd415dec0 con 0x7efdd40c42a0 2023-10-30T13:56:02.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efddabc4640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd41599a0 con 0x7efdd4153c80 2023-10-30T13:56:02.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd9161640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7efdd40c42a0 0x7efdd40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd3fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7efdd40c4a10 0x7efdd415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:48724/0 (socket says 172.21.15.31:48724) 2023-10-30T13:56:02.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd3fff640 1 -- 172.21.15.31:0/3189687629 learned_addr learned my addr 172.21.15.31:0/3189687629 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:02.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd8960640 1 --2- 172.21.15.31:0/3189687629 >> v2:172.21.15.33:3300/0 conn(0x7efdd4153c80 0x7efdd415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd9161640 1 -- 172.21.15.31:0/3189687629 >> v2:172.21.15.33:3300/0 conn(0x7efdd4153c80 msgr2=0x7efdd415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:02.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd9161640 1 --2- 172.21.15.31:0/3189687629 >> v2:172.21.15.33:3300/0 conn(0x7efdd4153c80 0x7efdd415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd9161640 1 -- 172.21.15.31:0/3189687629 >> v1:172.21.15.31:6789/0 conn(0x7efdd40c4a10 legacy=0x7efdd415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd9161640 1 -- 172.21.15.31:0/3189687629 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd415d620 con 0x7efdd40c42a0 2023-10-30T13:56:02.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.919+0000 7efdd8960640 1 --2- 172.21.15.31:0/3189687629 >> v2:172.21.15.33:3300/0 conn(0x7efdd4153c80 0x7efdd415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:02.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efdd9161640 1 --2- 172.21.15.31:0/3189687629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd40ca1e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7efdd40d00f0 tx=0x7efdcc01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a6198d6b1a27a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:02.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efdd37fe640 1 -- 172.21.15.31:0/3189687629 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc0b0440 con 0x7efdd40c42a0 2023-10-30T13:56:02.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efdd37fe640 1 -- 172.21.15.31:0/3189687629 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdcc0b05a0 con 0x7efdd40c42a0 2023-10-30T13:56:02.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efdd37fe640 1 -- 172.21.15.31:0/3189687629 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc0b0930 con 0x7efdd40c42a0 2023-10-30T13:56:02.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 -- 172.21.15.31:0/3189687629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 msgr2=0x7efdd40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 --2- 172.21.15.31:0/3189687629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd40ca1e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7efdd40d00f0 tx=0x7efdcc01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 -- 172.21.15.31:0/3189687629 shutdown_connections 2023-10-30T13:56:02.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 --2- 172.21.15.31:0/3189687629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd40ca1e0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 --2- 172.21.15.31:0/3189687629 >> v2:172.21.15.33:3300/0 conn(0x7efdd4153c80 0x7efdd415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 -- 172.21.15.31:0/3189687629 >> 172.21.15.31:0/3189687629 conn(0x7efdd40715c0 msgr2=0x7efdd40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:02.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 -- 172.21.15.31:0/3189687629 shutdown_connections 2023-10-30T13:56:02.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.923+0000 7efddabc4640 1 -- 172.21.15.31:0/3189687629 wait complete. 2023-10-30T13:56:02.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 Processor -- start 2023-10-30T13:56:02.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 -- start start 2023-10-30T13:56:02.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd4152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efdd40c4a10 0x7efdd4154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdd4153c80 0x7efdd41f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd414a4d0 con 0x7efdd40c42a0 2023-10-30T13:56:02.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd40cef00 con 0x7efdd4153c80 2023-10-30T13:56:02.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd4150270 con 0x7efdd40c4a10 2023-10-30T13:56:02.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd4152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd4152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47434/0 (socket says 172.21.15.31:47434) 2023-10-30T13:56:02.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 -- 172.21.15.31:0/2888882286 learned_addr learned my addr 172.21.15.31:0/2888882286 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:02.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd3fff640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efdd40c4a10 0x7efdd4154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd9161640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdd4153c80 0x7efdd41f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 -- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efdd40c4a10 msgr2=0x7efdd4154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efdd40c4a10 0x7efdd4154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 -- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdd4153c80 msgr2=0x7efdd41f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdd4153c80 0x7efdd41f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 -- 172.21.15.31:0/2888882286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc009a70 con 0x7efdd40c42a0 2023-10-30T13:56:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd8960640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd4152950 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7efdc800d800 tx=0x7efdc800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd17fa640 1 -- 172.21.15.31:0/2888882286 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc80096c0 con 0x7efdd40c42a0 2023-10-30T13:56:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd17fa640 1 -- 172.21.15.31:0/2888882286 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8009820 con 0x7efdd40c42a0 2023-10-30T13:56:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd40cebd0 con 0x7efdd40c42a0 2023-10-30T13:56:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efdd17fa640 1 -- 172.21.15.31:0/2888882286 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8009bd0 con 0x7efdd40c42a0 2023-10-30T13:56:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.927+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd40cf790 con 0x7efdd40c42a0 2023-10-30T13:56:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.931+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd40cb660 con 0x7efdd40c42a0 2023-10-30T13:56:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.931+0000 7efdd17fa640 1 -- 172.21.15.31:0/2888882286 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efdc8009d30 con 0x7efdd40c42a0 2023-10-30T13:56:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.931+0000 7efdd17fa640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efdb80664a0 0x7efdb8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.931+0000 7efdd17fa640 1 -- 172.21.15.31:0/2888882286 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7efdc8084b30 con 0x7efdd40c42a0 2023-10-30T13:56:02.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.931+0000 7efdd3fff640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efdb80664a0 0x7efdb8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:02.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.931+0000 7efdd3fff640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efdb80664a0 0x7efdb8068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7efdd40b19f0 tx=0x7efdc0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:02.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:02.935+0000 7efdd17fa640 1 -- 172.21.15.31:0/2888882286 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efdc8051a90 con 0x7efdd40c42a0 2023-10-30T13:56:03.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdd40cb8d0 con 0x7efdd40c42a0 2023-10-30T13:56:03.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efdd17fa640 1 -- 172.21.15.31:0/2888882286 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5701 (secure 0 0 0) 0x7efdc80550b0 con 0x7efdd40c42a0 2023-10-30T13:56:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:03.130 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:02.130169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2023-10-30T13:56:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efdb80664a0 msgr2=0x7efdb8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efdb80664a0 0x7efdb8068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7efdd40b19f0 tx=0x7efdc0009040 comp rx=0 tx=0).stop 2023-10-30T13:56:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 msgr2=0x7efdd4152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd4152950 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7efdc800d800 tx=0x7efdc800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 shutdown_connections 2023-10-30T13:56:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efdb80664a0 0x7efdb8068960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdd4153c80 0x7efdd41f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efdd40c4a10 0x7efdd4154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 --2- 172.21.15.31:0/2888882286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd40c42a0 0x7efdd4152950 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 >> 172.21.15.31:0/2888882286 conn(0x7efdd40715c0 msgr2=0x7efdd4158790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 shutdown_connections 2023-10-30T13:56:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:03.127+0000 7efddabc4640 1 -- 172.21.15.31:0/2888882286 wait complete. 2023-10-30T13:56:03.151 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5813": { "addr": "172.21.15.33:6836/2281022644", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 2281022644, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 2281022644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5813, "incarnation": 84, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:56:02.130169+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5813 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:56:05.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:05.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.303+0000 7f6e6f4d6640 1 Processor -- start 2023-10-30T13:56:05.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.303+0000 7f6e6f4d6640 1 -- start start 2023-10-30T13:56:05.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6f4d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6e68150830 0x7f6e6806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6f4d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68070520 0x7f6e680709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6f4d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e680c2310 con 0x7f6e680cb7d0 2023-10-30T13:56:05.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6f4d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e68156710 con 0x7f6e68150830 2023-10-30T13:56:05.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6f4d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e680c4360 con 0x7f6e68070520 2023-10-30T13:56:05.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6d272640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6e68150830 0x7f6e6806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6d272640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6e68150830 0x7f6e6806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60966/0 (socket says 172.21.15.31:60966) 2023-10-30T13:56:05.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6d272640 1 -- 172.21.15.31:0/1841897238 learned_addr learned my addr 172.21.15.31:0/1841897238 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:05.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6ca71640 1 --1- 172.21.15.31:0/1841897238 >> v1:172.21.15.31:6789/0 conn(0x7f6e680cb7d0 0x7f6e680c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49684/0 (socket says 172.21.15.31:49684) 2023-10-30T13:56:05.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6da73640 1 --2- 172.21.15.31:0/1841897238 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68070520 0x7f6e680709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6d272640 1 -- 172.21.15.31:0/1841897238 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68070520 msgr2=0x7f6e680709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:05.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6d272640 1 --2- 172.21.15.31:0/1841897238 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68070520 0x7f6e680709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6d272640 1 -- 172.21.15.31:0/1841897238 >> v1:172.21.15.31:6789/0 conn(0x7f6e680cb7d0 legacy=0x7f6e680c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6d272640 1 -- 172.21.15.31:0/1841897238 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e681569b0 con 0x7f6e68150830 2023-10-30T13:56:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e5ffff640 1 -- 172.21.15.31:0/1841897238 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163441644 0 0) 0x7f6e680c2310 con 0x7f6e680cb7d0 2023-10-30T13:56:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.307+0000 7f6e6da73640 1 --2- 172.21.15.31:0/1841897238 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68070520 0x7f6e680709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:05.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6d272640 1 --2- 172.21.15.31:0/1841897238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68150830 0x7f6e6806b500 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009870 tx=0x7f6e6401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e1ea86e453a6c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e5ffff640 1 -- 172.21.15.31:0/1841897238 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640a8a30 con 0x7f6e68150830 2023-10-30T13:56:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e5ffff640 1 -- 172.21.15.31:0/1841897238 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e640a8b90 con 0x7f6e68150830 2023-10-30T13:56:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e5ffff640 1 -- 172.21.15.31:0/1841897238 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640b0440 con 0x7f6e68150830 2023-10-30T13:56:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/1841897238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68150830 msgr2=0x7f6e6806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/1841897238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68150830 0x7f6e6806b500 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009870 tx=0x7f6e6401ed30 comp rx=0 tx=0).stop 2023-10-30T13:56:05.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/1841897238 shutdown_connections 2023-10-30T13:56:05.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/1841897238 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68070520 0x7f6e680709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/1841897238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68150830 0x7f6e6806b500 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/1841897238 >> 172.21.15.31:0/1841897238 conn(0x7f6e680cac70 msgr2=0x7f6e680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:05.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/1841897238 shutdown_connections 2023-10-30T13:56:05.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/1841897238 wait complete. 2023-10-30T13:56:05.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 Processor -- start 2023-10-30T13:56:05.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.311+0000 7f6e6f4d6640 1 -- start start 2023-10-30T13:56:05.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68070520 0x7f6e681511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e680cb7d0 0x7f6e68149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68150830 0x7f6e6814b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e681f2f90 con 0x7f6e68070520 2023-10-30T13:56:05.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68155480 con 0x7f6e680cb7d0 2023-10-30T13:56:05.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e681f2cb0 con 0x7f6e68150830 2023-10-30T13:56:05.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68070520 0x7f6e681511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68070520 0x7f6e681511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60970/0 (socket says 172.21.15.31:60970) 2023-10-30T13:56:05.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 -- 172.21.15.31:0/838930151 learned_addr learned my addr 172.21.15.31:0/838930151 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:05.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6da73640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68150830 0x7f6e6814b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6ca71640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e680cb7d0 0x7f6e68149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 -- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68150830 msgr2=0x7f6e6814b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68150830 0x7f6e6814b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 -- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e680cb7d0 msgr2=0x7f6e68149130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e680cb7d0 0x7f6e68149130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 -- 172.21.15.31:0/838930151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640094d0 con 0x7f6e68070520 2023-10-30T13:56:05.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6d272640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68070520 0x7f6e681511d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6e640b3040 tx=0x7f6e6401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:05.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e5dffb640 1 -- 172.21.15.31:0/838930151 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640be070 con 0x7f6e68070520 2023-10-30T13:56:05.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e5dffb640 1 -- 172.21.15.31:0/838930151 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e640ba400 con 0x7f6e68070520 2023-10-30T13:56:05.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e68153be0 con 0x7f6e68070520 2023-10-30T13:56:05.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e5dffb640 1 -- 172.21.15.31:0/838930151 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640ba7d0 con 0x7f6e68070520 2023-10-30T13:56:05.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e680c69a0 con 0x7f6e68070520 2023-10-30T13:56:05.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e68153a40 con 0x7f6e68070520 2023-10-30T13:56:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.315+0000 7f6e5dffb640 1 -- 172.21.15.31:0/838930151 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e640c3410 con 0x7f6e68070520 2023-10-30T13:56:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.319+0000 7f6e5dffb640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6e4c0664a0 0x7f6e4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.319+0000 7f6e5dffb640 1 -- 172.21.15.31:0/838930151 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f6e64121e20 con 0x7f6e68070520 2023-10-30T13:56:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.319+0000 7f6e6ca71640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6e4c0664a0 0x7f6e4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.319+0000 7f6e6ca71640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6e4c0664a0 0x7f6e4c068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6e681541c0 tx=0x7f6e58007b60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:05.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.323+0000 7f6e5dffb640 1 -- 172.21.15.31:0/838930151 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e68153a40 con 0x7f6e68070520 2023-10-30T13:56:05.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e68153a40 con 0x7f6e68070520 2023-10-30T13:56:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e5dffb640 1 -- 172.21.15.31:0/838930151 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f6e68153a40 con 0x7f6e68070520 2023-10-30T13:56:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:05.518 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:02.130169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2023-10-30T13:56:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6e4c0664a0 msgr2=0x7f6e4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6e4c0664a0 0x7f6e4c068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6e681541c0 tx=0x7f6e58007b60 comp rx=0 tx=0).stop 2023-10-30T13:56:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68070520 msgr2=0x7f6e681511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68070520 0x7f6e681511d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6e640b3040 tx=0x7f6e6401bad0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 shutdown_connections 2023-10-30T13:56:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6e4c0664a0 0x7f6e4c068960 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6e68150830 0x7f6e6814b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e680cb7d0 0x7f6e68149130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 --2- 172.21.15.31:0/838930151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e68070520 0x7f6e681511d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 >> 172.21.15.31:0/838930151 conn(0x7f6e680cac70 msgr2=0x7f6e6814dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 shutdown_connections 2023-10-30T13:56:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:05.515+0000 7f6e6f4d6640 1 -- 172.21.15.31:0/838930151 wait complete. 2023-10-30T13:56:05.538 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:07.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:07.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.691+0000 7f8511868640 1 Processor -- start 2023-10-30T13:56:07.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.691+0000 7f8511868640 1 -- start start 2023-10-30T13:56:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8511868640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8511868640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0c2310 con 0x7f850c0c3e40 2023-10-30T13:56:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8511868640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c156710 con 0x7f850c150830 2023-10-30T13:56:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8511868640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c06bc10 con 0x7f850c0cb7d0 2023-10-30T13:56:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f850b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f850c0c3e40 0x7f850c070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49700/0 (socket says 172.21.15.31:49700) 2023-10-30T13:56:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f850b7fe640 1 -- 172.21.15.31:0/478248498 learned_addr learned my addr 172.21.15.31:0/478248498 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:07.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8503fff640 1 --2- 172.21.15.31:0/478248498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:07.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f850a7fc640 1 -- 172.21.15.31:0/478248498 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010406768 0 0) 0x7f850c0c2310 con 0x7f850c0c3e40 2023-10-30T13:56:07.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f850a7fc640 1 -- 172.21.15.31:0/478248498 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f8003530 con 0x7f850c0c3e40 2023-10-30T13:56:07.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8503fff640 1 -- 172.21.15.31:0/478248498 >> v1:172.21.15.33:6789/0 conn(0x7f850c150830 legacy=0x7f850c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:07.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8503fff640 1 -- 172.21.15.31:0/478248498 >> v1:172.21.15.31:6789/0 conn(0x7f850c0c3e40 legacy=0x7f850c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:07.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.695+0000 7f8503fff640 1 -- 172.21.15.31:0/478248498 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0c4360 con 0x7f850c0cb7d0 2023-10-30T13:56:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8503fff640 1 --2- 172.21.15.31:0/478248498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c0c3900 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84f40097f0 tx=0x7f84f401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8e7d8d4d6fd1b8d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f850a7fc640 1 -- 172.21.15.31:0/478248498 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f40a89b0 con 0x7f850c0cb7d0 2023-10-30T13:56:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f850a7fc640 1 -- 172.21.15.31:0/478248498 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f40a8b10 con 0x7f850c0cb7d0 2023-10-30T13:56:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f850a7fc640 1 -- 172.21.15.31:0/478248498 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f40a8e80 con 0x7f850c0cb7d0 2023-10-30T13:56:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 -- 172.21.15.31:0/478248498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 msgr2=0x7f850c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 --2- 172.21.15.31:0/478248498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c0c3900 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84f40097f0 tx=0x7f84f401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 -- 172.21.15.31:0/478248498 shutdown_connections 2023-10-30T13:56:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 --2- 172.21.15.31:0/478248498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c0c3900 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 -- 172.21.15.31:0/478248498 >> 172.21.15.31:0/478248498 conn(0x7f850c0cac70 msgr2=0x7f850c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:07.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 -- 172.21.15.31:0/478248498 shutdown_connections 2023-10-30T13:56:07.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 -- 172.21.15.31:0/478248498 wait complete. 2023-10-30T13:56:07.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 Processor -- start 2023-10-30T13:56:07.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.699+0000 7f8511868640 1 -- start start 2023-10-30T13:56:07.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f850c0c3e40 0x7f850c1ef710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c1f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f850c150830 0x7f850c073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1f26e0 con 0x7f850c150830 2023-10-30T13:56:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1efc50 con 0x7f850c0c3e40 2023-10-30T13:56:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c157d30 con 0x7f850c0cb7d0 2023-10-30T13:56:07.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f850b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f850c150830 0x7f850c073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:07.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8503fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c1f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:07.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f850b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f850c150830 0x7f850c073620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60978/0 (socket says 172.21.15.31:60978) 2023-10-30T13:56:07.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f850b7fe640 1 -- 172.21.15.31:0/1428023773 learned_addr learned my addr 172.21.15.31:0/1428023773 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:07.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f850affd640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f850c0c3e40 0x7f850c1ef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:07.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8503fff640 1 -- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f850c0c3e40 msgr2=0x7f850c1ef710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:07.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8503fff640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f850c0c3e40 0x7f850c1ef710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8503fff640 1 -- 172.21.15.31:0/1428023773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f850c150830 msgr2=0x7f850c073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:07.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8503fff640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f850c150830 0x7f850c073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8503fff640 1 -- 172.21.15.31:0/1428023773 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f4009420 con 0x7f850c0cb7d0 2023-10-30T13:56:07.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8503fff640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c1f2b00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f850c0c24a0 tx=0x7f84f401d4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:07.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8508ff9640 1 -- 172.21.15.31:0/1428023773 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f40b1e80 con 0x7f850c0cb7d0 2023-10-30T13:56:07.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8508ff9640 1 -- 172.21.15.31:0/1428023773 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4006e20 con 0x7f850c0cb7d0 2023-10-30T13:56:07.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c0c9cc0 con 0x7f850c0cb7d0 2023-10-30T13:56:07.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8508ff9640 1 -- 172.21.15.31:0/1428023773 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4004230 con 0x7f850c0cb7d0 2023-10-30T13:56:07.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c0c9700 con 0x7f850c0cb7d0 2023-10-30T13:56:07.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.703+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c066510 con 0x7f850c0cb7d0 2023-10-30T13:56:07.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.707+0000 7f8508ff9640 1 -- 172.21.15.31:0/1428023773 <== mon.2 v2:172.21.15.33:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84f4004390 con 0x7f850c0cb7d0 2023-10-30T13:56:07.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.707+0000 7f8508ff9640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84ec0664d0 0x7f84ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:07.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.707+0000 7f8508ff9640 1 -- 172.21.15.31:0/1428023773 <== mon.2 v2:172.21.15.33:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f84f4122e50 con 0x7f850c0cb7d0 2023-10-30T13:56:07.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.707+0000 7f850affd640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84ec0664d0 0x7f84ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:07.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.707+0000 7f850affd640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84ec0664d0 0x7f84ec068990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f84fc001810 tx=0x7f84fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.723+0000 7f8508ff9640 1 -- 172.21.15.31:0/1428023773 <== mon.2 v2:172.21.15.33:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84f40f0100 con 0x7f850c0cb7d0 2023-10-30T13:56:07.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.915+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850c066510 con 0x7f850c0cb7d0 2023-10-30T13:56:07.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.915+0000 7f8508ff9640 1 -- 172.21.15.31:0/1428023773 <== mon.2 v2:172.21.15.33:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f84f40f3720 con 0x7f850c0cb7d0 2023-10-30T13:56:07.919 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:07.920 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:02.130169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:07.920 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2023-10-30T13:56:07.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84ec0664d0 msgr2=0x7f84ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:07.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84ec0664d0 0x7f84ec068990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f84fc001810 tx=0x7f84fc009040 comp rx=0 tx=0).stop 2023-10-30T13:56:07.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 msgr2=0x7f850c1f2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:07.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c1f2b00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f850c0c24a0 tx=0x7f84f401d4f0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 shutdown_connections 2023-10-30T13:56:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f84ec0664d0 0x7f84ec068990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f850c150830 0x7f850c073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f850c0cb7d0 0x7f850c1f2b00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 --2- 172.21.15.31:0/1428023773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f850c0c3e40 0x7f850c1ef710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 >> 172.21.15.31:0/1428023773 conn(0x7f850c0cac70 msgr2=0x7f850c06b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 shutdown_connections 2023-10-30T13:56:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:07.919+0000 7f8511868640 1 -- 172.21.15.31:0/1428023773 wait complete. 2023-10-30T13:56:07.941 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:09.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 Processor -- start 2023-10-30T13:56:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 -- start start 2023-10-30T13:56:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa1fc1516c0 0x7fa1fc0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc0c82d0 0x7fa1fc0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa1fc0cd4a0 0x7fa1fc0c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1fc0c74a0 con 0x7fa1fc1516c0 2023-10-30T13:56:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1fc151aa0 con 0x7fa1fc0cd4a0 2023-10-30T13:56:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa203ec9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc158af0 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa201c65640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa1fc1516c0 0x7fa1fc0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa201c65640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa1fc1516c0 0x7fa1fc0c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60984/0 (socket says 172.21.15.31:60984) 2023-10-30T13:56:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.083+0000 7fa201c65640 1 -- 172.21.15.31:0/643700628 learned_addr learned my addr 172.21.15.31:0/643700628 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa202466640 1 --2- 172.21.15.31:0/643700628 >> v2:172.21.15.33:3300/0 conn(0x7fa1fc0cd4a0 0x7fa1fc0c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa201464640 1 --2- 172.21.15.31:0/643700628 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc0c82d0 0x7fa1fc0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa201c65640 1 -- 172.21.15.31:0/643700628 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc0c82d0 msgr2=0x7fa1fc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:10.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa201c65640 1 --2- 172.21.15.31:0/643700628 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc0c82d0 0x7fa1fc0ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa201c65640 1 -- 172.21.15.31:0/643700628 >> v2:172.21.15.33:3300/0 conn(0x7fa1fc0cd4a0 msgr2=0x7fa1fc0c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:10.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa201c65640 1 --2- 172.21.15.31:0/643700628 >> v2:172.21.15.33:3300/0 conn(0x7fa1fc0cd4a0 0x7fa1fc0c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa201c65640 1 -- 172.21.15.31:0/643700628 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc158510 con 0x7fa1fc1516c0 2023-10-30T13:56:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa201c65640 1 --2- 172.21.15.31:0/643700628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc1516c0 0x7fa1fc0c7d90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc071f60 tx=0x7fa1f801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8629acc91b792243 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa200c63640 1 -- 172.21.15.31:0/643700628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80ab600 con 0x7fa1fc1516c0 2023-10-30T13:56:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa200c63640 1 -- 172.21.15.31:0/643700628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f80a7050 con 0x7fa1fc1516c0 2023-10-30T13:56:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.087+0000 7fa200c63640 1 -- 172.21.15.31:0/643700628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80ab980 con 0x7fa1fc1516c0 2023-10-30T13:56:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- 172.21.15.31:0/643700628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc1516c0 msgr2=0x7fa1fc0c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 --2- 172.21.15.31:0/643700628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc1516c0 0x7fa1fc0c7d90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc071f60 tx=0x7fa1f801c500 comp rx=0 tx=0).stop 2023-10-30T13:56:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- 172.21.15.31:0/643700628 shutdown_connections 2023-10-30T13:56:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 --2- 172.21.15.31:0/643700628 >> v2:172.21.15.33:3300/0 conn(0x7fa1fc0cd4a0 0x7fa1fc0c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 --2- 172.21.15.31:0/643700628 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc0c82d0 0x7fa1fc0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 --2- 172.21.15.31:0/643700628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc1516c0 0x7fa1fc0c7d90 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- 172.21.15.31:0/643700628 >> 172.21.15.31:0/643700628 conn(0x7fa1fc071fb0 msgr2=0x7fa1fc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- 172.21.15.31:0/643700628 shutdown_connections 2023-10-30T13:56:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- 172.21.15.31:0/643700628 wait complete. 2023-10-30T13:56:10.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 Processor -- start 2023-10-30T13:56:10.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- start start 2023-10-30T13:56:10.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc0c82d0 0x7fa1fc145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1fc0cd4a0 0x7fa1fc144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc1516c0 0x7fa1fc148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc1f0ba0 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc14b750 con 0x7fa1fc0cd4a0 2023-10-30T13:56:10.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa203ec9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc146bf0 con 0x7fa1fc1516c0 2023-10-30T13:56:10.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc0c82d0 0x7fa1fc145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc0c82d0 0x7fa1fc145090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60986/0 (socket says 172.21.15.31:60986) 2023-10-30T13:56:10.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 -- 172.21.15.31:0/3471702234 learned_addr learned my addr 172.21.15.31:0/3471702234 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:10.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201464640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1fc0cd4a0 0x7fa1fc144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa202466640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc1516c0 0x7fa1fc148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 -- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc1516c0 msgr2=0x7fa1fc148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc1516c0 0x7fa1fc148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 -- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1fc0cd4a0 msgr2=0x7fa1fc144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1fc0cd4a0 0x7fa1fc144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa201c65640 1 -- 172.21.15.31:0/3471702234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f80094d0 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.091+0000 7fa202466640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc1516c0 0x7fa1fc148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:10.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa201c65640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc0c82d0 0x7fa1fc145090 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80ae040 tx=0x7fa1f801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:10.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa1f27fc640 1 -- 172.21.15.31:0/3471702234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80b6620 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa1f27fc640 1 -- 172.21.15.31:0/3471702234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f80b67a0 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa1f27fc640 1 -- 172.21.15.31:0/3471702234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80b6b50 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc1eff20 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc1f04e0 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc04e170 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa1f27fc640 1 -- 172.21.15.31:0/3471702234 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1f80b2040 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa1f27fc640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa1e00664a0 0x7fa1e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa1f27fc640 1 -- 172.21.15.31:0/3471702234 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa1f811e720 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.095+0000 7fa201464640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa1e00664a0 0x7fa1e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.099+0000 7fa201464640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa1e00664a0 0x7fa1e0068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc151d50 tx=0x7fa1ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:10.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.099+0000 7fa1f27fc640 1 -- 172.21.15.31:0/3471702234 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1f80eb710 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.291+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1fc0c7b70 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.291+0000 7fa1f27fc640 1 -- 172.21.15.31:0/3471702234 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fa1f80b6cb0 con 0x7fa1fc0c82d0 2023-10-30T13:56:10.296 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:10.296 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:02.130169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:10.297 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2023-10-30T13:56:10.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa1e00664a0 msgr2=0x7fa1e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa1e00664a0 0x7fa1e0068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc151d50 tx=0x7fa1ec008040 comp rx=0 tx=0).stop 2023-10-30T13:56:10.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc0c82d0 msgr2=0x7fa1fc145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc0c82d0 0x7fa1fc145090 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80ae040 tx=0x7fa1f801ad40 comp rx=0 tx=0).stop 2023-10-30T13:56:10.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 shutdown_connections 2023-10-30T13:56:10.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa1e00664a0 0x7fa1e0068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa1fc1516c0 0x7fa1fc148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1fc0cd4a0 0x7fa1fc144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 --2- 172.21.15.31:0/3471702234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1fc0c82d0 0x7fa1fc145090 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 >> 172.21.15.31:0/3471702234 conn(0x7fa1fc071fb0 msgr2=0x7fa1fc156250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 shutdown_connections 2023-10-30T13:56:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:10.295+0000 7fa203ec9640 1 -- 172.21.15.31:0/3471702234 wait complete. 2023-10-30T13:56:10.317 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:12.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:12.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f165c4640 1 Processor -- start 2023-10-30T13:56:12.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f165c4640 1 -- start start 2023-10-30T13:56:12.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f165c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f10153c80 0x7f0f1015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f165c4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f10159f20 con 0x7f0f100c4a10 2023-10-30T13:56:12.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f165c4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1015d620 con 0x7f0f100c42a0 2023-10-30T13:56:12.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f165c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1015dec0 con 0x7f0f10153c80 2023-10-30T13:56:12.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0f7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f0f100c4a10 0x7f0f1015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49710/0 (socket says 172.21.15.31:49710) 2023-10-30T13:56:12.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0f7fe640 1 -- 172.21.15.31:0/1444961894 learned_addr learned my addr 172.21.15.31:0/1444961894 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:12.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0ffff640 1 --2- 172.21.15.31:0/1444961894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f10153c80 0x7f0f1015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0effd640 1 -- 172.21.15.31:0/1444961894 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706956304 0 0) 0x7f0f10159f20 con 0x7f0f100c4a10 2023-10-30T13:56:12.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0effd640 1 -- 172.21.15.31:0/1444961894 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef8003530 con 0x7f0f100c4a10 2023-10-30T13:56:12.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0effd640 1 -- 172.21.15.31:0/1444961894 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210094826 0 0) 0x7f0f1015d620 con 0x7f0f100c42a0 2023-10-30T13:56:12.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0effd640 1 -- 172.21.15.31:0/1444961894 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10159f20 con 0x7f0f100c42a0 2023-10-30T13:56:12.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0ffff640 1 -- 172.21.15.31:0/1444961894 >> v1:172.21.15.33:6789/0 conn(0x7f0f100c42a0 legacy=0x7f0f100ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0ffff640 1 -- 172.21.15.31:0/1444961894 >> v1:172.21.15.31:6789/0 conn(0x7f0f100c4a10 legacy=0x7f0f1015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.470+0000 7f0f0ffff640 1 -- 172.21.15.31:0/1444961894 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1015d900 con 0x7f0f10153c80 2023-10-30T13:56:12.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f0ffff640 1 --2- 172.21.15.31:0/1444961894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f10153c80 0x7f0f1015cf90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f000097f0 tx=0x7f0f0001ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aca5559a029b4acc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:12.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f0effd640 1 -- 172.21.15.31:0/1444961894 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000a89b0 con 0x7f0f10153c80 2023-10-30T13:56:12.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f0effd640 1 -- 172.21.15.31:0/1444961894 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f000a8b10 con 0x7f0f10153c80 2023-10-30T13:56:12.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f0effd640 1 -- 172.21.15.31:0/1444961894 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000a8e80 con 0x7f0f10153c80 2023-10-30T13:56:12.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f165c4640 1 -- 172.21.15.31:0/1444961894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f10153c80 msgr2=0x7f0f1015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f165c4640 1 --2- 172.21.15.31:0/1444961894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f10153c80 0x7f0f1015cf90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f000097f0 tx=0x7f0f0001ecb0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f165c4640 1 -- 172.21.15.31:0/1444961894 shutdown_connections 2023-10-30T13:56:12.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f165c4640 1 --2- 172.21.15.31:0/1444961894 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f10153c80 0x7f0f1015cf90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f165c4640 1 -- 172.21.15.31:0/1444961894 >> 172.21.15.31:0/1444961894 conn(0x7f0f100715c0 msgr2=0x7f0f100719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:12.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f165c4640 1 -- 172.21.15.31:0/1444961894 shutdown_connections 2023-10-30T13:56:12.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.474+0000 7f0f165c4640 1 -- 172.21.15.31:0/1444961894 wait complete. 2023-10-30T13:56:12.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 Processor -- start 2023-10-30T13:56:12.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 -- start start 2023-10-30T13:56:12.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f100c42a0 0x7f0f100c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f100c4a10 0x7f0f100ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f10153c80 0x7f0f100ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1015ce90 con 0x7f0f10153c80 2023-10-30T13:56:12.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1014fce0 con 0x7f0f100c42a0 2023-10-30T13:56:12.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f101f3190 con 0x7f0f100c4a10 2023-10-30T13:56:12.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f0ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f100c42a0 0x7f0f100c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f10153c80 0x7f0f100ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f0f7fe640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f100c4a10 0x7f0f100ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f0ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f100c42a0 0x7f0f100c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:51096/0 (socket says 172.21.15.31:51096) 2023-10-30T13:56:12.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f10153c80 0x7f0f100ce5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60998/0 (socket says 172.21.15.31:60998) 2023-10-30T13:56:12.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f0ffff640 1 -- 172.21.15.31:0/3596965438 learned_addr learned my addr 172.21.15.31:0/3596965438 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:12.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 -- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f100c4a10 msgr2=0x7f0f100ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f100c4a10 0x7f0f100ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 -- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f100c42a0 msgr2=0x7f0f100c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f100c42a0 0x7f0f100c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 -- 172.21.15.31:0/3596965438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f00009420 con 0x7f0f10153c80 2023-10-30T13:56:12.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f14b61640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f10153c80 0x7f0f100ce5d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0f08002240 tx=0x7f0f0800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:12.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f0cff9640 1 -- 172.21.15.31:0/3596965438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08006d60 con 0x7f0f10153c80 2023-10-30T13:56:12.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f0cff9640 1 -- 172.21.15.31:0/3596965438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f08005020 con 0x7f0f10153c80 2023-10-30T13:56:12.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f0cff9640 1 -- 172.21.15.31:0/3596965438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08005430 con 0x7f0f10153c80 2023-10-30T13:56:12.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f101ef190 con 0x7f0f10153c80 2023-10-30T13:56:12.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.478+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f101ef750 con 0x7f0f10153c80 2023-10-30T13:56:12.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.482+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1004e170 con 0x7f0f10153c80 2023-10-30T13:56:12.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.482+0000 7f0f0cff9640 1 -- 172.21.15.31:0/3596965438 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f0800a640 con 0x7f0f10153c80 2023-10-30T13:56:12.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.482+0000 7f0f0cff9640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0efc0664a0 0x7f0efc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.482+0000 7f0f0cff9640 1 -- 172.21.15.31:0/3596965438 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f0f08083fc0 con 0x7f0f10153c80 2023-10-30T13:56:12.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.482+0000 7f0f0ffff640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0efc0664a0 0x7f0efc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.482+0000 7f0f0ffff640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0efc0664a0 0x7f0efc068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009740 tx=0x7f0f000aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:12.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.486+0000 7f0f0cff9640 1 -- 172.21.15.31:0/3596965438 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f08054a70 con 0x7f0f10153c80 2023-10-30T13:56:12.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.678+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f100cad40 con 0x7f0f10153c80 2023-10-30T13:56:12.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.678+0000 7f0f0cff9640 1 -- 172.21.15.31:0/3596965438 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f0f08054600 con 0x7f0f10153c80 2023-10-30T13:56:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:12.684 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:02.130169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2023-10-30T13:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0efc0664a0 msgr2=0x7f0efc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0efc0664a0 0x7f0efc068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009740 tx=0x7f0f000aa070 comp rx=0 tx=0).stop 2023-10-30T13:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f10153c80 msgr2=0x7f0f100ce5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f10153c80 0x7f0f100ce5d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0f08002240 tx=0x7f0f0800cb90 comp rx=0 tx=0).stop 2023-10-30T13:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 shutdown_connections 2023-10-30T13:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0efc0664a0 0x7f0efc068960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f10153c80 0x7f0f100ce5d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0f100c4a10 0x7f0f100ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 --2- 172.21.15.31:0/3596965438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f100c42a0 0x7f0f100c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 >> 172.21.15.31:0/3596965438 conn(0x7f0f100715c0 msgr2=0x7f0f100719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 shutdown_connections 2023-10-30T13:56:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:12.682+0000 7f0f165c4640 1 -- 172.21.15.31:0/3596965438 wait complete. 2023-10-30T13:56:12.704 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:14.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:14.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.854+0000 7f7f2b965640 1 Processor -- start 2023-10-30T13:56:14.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.854+0000 7f7f2b965640 1 -- start start 2023-10-30T13:56:14.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f2b965640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7f2406c420 0x7f7f240c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f2b965640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f24071e00 con 0x7f7f2406bb40 2023-10-30T13:56:14.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f2b965640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f24070020 con 0x7f7f2406b760 2023-10-30T13:56:14.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f2b965640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f24070f10 con 0x7f7f2406c420 2023-10-30T13:56:14.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f7f2406bb40 0x7f7f24157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:35922/0 (socket says 172.21.15.31:35922) 2023-10-30T13:56:14.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 -- 172.21.15.31:0/2171350700 learned_addr learned my addr 172.21.15.31:0/2171350700 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:14.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f29f02640 1 --2- 172.21.15.31:0/2171350700 >> v2:172.21.15.33:3300/0 conn(0x7f7f2406c420 0x7f7f240c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422865329 0 0) 0x7f7f24071e00 con 0x7f7f2406bb40 2023-10-30T13:56:14.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f10003530 con 0x7f7f2406bb40 2023-10-30T13:56:14.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687781656 0 0) 0x7f7f10003530 con 0x7f7f2406bb40 2023-10-30T13:56:14.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 >> v2:172.21.15.33:3300/0 conn(0x7f7f2406c420 msgr2=0x7f7f240c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:14.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 --2- 172.21.15.31:0/2171350700 >> v2:172.21.15.33:3300/0 conn(0x7f7f2406c420 0x7f7f240c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:14.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 >> v1:172.21.15.33:6789/0 conn(0x7f7f2406b760 legacy=0x7f7f2415e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:14.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240711f0 con 0x7f7f2406bb40 2023-10-30T13:56:14.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f7f14001680 con 0x7f7f2406bb40 2023-10-30T13:56:14.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 >> v1:172.21.15.31:6789/0 conn(0x7f7f2406bb40 legacy=0x7f7f24157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:14.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f10003980 0x7f7f10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f100047d0 0x7f7f10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f10005190 0x7f7f1000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24070f10 con 0x7f7f100047d0 2023-10-30T13:56:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1000c030 con 0x7f7f10005190 2023-10-30T13:56:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1000c1f0 con 0x7f7f10003980 2023-10-30T13:56:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f100047d0 0x7f7f10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f29701640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f10003980 0x7f7f10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f29f02640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f10005190 0x7f7f1000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 -- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f10003980 msgr2=0x7f7f10003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f10003980 0x7f7f10003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 -- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f10005190 msgr2=0x7f7f1000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f10005190 0x7f7f1000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f28f00640 1 -- 172.21.15.31:0/2171350700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f10003530 con 0x7f7f100047d0 2023-10-30T13:56:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.858+0000 7f7f29f02640 1 -- 172.21.15.31:0/2171350700 reap_dead start 2023-10-30T13:56:14.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f28f00640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f100047d0 0x7f7f10004c50 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c210 tx=0x7f7f1401f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a3aa47f773d0b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:14.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14005ae0 con 0x7f7f100047d0 2023-10-30T13:56:14.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f140ab070 con 0x7f7f100047d0 2023-10-30T13:56:14.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f1bfff640 1 -- 172.21.15.31:0/2171350700 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14007070 con 0x7f7f100047d0 2023-10-30T13:56:14.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 -- 172.21.15.31:0/2171350700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f100047d0 msgr2=0x7f7f10004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:14.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f100047d0 0x7f7f10004c50 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c210 tx=0x7f7f1401f330 comp rx=0 tx=0).stop 2023-10-30T13:56:14.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 -- 172.21.15.31:0/2171350700 shutdown_connections 2023-10-30T13:56:14.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 --2- 172.21.15.31:0/2171350700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f100047d0 0x7f7f10004c50 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:14.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 -- 172.21.15.31:0/2171350700 >> 172.21.15.31:0/2171350700 conn(0x7f7f240cac70 msgr2=0x7f7f240cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:14.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 -- 172.21.15.31:0/2171350700 shutdown_connections 2023-10-30T13:56:14.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 -- 172.21.15.31:0/2171350700 wait complete. 2023-10-30T13:56:14.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 Processor -- start 2023-10-30T13:56:14.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.862+0000 7f7f2b965640 1 -- start start 2023-10-30T13:56:14.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f100047d0 0x7f7f241f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f240c4d20 0x7f7f2415ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f241f4520 0x7f7f241508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2415e720 con 0x7f7f240c4d20 2023-10-30T13:56:14.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24156ed0 con 0x7f7f100047d0 2023-10-30T13:56:14.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f241f2690 con 0x7f7f241f4520 2023-10-30T13:56:14.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f240c4d20 0x7f7f2415ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f240c4d20 0x7f7f2415ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58086/0 (socket says 172.21.15.31:58086) 2023-10-30T13:56:14.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 -- 172.21.15.31:0/3781075088 learned_addr learned my addr 172.21.15.31:0/3781075088 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:14.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f29701640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f100047d0 0x7f7f241f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f29f02640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f241f4520 0x7f7f241508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 -- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f241f4520 msgr2=0x7f7f241508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:14.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f241f4520 0x7f7f241508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:14.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 -- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f100047d0 msgr2=0x7f7f241f2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:14.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f100047d0 0x7f7f241f2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:14.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 -- 172.21.15.31:0/3781075088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f241f49a0 con 0x7f7f240c4d20 2023-10-30T13:56:14.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f29701640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f100047d0 0x7f7f241f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:56:14.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f28f00640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f240c4d20 0x7f7f2415ea30 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7f1401f160 tx=0x7f7f10005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:14.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f19ffb640 1 -- 172.21.15.31:0/3781075088 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14004b30 con 0x7f7f240c4d20 2023-10-30T13:56:14.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f19ffb640 1 -- 172.21.15.31:0/3781075088 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f14004c90 con 0x7f7f240c4d20 2023-10-30T13:56:14.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f241f1aa0 con 0x7f7f240c4d20 2023-10-30T13:56:14.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f19ffb640 1 -- 172.21.15.31:0/3781075088 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14007070 con 0x7f7f240c4d20 2023-10-30T13:56:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f241f1d60 con 0x7f7f240c4d20 2023-10-30T13:56:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.866+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2404e170 con 0x7f7f240c4d20 2023-10-30T13:56:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.870+0000 7f7f19ffb640 1 -- 172.21.15.31:0/3781075088 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f140b9030 con 0x7f7f240c4d20 2023-10-30T13:56:14.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.870+0000 7f7f19ffb640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f080664a0 0x7f7f08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:14.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.870+0000 7f7f19ffb640 1 -- 172.21.15.31:0/3781075088 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f7f140b7070 con 0x7f7f240c4d20 2023-10-30T13:56:14.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.870+0000 7f7f29701640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f080664a0 0x7f7f08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:14.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.870+0000 7f7f29701640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f080664a0 0x7f7f08068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7f20007fd0 tx=0x7f7f20009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:14.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:14.874+0000 7f7f19ffb640 1 -- 172.21.15.31:0/3781075088 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f140ee170 con 0x7f7f240c4d20 2023-10-30T13:56:15.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.062+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f24066510 con 0x7f7f240c4d20 2023-10-30T13:56:15.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.062+0000 7f7f19ffb640 1 -- 172.21.15.31:0/3781075088 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f7f140f1790 con 0x7f7f240c4d20 2023-10-30T13:56:15.067 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:15.068 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:13.769680+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:15.068 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 85 2023-10-30T13:56:15.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f080664a0 msgr2=0x7f7f08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f080664a0 0x7f7f08068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7f20007fd0 tx=0x7f7f20009040 comp rx=0 tx=0).stop 2023-10-30T13:56:15.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f240c4d20 msgr2=0x7f7f2415ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f240c4d20 0x7f7f2415ea30 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7f1401f160 tx=0x7f7f10005800 comp rx=0 tx=0).stop 2023-10-30T13:56:15.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 shutdown_connections 2023-10-30T13:56:15.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7f080664a0 0x7f7f08068960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7f241f4520 0x7f7f241508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f240c4d20 0x7f7f2415ea30 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 --2- 172.21.15.31:0/3781075088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f100047d0 0x7f7f241f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 >> 172.21.15.31:0/3781075088 conn(0x7f7f240cac70 msgr2=0x7f7f241577f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:15.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 shutdown_connections 2023-10-30T13:56:15.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:15.066+0000 7f7f2b965640 1 -- 172.21.15.31:0/3781075088 wait complete. 2023-10-30T13:56:15.089 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 85, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 85, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5813": { "addr": "172.21.15.33:6836/2281022644", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 2281022644, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 2281022644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5813, "incarnation": 84, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:56:13.769680+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5813 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:56:17.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c37c9c640 1 Processor -- start 2023-10-30T13:56:17.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c37c9c640 1 -- start start 2023-10-30T13:56:17.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c37c9c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4c3006bb40 0x7f4c30157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c37c9c640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c30071e00 con 0x7f4c3006b760 2023-10-30T13:56:17.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c37c9c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c30070020 con 0x7f4c3006c420 2023-10-30T13:56:17.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c37c9c640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c30070f10 con 0x7f4c3006bb40 2023-10-30T13:56:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c35237640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4c3006bb40 0x7f4c30157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c35237640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4c3006bb40 0x7f4c30157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58100/0 (socket says 172.21.15.31:58100) 2023-10-30T13:56:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c35237640 1 -- 172.21.15.31:0/494742541 learned_addr learned my addr 172.21.15.31:0/494742541 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c35a38640 1 --1- 172.21.15.31:0/494742541 >> v1:172.21.15.31:6789/0 conn(0x7f4c3006b760 0x7f4c3015e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:35936/0 (socket says 172.21.15.31:35936) 2023-10-30T13:56:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c35237640 1 -- 172.21.15.31:0/494742541 >> v1:172.21.15.33:6789/0 conn(0x7f4c3006c420 legacy=0x7f4c300c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c35237640 1 -- 172.21.15.31:0/494742541 >> v1:172.21.15.31:6789/0 conn(0x7f4c3006b760 legacy=0x7f4c3015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.242+0000 7f4c35237640 1 -- 172.21.15.31:0/494742541 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c300711f0 con 0x7f4c3006bb40 2023-10-30T13:56:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c35237640 1 --2- 172.21.15.31:0/494742541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c30157a20 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009870 tx=0x7f4c2001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad3a6b16f7e6459 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c34a36640 1 -- 172.21.15.31:0/494742541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c200aa670 con 0x7f4c3006bb40 2023-10-30T13:56:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c34a36640 1 -- 172.21.15.31:0/494742541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c200aa7d0 con 0x7f4c3006bb40 2023-10-30T13:56:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c34a36640 1 -- 172.21.15.31:0/494742541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c200aab60 con 0x7f4c3006bb40 2023-10-30T13:56:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c37c9c640 1 -- 172.21.15.31:0/494742541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 msgr2=0x7f4c30157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/494742541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c30157a20 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009870 tx=0x7f4c2001c510 comp rx=0 tx=0).stop 2023-10-30T13:56:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c37c9c640 1 -- 172.21.15.31:0/494742541 shutdown_connections 2023-10-30T13:56:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/494742541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c30157a20 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c37c9c640 1 -- 172.21.15.31:0/494742541 >> 172.21.15.31:0/494742541 conn(0x7f4c300cac70 msgr2=0x7f4c300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c37c9c640 1 -- 172.21.15.31:0/494742541 shutdown_connections 2023-10-30T13:56:17.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.246+0000 7f4c37c9c640 1 -- 172.21.15.31:0/494742541 wait complete. 2023-10-30T13:56:17.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 Processor -- start 2023-10-30T13:56:17.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 -- start start 2023-10-30T13:56:17.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c3006b760 0x7f4c3014f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c300c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c3006c420 0x7f4c301f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c300c3a20 con 0x7f4c3006bb40 2023-10-30T13:56:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c300c3130 con 0x7f4c3006b760 2023-10-30T13:56:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c30152750 con 0x7f4c3006c420 2023-10-30T13:56:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c300c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c300c3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58104/0 (socket says 172.21.15.31:58104) 2023-10-30T13:56:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 -- 172.21.15.31:0/3354933193 learned_addr learned my addr 172.21.15.31:0/3354933193 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:17.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35a38640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c3006b760 0x7f4c3014f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c36239640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c3006c420 0x7f4c301f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 -- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c3006c420 msgr2=0x7f4c301f9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c3006c420 0x7f4c301f9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 -- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c3006b760 msgr2=0x7f4c3014f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c3006b760 0x7f4c3014f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 -- 172.21.15.31:0/3354933193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c200094d0 con 0x7f4c3006bb40 2023-10-30T13:56:17.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c35237640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c300c3eb0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4c20019120 tx=0x7f4c2001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:17.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c267fc640 1 -- 172.21.15.31:0/3354933193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c200b38c0 con 0x7f4c3006bb40 2023-10-30T13:56:17.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c267fc640 1 -- 172.21.15.31:0/3354933193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c200b3a20 con 0x7f4c3006bb40 2023-10-30T13:56:17.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c301faab0 con 0x7f4c3006bb40 2023-10-30T13:56:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c267fc640 1 -- 172.21.15.31:0/3354933193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c200b3dd0 con 0x7f4c3006bb40 2023-10-30T13:56:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c300c36b0 con 0x7f4c3006bb40 2023-10-30T13:56:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.250+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c30066510 con 0x7f4c3006bb40 2023-10-30T13:56:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.254+0000 7f4c267fc640 1 -- 172.21.15.31:0/3354933193 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c200b2380 con 0x7f4c3006bb40 2023-10-30T13:56:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.254+0000 7f4c267fc640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4c140664a0 0x7f4c14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.254+0000 7f4c267fc640 1 -- 172.21.15.31:0/3354933193 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f4c2011ec10 con 0x7f4c3006bb40 2023-10-30T13:56:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.254+0000 7f4c35a38640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4c140664a0 0x7f4c14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.254+0000 7f4c35a38640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4c140664a0 0x7f4c14068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c000f70 tx=0x7f4c2c007c90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:17.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.258+0000 7f4c267fc640 1 -- 172.21.15.31:0/3354933193 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c200ebcb0 con 0x7f4c3006bb40 2023-10-30T13:56:17.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.450+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c3004e170 con 0x7f4c3006bb40 2023-10-30T13:56:17.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.450+0000 7f4c267fc640 1 -- 172.21.15.31:0/3354933193 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f4c200ef2d0 con 0x7f4c3006bb40 2023-10-30T13:56:17.456 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:17.456 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:13.769680+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:17.457 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 85 2023-10-30T13:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4c140664a0 msgr2=0x7f4c14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4c140664a0 0x7f4c14068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c000f70 tx=0x7f4c2c007c90 comp rx=0 tx=0).stop 2023-10-30T13:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 msgr2=0x7f4c300c3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c300c3eb0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4c20019120 tx=0x7f4c2001ad50 comp rx=0 tx=0).stop 2023-10-30T13:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 shutdown_connections 2023-10-30T13:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4c140664a0 0x7f4c14068960 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4c3006c420 0x7f4c301f9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c3006bb40 0x7f4c300c3eb0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 --2- 172.21.15.31:0/3354933193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c3006b760 0x7f4c3014f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 >> 172.21.15.31:0/3354933193 conn(0x7f4c300cac70 msgr2=0x7f4c301518f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 shutdown_connections 2023-10-30T13:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:17.454+0000 7f4c37c9c640 1 -- 172.21.15.31:0/3354933193 wait complete. 2023-10-30T13:56:17.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:19.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:19.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.626+0000 7f40de33e640 1 Processor -- start 2023-10-30T13:56:19.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40de33e640 1 -- start start 2023-10-30T13:56:19.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40de33e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 0x7f40d806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40de33e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d80c55f0 con 0x7f40d806be90 2023-10-30T13:56:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40de33e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d81567b0 con 0x7f40d80c6320 2023-10-30T13:56:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40de33e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d80c5f00 con 0x7f40d814fc50 2023-10-30T13:56:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f40d806be90 0x7f40d8155c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:35940/0 (socket says 172.21.15.31:35940) 2023-10-30T13:56:19.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 -- 172.21.15.31:0/543668733 learned_addr learned my addr 172.21.15.31:0/543668733 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:19.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d7fff640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 0x7f40d806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78666206 0 0) 0x7f40d80c55f0 con 0x7f40d806be90 2023-10-30T13:56:19.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c0003450 con 0x7f40d806be90 2023-10-30T13:56:19.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 152216336 0 0) 0x7f40c0003450 con 0x7f40d806be90 2023-10-30T13:56:19.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 msgr2=0x7f40d806fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:19.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 0x7f40d806fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 >> v1:172.21.15.33:6789/0 conn(0x7f40d80c6320 legacy=0x7f40d8070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:19.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d8156a70 con 0x7f40d806be90 2023-10-30T13:56:19.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f40d0001a00 con 0x7f40d806be90 2023-10-30T13:56:19.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 >> v1:172.21.15.31:6789/0 conn(0x7f40d806be90 legacy=0x7f40d8155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:19.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 0x7f40c0003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40c0003fe0 0x7f40c000a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c000af40 0x7f40c000d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d80c5f00 con 0x7f40c000af40 2023-10-30T13:56:19.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c000dc90 con 0x7f40c0003fe0 2023-10-30T13:56:19.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c000de50 con 0x7f40d814fc50 2023-10-30T13:56:19.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c000af40 0x7f40c000d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d7fff640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 0x7f40c0003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d77fe640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40c0003fe0 0x7f40c000a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 -- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 msgr2=0x7f40c0003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:19.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d814fc50 0x7f40c0003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 -- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40c0003fe0 msgr2=0x7f40c000a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:19.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40c0003fe0 0x7f40c000a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40dc8db640 1 -- 172.21.15.31:0/543668733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0003450 con 0x7f40c000af40 2023-10-30T13:56:19.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.630+0000 7f40d77fe640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40c0003fe0 0x7f40c000a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:19.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40dc8db640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c000af40 0x7f40c000d3b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f40d000a830 tx=0x7f40d001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0f38f8a48ef906 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d00ad630 con 0x7f40c000af40 2023-10-30T13:56:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d00a8070 con 0x7f40c000af40 2023-10-30T13:56:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40d6ffd640 1 -- 172.21.15.31:0/543668733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d00ada30 con 0x7f40c000af40 2023-10-30T13:56:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40de33e640 1 -- 172.21.15.31:0/543668733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c000af40 msgr2=0x7f40c000d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40de33e640 1 --2- 172.21.15.31:0/543668733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c000af40 0x7f40c000d3b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f40d000a830 tx=0x7f40d001d340 comp rx=0 tx=0).stop 2023-10-30T13:56:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40dc8db640 1 -- 172.21.15.31:0/543668733 reap_dead start 2023-10-30T13:56:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40de33e640 1 -- 172.21.15.31:0/543668733 shutdown_connections 2023-10-30T13:56:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40de33e640 1 -- 172.21.15.31:0/543668733 >> 172.21.15.31:0/543668733 conn(0x7f40d80cac70 msgr2=0x7f40d80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40de33e640 1 -- 172.21.15.31:0/543668733 shutdown_connections 2023-10-30T13:56:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.634+0000 7f40de33e640 1 -- 172.21.15.31:0/543668733 wait complete. 2023-10-30T13:56:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 Processor -- start 2023-10-30T13:56:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 -- start start 2023-10-30T13:56:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d80c9e30 0x7f40d8157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d80ca210 0x7f40d8158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d8156e00 0x7f40d81e2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d8157810 con 0x7f40d8156e00 2023-10-30T13:56:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d80c4ef0 con 0x7f40d80ca210 2023-10-30T13:56:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d80c3ff0 con 0x7f40d80c9e30 2023-10-30T13:56:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d8156e00 0x7f40d81e2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d8156e00 0x7f40d81e2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58110/0 (socket says 172.21.15.31:58110) 2023-10-30T13:56:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 -- 172.21.15.31:0/2883987448 learned_addr learned my addr 172.21.15.31:0/2883987448 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:19.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40d7fff640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d80c9e30 0x7f40d8157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40d77fe640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d80ca210 0x7f40d8158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 -- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d80c9e30 msgr2=0x7f40d8157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:19.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d80c9e30 0x7f40d8157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 -- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d80ca210 msgr2=0x7f40d8158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:19.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d80ca210 0x7f40d8158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 -- 172.21.15.31:0/2883987448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d80c4370 con 0x7f40d8156e00 2023-10-30T13:56:19.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40d77fe640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d80ca210 0x7f40d8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:19.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40dc8db640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d8156e00 0x7f40d81e2650 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f40d001dbe0 tx=0x7f40c0003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:19.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40d4ff9640 1 -- 172.21.15.31:0/2883987448 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0001fd0 con 0x7f40d8156e00 2023-10-30T13:56:19.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40d4ff9640 1 -- 172.21.15.31:0/2883987448 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d00011d0 con 0x7f40d8156e00 2023-10-30T13:56:19.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40d4ff9640 1 -- 172.21.15.31:0/2883987448 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d00ad430 con 0x7f40d8156e00 2023-10-30T13:56:19.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d80c4650 con 0x7f40d8156e00 2023-10-30T13:56:19.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.638+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d80c4c10 con 0x7f40d8156e00 2023-10-30T13:56:19.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.642+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d8157b60 con 0x7f40d8156e00 2023-10-30T13:56:19.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.642+0000 7f40d4ff9640 1 -- 172.21.15.31:0/2883987448 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40d00aa030 con 0x7f40d8156e00 2023-10-30T13:56:19.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.642+0000 7f40d4ff9640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40c40664d0 0x7f40c4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:19.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.642+0000 7f40d4ff9640 1 -- 172.21.15.31:0/2883987448 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f40d011e540 con 0x7f40d8156e00 2023-10-30T13:56:19.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.642+0000 7f40d7fff640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40c40664d0 0x7f40c4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:19.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.642+0000 7f40d7fff640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40c40664d0 0x7f40c4068990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f40d814ea60 tx=0x7f40c8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:19.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.646+0000 7f40d4ff9640 1 -- 172.21.15.31:0/2883987448 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40d8157b60 con 0x7f40d8156e00 2023-10-30T13:56:19.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40d8157b60 con 0x7f40d8156e00 2023-10-30T13:56:19.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40d4ff9640 1 -- 172.21.15.31:0/2883987448 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f40d8157b60 con 0x7f40d8156e00 2023-10-30T13:56:19.846 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:19.846 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:19.285229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:19.846 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 86 2023-10-30T13:56:19.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40c40664d0 msgr2=0x7f40c4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:19.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40c40664d0 0x7f40c4068990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f40d814ea60 tx=0x7f40c8009190 comp rx=0 tx=0).stop 2023-10-30T13:56:19.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d8156e00 msgr2=0x7f40d81e2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:19.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d8156e00 0x7f40d81e2650 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f40d001dbe0 tx=0x7f40c0003700 comp rx=0 tx=0).stop 2023-10-30T13:56:19.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 shutdown_connections 2023-10-30T13:56:19.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f40c40664d0 0x7f40c4068990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d8156e00 0x7f40d81e2650 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d80ca210 0x7f40d8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 --2- 172.21.15.31:0/2883987448 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f40d80c9e30 0x7f40d8157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:19.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 >> 172.21.15.31:0/2883987448 conn(0x7f40d80cac70 msgr2=0x7f40d80c6700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:19.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 shutdown_connections 2023-10-30T13:56:19.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:19.842+0000 7f40de33e640 1 -- 172.21.15.31:0/2883987448 wait complete. 2023-10-30T13:56:19.867 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:21.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:22.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f34190ea640 1 Processor -- start 2023-10-30T13:56:22.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f34190ea640 1 -- start start 2023-10-30T13:56:22.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f34190ea640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3414153c80 0x7f341415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:22.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f34190ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c42a0 0x7f34140ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:22.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f34190ea640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3414159f20 con 0x7f34140c4a10 2023-10-30T13:56:22.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f34190ea640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341415dec0 con 0x7f3414153c80 2023-10-30T13:56:22.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f34190ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34141599a0 con 0x7f34140c42a0 2023-10-30T13:56:22.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3412d9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3414153c80 0x7f341415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:22.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3412d9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3414153c80 0x7f341415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58114/0 (socket says 172.21.15.31:58114) 2023-10-30T13:56:22.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f341259c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f34140c4a10 0x7f341415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:35954/0 (socket says 172.21.15.31:35954) 2023-10-30T13:56:22.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3412d9d640 1 -- 172.21.15.31:0/1445958734 learned_addr learned my addr 172.21.15.31:0/1445958734 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:22.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f341359e640 1 --2- 172.21.15.31:0/1445958734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c42a0 0x7f34140ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:22.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3411d9b640 1 -- 172.21.15.31:0/1445958734 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961611599 0 0) 0x7f3414159f20 con 0x7f34140c4a10 2023-10-30T13:56:22.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3411d9b640 1 -- 172.21.15.31:0/1445958734 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc003530 con 0x7f34140c4a10 2023-10-30T13:56:22.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3412d9d640 1 -- 172.21.15.31:0/1445958734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c42a0 msgr2=0x7f34140ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:22.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3412d9d640 1 --2- 172.21.15.31:0/1445958734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c42a0 0x7f34140ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3412d9d640 1 -- 172.21.15.31:0/1445958734 >> v1:172.21.15.31:6789/0 conn(0x7f34140c4a10 legacy=0x7f341415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:22.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f3412d9d640 1 -- 172.21.15.31:0/1445958734 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341415d620 con 0x7f3414153c80 2023-10-30T13:56:22.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.018+0000 7f341359e640 1 --2- 172.21.15.31:0/1445958734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c42a0 0x7f34140ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:22.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f3412d9d640 1 --2- 172.21.15.31:0/1445958734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f341415cf90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3404009870 tx=0x7f340401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35d7423ab7481ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f3411d9b640 1 -- 172.21.15.31:0/1445958734 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34040a8a30 con 0x7f3414153c80 2023-10-30T13:56:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f3411d9b640 1 -- 172.21.15.31:0/1445958734 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34040a8b90 con 0x7f3414153c80 2023-10-30T13:56:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f3411d9b640 1 -- 172.21.15.31:0/1445958734 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34040b0440 con 0x7f3414153c80 2023-10-30T13:56:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 -- 172.21.15.31:0/1445958734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 msgr2=0x7f341415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 --2- 172.21.15.31:0/1445958734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f341415cf90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3404009870 tx=0x7f340401ed30 comp rx=0 tx=0).stop 2023-10-30T13:56:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 -- 172.21.15.31:0/1445958734 shutdown_connections 2023-10-30T13:56:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 --2- 172.21.15.31:0/1445958734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c42a0 0x7f34140ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 --2- 172.21.15.31:0/1445958734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f341415cf90 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 -- 172.21.15.31:0/1445958734 >> 172.21.15.31:0/1445958734 conn(0x7f34140715c0 msgr2=0x7f34140719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:22.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 -- 172.21.15.31:0/1445958734 shutdown_connections 2023-10-30T13:56:22.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.022+0000 7f34190ea640 1 -- 172.21.15.31:0/1445958734 wait complete. 2023-10-30T13:56:22.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 Processor -- start 2023-10-30T13:56:22.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 -- start start 2023-10-30T13:56:22.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34140c42a0 0x7f34140ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:22.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c4a10 0x7f34141f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:22.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f34141f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:22.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34141ef380 con 0x7f3414153c80 2023-10-30T13:56:22.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34141ee7d0 con 0x7f34140c42a0 2023-10-30T13:56:22.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341415afa0 con 0x7f34140c4a10 2023-10-30T13:56:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f34141f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f34141f22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58120/0 (socket says 172.21.15.31:58120) 2023-10-30T13:56:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 -- 172.21.15.31:0/1231757339 learned_addr learned my addr 172.21.15.31:0/1231757339 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341259c640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c4a10 0x7f34141f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f3412d9d640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34140c42a0 0x7f34140ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:22.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 -- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c4a10 msgr2=0x7f34141f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:22.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c4a10 0x7f34141f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 -- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34140c42a0 msgr2=0x7f34140ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:22.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34140c42a0 0x7f34140ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 -- 172.21.15.31:0/1231757339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34040094d0 con 0x7f3414153c80 2023-10-30T13:56:22.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f341359e640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f34141f22c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f340c007f20 tx=0x7f340c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:22.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f33fb7fe640 1 -- 172.21.15.31:0/1231757339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c00dee0 con 0x7f3414153c80 2023-10-30T13:56:22.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f33fb7fe640 1 -- 172.21.15.31:0/1231757339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c00f040 con 0x7f3414153c80 2023-10-30T13:56:22.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34141eea50 con 0x7f3414153c80 2023-10-30T13:56:22.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f33fb7fe640 1 -- 172.21.15.31:0/1231757339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c003de0 con 0x7f3414153c80 2023-10-30T13:56:22.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.026+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34141ef010 con 0x7f3414153c80 2023-10-30T13:56:22.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.030+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34141ef1d0 con 0x7f3414153c80 2023-10-30T13:56:22.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.030+0000 7f33fb7fe640 1 -- 172.21.15.31:0/1231757339 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f340c004030 con 0x7f3414153c80 2023-10-30T13:56:22.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.030+0000 7f33fb7fe640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34000664a0 0x7f3400068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:22.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.030+0000 7f33fb7fe640 1 -- 172.21.15.31:0/1231757339 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f340c084710 con 0x7f3414153c80 2023-10-30T13:56:22.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.030+0000 7f3412d9d640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34000664a0 0x7f3400068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:22.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.030+0000 7f3412d9d640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34000664a0 0x7f3400068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f34040b2000 tx=0x7f3404008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:22.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.034+0000 7f33fb7fe640 1 -- 172.21.15.31:0/1231757339 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f340c051680 con 0x7f3414153c80 2023-10-30T13:56:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.226+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f341404e170 con 0x7f3414153c80 2023-10-30T13:56:22.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.226+0000 7f33fb7fe640 1 -- 172.21.15.31:0/1231757339 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f340c054ca0 con 0x7f3414153c80 2023-10-30T13:56:22.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:22.232 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:19.285229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:22.232 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 86 2023-10-30T13:56:22.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34000664a0 msgr2=0x7f3400068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:22.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34000664a0 0x7f3400068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f34040b2000 tx=0x7f3404008040 comp rx=0 tx=0).stop 2023-10-30T13:56:22.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 msgr2=0x7f34141f22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:22.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f34141f22c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f340c007f20 tx=0x7f340c00d560 comp rx=0 tx=0).stop 2023-10-30T13:56:22.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 shutdown_connections 2023-10-30T13:56:22.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f34000664a0 0x7f3400068960 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3414153c80 0x7f34141f22c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f34140c4a10 0x7f34141f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 --2- 172.21.15.31:0/1231757339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34140c42a0 0x7f34140ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:22.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 >> 172.21.15.31:0/1231757339 conn(0x7f34140715c0 msgr2=0x7f34140c5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:22.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 shutdown_connections 2023-10-30T13:56:22.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:22.230+0000 7f34190ea640 1 -- 172.21.15.31:0/1231757339 wait complete. 2023-10-30T13:56:22.252 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:24.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:24.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.406+0000 7fa908691640 1 Processor -- start 2023-10-30T13:56:24.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.406+0000 7fa908691640 1 -- start start 2023-10-30T13:56:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.406+0000 7fa908691640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa900153c10 0x7fa900065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.406+0000 7fa908691640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa900159970 0x7fa900159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.406+0000 7fa908691640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90006e390 con 0x7fa900070c30 2023-10-30T13:56:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.406+0000 7fa908691640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa900072390 con 0x7fa900153c10 2023-10-30T13:56:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.406+0000 7fa908691640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9000660e0 con 0x7fa900159970 2023-10-30T13:56:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa900153c10 0x7fa900065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa900153c10 0x7fa900065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58130/0 (socket says 172.21.15.31:58130) 2023-10-30T13:56:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 -- 172.21.15.31:0/900288302 learned_addr learned my addr 172.21.15.31:0/900288302 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa906c2e640 1 --2- 172.21.15.31:0/900288302 >> v2:172.21.15.33:3300/0 conn(0x7fa900159970 0x7fa900159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:24.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 -- 172.21.15.31:0/900288302 >> v2:172.21.15.33:3300/0 conn(0x7fa900159970 msgr2=0x7fa900159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:24.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 --2- 172.21.15.31:0/900288302 >> v2:172.21.15.33:3300/0 conn(0x7fa900159970 0x7fa900159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 -- 172.21.15.31:0/900288302 >> v1:172.21.15.31:6789/0 conn(0x7fa900070c30 legacy=0x7fa900071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:24.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 -- 172.21.15.31:0/900288302 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9000bec10 con 0x7fa900153c10 2023-10-30T13:56:24.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90542b640 1 -- 172.21.15.31:0/900288302 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733348841 0 0) 0x7fa90006e390 con 0x7fa900070c30 2023-10-30T13:56:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90642d640 1 --2- 172.21.15.31:0/900288302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900153c10 0x7fa900065ba0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa90015df80 tx=0x7fa8fc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50404b34194703 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90542b640 1 -- 172.21.15.31:0/900288302 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0ab600 con 0x7fa900153c10 2023-10-30T13:56:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90542b640 1 -- 172.21.15.31:0/900288302 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8fc0a7050 con 0x7fa900153c10 2023-10-30T13:56:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.410+0000 7fa90542b640 1 -- 172.21.15.31:0/900288302 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0ab980 con 0x7fa900153c10 2023-10-30T13:56:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- 172.21.15.31:0/900288302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900153c10 msgr2=0x7fa900065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 --2- 172.21.15.31:0/900288302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900153c10 0x7fa900065ba0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa90015df80 tx=0x7fa8fc01c470 comp rx=0 tx=0).stop 2023-10-30T13:56:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- 172.21.15.31:0/900288302 shutdown_connections 2023-10-30T13:56:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 --2- 172.21.15.31:0/900288302 >> v2:172.21.15.33:3300/0 conn(0x7fa900159970 0x7fa900159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 --2- 172.21.15.31:0/900288302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900153c10 0x7fa900065ba0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- 172.21.15.31:0/900288302 >> 172.21.15.31:0/900288302 conn(0x7fa9000c7f60 msgr2=0x7fa9000c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- 172.21.15.31:0/900288302 shutdown_connections 2023-10-30T13:56:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- 172.21.15.31:0/900288302 wait complete. 2023-10-30T13:56:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 Processor -- start 2023-10-30T13:56:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- start start 2023-10-30T13:56:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa900070c30 0x7fa900143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa900153c10 0x7fa9001587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900159970 0x7fa90014e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001437c0 con 0x7fa900159970 2023-10-30T13:56:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001591c0 con 0x7fa900153c10 2023-10-30T13:56:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa908691640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001f2170 con 0x7fa900070c30 2023-10-30T13:56:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900159970 0x7fa90014e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900159970 0x7fa90014e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58144/0 (socket says 172.21.15.31:58144) 2023-10-30T13:56:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 -- 172.21.15.31:0/2554369165 learned_addr learned my addr 172.21.15.31:0/2554369165 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa90642d640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa900070c30 0x7fa900143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa905c2c640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa900153c10 0x7fa9001587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 -- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa900070c30 msgr2=0x7fa900143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa900070c30 0x7fa900143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 -- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa900153c10 msgr2=0x7fa9001587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:24.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa900153c10 0x7fa9001587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa906c2e640 1 -- 172.21.15.31:0/2554369165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0094d0 con 0x7fa900159970 2023-10-30T13:56:24.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.414+0000 7fa905c2c640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa900153c10 0x7fa9001587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:24.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa906c2e640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900159970 0x7fa90014e560 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8007e60 tx=0x7fa8f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:24.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa8f6ffd640 1 -- 172.21.15.31:0/2554369165 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8014420 con 0x7fa900159970 2023-10-30T13:56:24.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa8f6ffd640 1 -- 172.21.15.31:0/2554369165 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f80145a0 con 0x7fa900159970 2023-10-30T13:56:24.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9001f2450 con 0x7fa900159970 2023-10-30T13:56:24.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa8f6ffd640 1 -- 172.21.15.31:0/2554369165 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8014950 con 0x7fa900159970 2023-10-30T13:56:24.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9001f2a10 con 0x7fa900159970 2023-10-30T13:56:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa900159300 con 0x7fa900159970 2023-10-30T13:56:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa8f6ffd640 1 -- 172.21.15.31:0/2554369165 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8f8014ab0 con 0x7fa900159970 2023-10-30T13:56:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa8f6ffd640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa8e40664a0 0x7fa8e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa8f6ffd640 1 -- 172.21.15.31:0/2554369165 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa8f8084c40 con 0x7fa900159970 2023-10-30T13:56:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.418+0000 7fa90642d640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa8e40664a0 0x7fa8e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.422+0000 7fa90642d640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa8e40664a0 0x7fa8e4068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0ad000 tx=0x7fa8fc0a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:24.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.422+0000 7fa8f6ffd640 1 -- 172.21.15.31:0/2554369165 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8f8051ba0 con 0x7fa900159970 2023-10-30T13:56:24.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.610+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa90014d350 con 0x7fa900159970 2023-10-30T13:56:24.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa8f6ffd640 1 -- 172.21.15.31:0/2554369165 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fa90014d350 con 0x7fa900159970 2023-10-30T13:56:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:24.617 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:19.285229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 86 2023-10-30T13:56:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa8e40664a0 msgr2=0x7fa8e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa8e40664a0 0x7fa8e4068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0ad000 tx=0x7fa8fc0a6000 comp rx=0 tx=0).stop 2023-10-30T13:56:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900159970 msgr2=0x7fa90014e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900159970 0x7fa90014e560 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8007e60 tx=0x7fa8f800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 shutdown_connections 2023-10-30T13:56:24.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa8e40664a0 0x7fa8e4068960 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa900159970 0x7fa90014e560 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa900153c10 0x7fa9001587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 --2- 172.21.15.31:0/2554369165 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa900070c30 0x7fa900143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 >> 172.21.15.31:0/2554369165 conn(0x7fa9000c7f60 msgr2=0x7fa9000bd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:24.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 shutdown_connections 2023-10-30T13:56:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:24.614+0000 7fa908691640 1 -- 172.21.15.31:0/2554369165 wait complete. 2023-10-30T13:56:24.638 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:26.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2fbbc4640 1 Processor -- start 2023-10-30T13:56:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2fbbc4640 1 -- start start 2023-10-30T13:56:26.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2fbbc4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7ff2f40c5ba0 0x7ff2f40c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:26.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2fbbc4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff2f40c5f80 0x7ff2f4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:26.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2fbbc4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f415cda0 con 0x7ff2f40c9b40 2023-10-30T13:56:26.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2fbbc4640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f415d990 con 0x7ff2f40c5f80 2023-10-30T13:56:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2fbbc4640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f4153f60 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f915f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff2f40c5f80 0x7ff2f4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f915f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7ff2f40c5f80 0x7ff2f4069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37738/0 (socket says 172.21.15.31:37738) 2023-10-30T13:56:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f915f640 1 -- 172.21.15.31:0/3041947546 learned_addr learned my addr 172.21.15.31:0/3041947546 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f9960640 1 --2- 172.21.15.31:0/3041947546 >> v2:172.21.15.33:3300/0 conn(0x7ff2f40c5ba0 0x7ff2f40c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:26.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f915f640 1 -- 172.21.15.31:0/3041947546 >> v2:172.21.15.33:3300/0 conn(0x7ff2f40c5ba0 msgr2=0x7ff2f40c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:26.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f915f640 1 --2- 172.21.15.31:0/3041947546 >> v2:172.21.15.33:3300/0 conn(0x7ff2f40c5ba0 0x7ff2f40c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:26.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f915f640 1 -- 172.21.15.31:0/3041947546 >> v1:172.21.15.33:6789/0 conn(0x7ff2f40c9b40 legacy=0x7ff2f41597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:26.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f915f640 1 -- 172.21.15.31:0/3041947546 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f406f490 con 0x7ff2f40c5f80 2023-10-30T13:56:26.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.950+0000 7ff2f9960640 1 --2- 172.21.15.31:0/3041947546 >> v2:172.21.15.33:3300/0 conn(0x7ff2f40c5ba0 0x7ff2f40c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:26.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2f915f640 1 --2- 172.21.15.31:0/3041947546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5f80 0x7ff2f4069e90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40c8740 tx=0x7ff2e401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8bfd562d1ade77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:26.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2f895e640 1 -- 172.21.15.31:0/3041947546 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40ab600 con 0x7ff2f40c5f80 2023-10-30T13:56:26.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2f895e640 1 -- 172.21.15.31:0/3041947546 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e40a7050 con 0x7ff2f40c5f80 2023-10-30T13:56:26.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2f895e640 1 -- 172.21.15.31:0/3041947546 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40ab980 con 0x7ff2f40c5f80 2023-10-30T13:56:26.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3041947546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5f80 msgr2=0x7ff2f4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:26.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3041947546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5f80 0x7ff2f4069e90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40c8740 tx=0x7ff2e401c500 comp rx=0 tx=0).stop 2023-10-30T13:56:26.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3041947546 shutdown_connections 2023-10-30T13:56:26.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3041947546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5f80 0x7ff2f4069e90 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:26.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3041947546 >> v2:172.21.15.33:3300/0 conn(0x7ff2f40c5ba0 0x7ff2f40c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:26.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3041947546 >> 172.21.15.31:0/3041947546 conn(0x7ff2f406afa0 msgr2=0x7ff2f406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:26.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3041947546 shutdown_connections 2023-10-30T13:56:26.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.954+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3041947546 wait complete. 2023-10-30T13:56:26.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 Processor -- start 2023-10-30T13:56:26.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 -- start start 2023-10-30T13:56:26.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5ba0 0x7ff2f414e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:26.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2f40c5f80 0x7ff2f41f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2f40c9b40 0x7ff2f415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4072d30 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f414e960 con 0x7ff2f40c9b40 2023-10-30T13:56:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40bf890 con 0x7ff2f40c5f80 2023-10-30T13:56:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5ba0 0x7ff2f414e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5ba0 0x7ff2f414e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37754/0 (socket says 172.21.15.31:37754) 2023-10-30T13:56:26.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 -- 172.21.15.31:0/3282335982 learned_addr learned my addr 172.21.15.31:0/3282335982 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:26.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f915f640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2f40c5f80 0x7ff2f41f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:26.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fa161640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2f40c9b40 0x7ff2f415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:26.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 -- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2f40c5f80 msgr2=0x7ff2f41f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:26.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2f40c5f80 0x7ff2f41f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:26.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 -- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2f40c9b40 msgr2=0x7ff2f415bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:26.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2f40c9b40 0x7ff2f415bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:26.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 -- 172.21.15.31:0/3282335982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40094d0 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f915f640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2f40c5f80 0x7ff2f41f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:56:26.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fa161640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2f40c9b40 0x7ff2f415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:56:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2f9960640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5ba0 0x7ff2f414e420 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000e810 tx=0x7ff2f000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2ea7fc640 1 -- 172.21.15.31:0/3282335982 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000c8d0 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2ea7fc640 1 -- 172.21.15.31:0/3282335982 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f000ca30 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f41f1b90 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2ea7fc640 1 -- 172.21.15.31:0/3282335982 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000cdc0 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.958+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f40b14c0 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.962+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f404e170 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.962+0000 7ff2ea7fc640 1 -- 172.21.15.31:0/3282335982 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff2f0013410 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.962+0000 7ff2ea7fc640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2d80664a0 0x7ff2d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:26.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.962+0000 7ff2ea7fc640 1 -- 172.21.15.31:0/3282335982 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7ff2f0084080 con 0x7ff2f40c5ba0 2023-10-30T13:56:26.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.962+0000 7ff2f915f640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2d80664a0 0x7ff2d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:26.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.966+0000 7ff2f915f640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2d80664a0 0x7ff2d8068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40ad000 tx=0x7ff2e4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:26.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:26.966+0000 7ff2ea7fc640 1 -- 172.21.15.31:0/3282335982 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2f0051100 con 0x7ff2f40c5ba0 2023-10-30T13:56:27.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2f40c9360 con 0x7ff2f40c5ba0 2023-10-30T13:56:27.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2ea7fc640 1 -- 172.21.15.31:0/3282335982 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5708 (secure 0 0 0) 0x7ff2f0054720 con 0x7ff2f40c5ba0 2023-10-30T13:56:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:27.162 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:24.444302+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:27.163 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2023-10-30T13:56:27.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2d80664a0 msgr2=0x7ff2d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2d80664a0 0x7ff2d8068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40ad000 tx=0x7ff2e4008040 comp rx=0 tx=0).stop 2023-10-30T13:56:27.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5ba0 msgr2=0x7ff2f414e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5ba0 0x7ff2f414e420 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000e810 tx=0x7ff2f000ecd0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 shutdown_connections 2023-10-30T13:56:27.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7ff2d80664a0 0x7ff2d8068960 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2f40c9b40 0x7ff2f415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7ff2f40c5f80 0x7ff2f41f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 --2- 172.21.15.31:0/3282335982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2f40c5ba0 0x7ff2f414e420 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 >> 172.21.15.31:0/3282335982 conn(0x7ff2f406afa0 msgr2=0x7ff2f40c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:27.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 shutdown_connections 2023-10-30T13:56:27.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:27.158+0000 7ff2fbbc4640 1 -- 172.21.15.31:0/3282335982 wait complete. 2023-10-30T13:56:27.184 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 87, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 87, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5813": { "addr": "172.21.15.33:6836/2281022644", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6834", "nonce": 2281022644, "type": "v2" }, { "addr": "172.21.15.33:6836", "nonce": 2281022644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5813, "incarnation": 84, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:56:24.444302+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5813 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:56:29.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:29.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:29.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:29.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.330+0000 7f43a953e640 1 Processor -- start 2023-10-30T13:56:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.330+0000 7f43a953e640 1 -- start start 2023-10-30T13:56:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a953e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a4153c80 0x7f43a415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a953e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f43a40c42a0 0x7f43a40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a953e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a4159f20 con 0x7f43a40c4a10 2023-10-30T13:56:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a953e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a415dec0 con 0x7f43a40c42a0 2023-10-30T13:56:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a953e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a41599a0 con 0x7f43a4153c80 2023-10-30T13:56:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a37fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f43a40c42a0 0x7f43a40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f439bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f43a40c4a10 0x7f43a415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:45580/0 (socket says 172.21.15.31:45580) 2023-10-30T13:56:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f439bfff640 1 -- 172.21.15.31:0/2076232048 learned_addr learned my addr 172.21.15.31:0/2076232048 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a2ffd640 1 --2- 172.21.15.31:0/2076232048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a4153c80 0x7f43a415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a37fe640 1 -- 172.21.15.31:0/2076232048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a4153c80 msgr2=0x7f43a415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a37fe640 1 --2- 172.21.15.31:0/2076232048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a4153c80 0x7f43a415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a37fe640 1 -- 172.21.15.31:0/2076232048 >> v1:172.21.15.31:6789/0 conn(0x7f43a40c4a10 legacy=0x7f43a415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a37fe640 1 -- 172.21.15.31:0/2076232048 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a415d620 con 0x7f43a40c42a0 2023-10-30T13:56:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.334+0000 7f43a27fc640 1 -- 172.21.15.31:0/2076232048 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608904558 0 0) 0x7f43a4159f20 con 0x7f43a40c4a10 2023-10-30T13:56:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a37fe640 1 --2- 172.21.15.31:0/2076232048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40ca1e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f439c009d90 tx=0x7f439c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd9b1d13ced00e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a27fc640 1 -- 172.21.15.31:0/2076232048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c0af600 con 0x7f43a40c42a0 2023-10-30T13:56:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a27fc640 1 -- 172.21.15.31:0/2076232048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c0ab050 con 0x7f43a40c42a0 2023-10-30T13:56:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a27fc640 1 -- 172.21.15.31:0/2076232048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c0af980 con 0x7f43a40c42a0 2023-10-30T13:56:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- 172.21.15.31:0/2076232048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 msgr2=0x7f43a40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 --2- 172.21.15.31:0/2076232048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40ca1e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f439c009d90 tx=0x7f439c01f130 comp rx=0 tx=0).stop 2023-10-30T13:56:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- 172.21.15.31:0/2076232048 shutdown_connections 2023-10-30T13:56:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 --2- 172.21.15.31:0/2076232048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40ca1e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 --2- 172.21.15.31:0/2076232048 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a4153c80 0x7f43a415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- 172.21.15.31:0/2076232048 >> 172.21.15.31:0/2076232048 conn(0x7f43a40715c0 msgr2=0x7f43a40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- 172.21.15.31:0/2076232048 shutdown_connections 2023-10-30T13:56:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- 172.21.15.31:0/2076232048 wait complete. 2023-10-30T13:56:29.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 Processor -- start 2023-10-30T13:56:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- start start 2023-10-30T13:56:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a40c4a10 0x7f43a415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43a4153c80 0x7f43a41f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c7720 con 0x7f43a40c42a0 2023-10-30T13:56:29.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a415b2e0 con 0x7f43a4153c80 2023-10-30T13:56:29.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.338+0000 7f43a953e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40cef80 con 0x7f43a40c4a10 2023-10-30T13:56:29.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37762/0 (socket says 172.21.15.31:37762) 2023-10-30T13:56:29.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 -- 172.21.15.31:0/577748584 learned_addr learned my addr 172.21.15.31:0/577748584 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:29.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f439bfff640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a40c4a10 0x7f43a415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a37fe640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43a4153c80 0x7f43a41f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 -- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a40c4a10 msgr2=0x7f43a415b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a40c4a10 0x7f43a415b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 -- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43a4153c80 msgr2=0x7f43a41f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43a4153c80 0x7f43a41f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 -- 172.21.15.31:0/577748584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c009a70 con 0x7f43a40c42a0 2023-10-30T13:56:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a2ffd640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40cd2a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f439400d5f0 tx=0x7f439400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f439b7fe640 1 -- 172.21.15.31:0/577748584 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43940125f0 con 0x7f43a40c42a0 2023-10-30T13:56:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f439b7fe640 1 -- 172.21.15.31:0/577748584 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4394012750 con 0x7f43a40c42a0 2023-10-30T13:56:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f439b7fe640 1 -- 172.21.15.31:0/577748584 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394012ae0 con 0x7f43a40c42a0 2023-10-30T13:56:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a40ce9e0 con 0x7f43a40c42a0 2023-10-30T13:56:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a415ac90 con 0x7f43a40c42a0 2023-10-30T13:56:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a404e170 con 0x7f43a40c42a0 2023-10-30T13:56:29.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f439b7fe640 1 -- 172.21.15.31:0/577748584 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4394012c40 con 0x7f43a40c42a0 2023-10-30T13:56:29.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.342+0000 7f439b7fe640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f43840664d0 0x7f4384068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.346+0000 7f439b7fe640 1 -- 172.21.15.31:0/577748584 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f4394084750 con 0x7f43a40c42a0 2023-10-30T13:56:29.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.346+0000 7f439bfff640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f43840664d0 0x7f4384068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.346+0000 7f439bfff640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f43840664d0 0x7f4384068990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43a4157680 tx=0x7f438c007b60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:29.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.350+0000 7f439b7fe640 1 -- 172.21.15.31:0/577748584 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43940517e0 con 0x7f43a40c42a0 2023-10-30T13:56:29.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.538+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43a40cad40 con 0x7f43a40c42a0 2023-10-30T13:56:29.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.538+0000 7f439b7fe640 1 -- 172.21.15.31:0/577748584 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5708 (secure 0 0 0) 0x7f4394004650 con 0x7f43a40c42a0 2023-10-30T13:56:29.545 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:29.545 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:24.444302+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:29.546 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2023-10-30T13:56:29.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f43840664d0 msgr2=0x7f4384068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f43840664d0 0x7f4384068990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43a4157680 tx=0x7f438c007b60 comp rx=0 tx=0).stop 2023-10-30T13:56:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 msgr2=0x7f43a40cd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40cd2a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f439400d5f0 tx=0x7f439400dab0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 shutdown_connections 2023-10-30T13:56:29.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f43840664d0 0x7f4384068990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43a4153c80 0x7f43a41f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f43a40c4a10 0x7f43a415b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 --2- 172.21.15.31:0/577748584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43a40c42a0 0x7f43a40cd2a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 >> 172.21.15.31:0/577748584 conn(0x7f43a40715c0 msgr2=0x7f43a40c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 shutdown_connections 2023-10-30T13:56:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:29.542+0000 7f43a953e640 1 -- 172.21.15.31:0/577748584 wait complete. 2023-10-30T13:56:29.566 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:31.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.714+0000 7f6d6d86b640 1 Processor -- start 2023-10-30T13:56:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.714+0000 7f6d6d86b640 1 -- start start 2023-10-30T13:56:31.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.714+0000 7f6d6d86b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6d681537f0 0x7f6d680701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:31.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.714+0000 7f6d6d86b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6806b2f0 0x7f6d6806b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:31.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.714+0000 7f6d6d86b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680c2aa0 con 0x7f6d6815a2e0 2023-10-30T13:56:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.714+0000 7f6d6d86b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d680c3ab0 con 0x7f6d681537f0 2023-10-30T13:56:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.714+0000 7f6d6d86b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680706e0 con 0x7f6d6806b2f0 2023-10-30T13:56:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6d681537f0 0x7f6d680701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6d681537f0 0x7f6d680701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37766/0 (socket says 172.21.15.31:37766) 2023-10-30T13:56:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 -- 172.21.15.31:0/3807381765 learned_addr learned my addr 172.21.15.31:0/3807381765 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d677fe640 1 --2- 172.21.15.31:0/3807381765 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6806b2f0 0x7f6d6806b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 -- 172.21.15.31:0/3807381765 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6806b2f0 msgr2=0x7f6d6806b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 --2- 172.21.15.31:0/3807381765 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6806b2f0 0x7f6d6806b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 -- 172.21.15.31:0/3807381765 >> v1:172.21.15.33:6789/0 conn(0x7f6d6815a2e0 legacy=0x7f6d680ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 -- 172.21.15.31:0/3807381765 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6806bc80 con 0x7f6d681537f0 2023-10-30T13:56:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d66ffd640 1 --2- 172.21.15.31:0/3807381765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d680701a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009870 tx=0x7f6d5801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920d98d22d24040f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d667fc640 1 -- 172.21.15.31:0/3807381765 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d580a8a30 con 0x7f6d681537f0 2023-10-30T13:56:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d667fc640 1 -- 172.21.15.31:0/3807381765 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d580a8b90 con 0x7f6d681537f0 2023-10-30T13:56:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.718+0000 7f6d667fc640 1 -- 172.21.15.31:0/3807381765 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d580b0440 con 0x7f6d681537f0 2023-10-30T13:56:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- 172.21.15.31:0/3807381765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 msgr2=0x7f6d680701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/3807381765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d680701a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009870 tx=0x7f6d5801ed30 comp rx=0 tx=0).stop 2023-10-30T13:56:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- 172.21.15.31:0/3807381765 shutdown_connections 2023-10-30T13:56:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/3807381765 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6806b2f0 0x7f6d6806b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/3807381765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d680701a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- 172.21.15.31:0/3807381765 >> 172.21.15.31:0/3807381765 conn(0x7f6d68071fb0 msgr2=0x7f6d680723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- 172.21.15.31:0/3807381765 shutdown_connections 2023-10-30T13:56:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- 172.21.15.31:0/3807381765 wait complete. 2023-10-30T13:56:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 Processor -- start 2023-10-30T13:56:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- start start 2023-10-30T13:56:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d6806b2f0 0x7f6d680c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d6815b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6815a2e0 0x7f6d680cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6815b3e0 con 0x7f6d681537f0 2023-10-30T13:56:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6815bc00 con 0x7f6d6806b2f0 2023-10-30T13:56:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d6d86b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d681f1fc0 con 0x7f6d6815a2e0 2023-10-30T13:56:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d6815b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d6815b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37782/0 (socket says 172.21.15.31:37782) 2023-10-30T13:56:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 -- 172.21.15.31:0/370728967 learned_addr learned my addr 172.21.15.31:0/370728967 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d677fe640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6815a2e0 0x7f6d680cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d66ffd640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d6806b2f0 0x7f6d680c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 -- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6815a2e0 msgr2=0x7f6d680cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6815a2e0 0x7f6d680cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 -- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d6806b2f0 msgr2=0x7f6d680c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d6806b2f0 0x7f6d680c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.722+0000 7f6d5ffff640 1 -- 172.21.15.31:0/370728967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d580094d0 con 0x7f6d681537f0 2023-10-30T13:56:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d5ffff640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d6815b6c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000c500 tx=0x7f6d5000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d64ff9640 1 -- 172.21.15.31:0/370728967 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50005220 con 0x7f6d681537f0 2023-10-30T13:56:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d64ff9640 1 -- 172.21.15.31:0/370728967 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d50005380 con 0x7f6d681537f0 2023-10-30T13:56:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d64ff9640 1 -- 172.21.15.31:0/370728967 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50007270 con 0x7f6d681537f0 2023-10-30T13:56:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d681f22a0 con 0x7f6d681537f0 2023-10-30T13:56:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d681f2860 con 0x7f6d681537f0 2023-10-30T13:56:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6804e170 con 0x7f6d681537f0 2023-10-30T13:56:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.726+0000 7f6d64ff9640 1 -- 172.21.15.31:0/370728967 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d500073d0 con 0x7f6d681537f0 2023-10-30T13:56:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.730+0000 7f6d64ff9640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6d480664a0 0x7f6d48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.730+0000 7f6d64ff9640 1 -- 172.21.15.31:0/370728967 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f6d50083c70 con 0x7f6d681537f0 2023-10-30T13:56:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.730+0000 7f6d66ffd640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6d480664a0 0x7f6d48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.730+0000 7f6d66ffd640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6d480664a0 0x7f6d48068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6d580b2000 tx=0x7f6d58008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.734+0000 7f6d64ff9640 1 -- 172.21.15.31:0/370728967 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d50050c70 con 0x7f6d681537f0 2023-10-30T13:56:31.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.926+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d680ca6d0 con 0x7f6d681537f0 2023-10-30T13:56:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.926+0000 7f6d64ff9640 1 -- 172.21.15.31:0/370728967 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f6d50054290 con 0x7f6d681537f0 2023-10-30T13:56:31.932 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:31.932 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:31.932 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 88 2023-10-30T13:56:31.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6d480664a0 msgr2=0x7f6d48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:31.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6d480664a0 0x7f6d48068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6d580b2000 tx=0x7f6d58008040 comp rx=0 tx=0).stop 2023-10-30T13:56:31.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 msgr2=0x7f6d6815b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:31.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d6815b6c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000c500 tx=0x7f6d5000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 shutdown_connections 2023-10-30T13:56:31.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6d480664a0 0x7f6d48068960 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6d6815a2e0 0x7f6d680cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d681537f0 0x7f6d6815b6c0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 --2- 172.21.15.31:0/370728967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d6806b2f0 0x7f6d680c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:31.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 >> 172.21.15.31:0/370728967 conn(0x7f6d68071fb0 msgr2=0x7f6d6814da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:31.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 shutdown_connections 2023-10-30T13:56:31.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:31.930+0000 7f6d6d86b640 1 -- 172.21.15.31:0/370728967 wait complete. 2023-10-30T13:56:31.953 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=6201 2023-10-30T13:56:31.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2023-10-30T13:56:38.954 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:56:38.954 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:56:38.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:56:38.959 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:56:38.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:39.055 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2023-10-30T13:56:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f2177c46640 1 Processor -- start 2023-10-30T13:56:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f2177c46640 1 -- start start 2023-10-30T13:56:39.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f2177c46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f2170071340 0x7f2170065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:39.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f2177c46640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217006e390 con 0x7f2170153c10 2023-10-30T13:56:39.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f2177c46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21700660e0 con 0x7f2170159970 2023-10-30T13:56:39.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f2177c46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2170072390 con 0x7f2170071340 2023-10-30T13:56:39.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f21759e2640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2170153c10 0x7f2170070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:56968/0 (socket says 172.21.15.31:56968) 2023-10-30T13:56:39.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f21759e2640 1 -- 172.21.15.31:0/231106788 learned_addr learned my addr 172.21.15.31:0/231106788 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:56:39.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.110+0000 7f21751e1640 1 --2- 172.21.15.31:0/231106788 >> v2:172.21.15.33:3300/0 conn(0x7f2170071340 0x7f2170065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:39.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.114+0000 7f21749e0640 1 -- 172.21.15.31:0/231106788 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183277678 0 0) 0x7f217006e390 con 0x7f2170153c10 2023-10-30T13:56:39.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.114+0000 7f21749e0640 1 -- 172.21.15.31:0/231106788 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f215c003530 con 0x7f2170153c10 2023-10-30T13:56:39.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.114+0000 7f21751e1640 1 -- 172.21.15.31:0/231106788 >> v1:172.21.15.33:6789/0 conn(0x7f2170159970 legacy=0x7f2170159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:39.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.114+0000 7f21751e1640 1 -- 172.21.15.31:0/231106788 >> v1:172.21.15.31:6789/0 conn(0x7f2170153c10 legacy=0x7f2170070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:39.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.114+0000 7f21751e1640 1 -- 172.21.15.31:0/231106788 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21700bec10 con 0x7f2170071340 2023-10-30T13:56:39.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.114+0000 7f21751e1640 1 --2- 172.21.15.31:0/231106788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170071340 0x7f2170065ba0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f21600097c0 tx=0x7f216001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=792473ababbc383a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:39.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.114+0000 7f21749e0640 1 -- 172.21.15.31:0/231106788 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600ab600 con 0x7f2170071340 2023-10-30T13:56:39.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f21749e0640 1 -- 172.21.15.31:0/231106788 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21600a7050 con 0x7f2170071340 2023-10-30T13:56:39.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f21749e0640 1 -- 172.21.15.31:0/231106788 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600ab960 con 0x7f2170071340 2023-10-30T13:56:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 -- 172.21.15.31:0/231106788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170071340 msgr2=0x7f2170065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 --2- 172.21.15.31:0/231106788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170071340 0x7f2170065ba0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f21600097c0 tx=0x7f216001c3f0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 -- 172.21.15.31:0/231106788 shutdown_connections 2023-10-30T13:56:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 --2- 172.21.15.31:0/231106788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170071340 0x7f2170065ba0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 -- 172.21.15.31:0/231106788 >> 172.21.15.31:0/231106788 conn(0x7f21700c7f60 msgr2=0x7f21700c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 -- 172.21.15.31:0/231106788 shutdown_connections 2023-10-30T13:56:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 -- 172.21.15.31:0/231106788 wait complete. 2023-10-30T13:56:39.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 Processor -- start 2023-10-30T13:56:39.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.118+0000 7f2177c46640 1 -- start start 2023-10-30T13:56:39.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2170071340 0x7f217014a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170153c10 0x7f21700ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2170159970 0x7f217013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2170148f30 con 0x7f2170159970 2023-10-30T13:56:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217014ad00 con 0x7f2170153c10 2023-10-30T13:56:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21701f51f0 con 0x7f2170071340 2023-10-30T13:56:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2170159970 0x7f217013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2170159970 0x7f217013edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60930/0 (socket says 172.21.15.31:60930) 2023-10-30T13:56:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 -- 172.21.15.31:0/2689924963 learned_addr learned my addr 172.21.15.31:0/2689924963 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:56:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21759e2640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2170071340 0x7f217014a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21751e1640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170153c10 0x7f21700ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:39.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 -- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2170071340 msgr2=0x7f217014a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:39.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2170071340 0x7f217014a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 -- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170153c10 msgr2=0x7f21700ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:39.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170153c10 0x7f21700ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 -- 172.21.15.31:0/2689924963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2160009420 con 0x7f2170159970 2023-10-30T13:56:39.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21761e3640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2170159970 0x7f217013edb0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f216800c730 tx=0x7f216800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21667fc640 1 -- 172.21.15.31:0/2689924963 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168003b80 con 0x7f2170159970 2023-10-30T13:56:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21667fc640 1 -- 172.21.15.31:0/2689924963 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2168003ce0 con 0x7f2170159970 2023-10-30T13:56:39.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21701f54d0 con 0x7f2170159970 2023-10-30T13:56:39.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21667fc640 1 -- 172.21.15.31:0/2689924963 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216800aa30 con 0x7f2170159970 2023-10-30T13:56:39.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21701f5a90 con 0x7f2170159970 2023-10-30T13:56:39.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217004e170 con 0x7f2170159970 2023-10-30T13:56:39.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.122+0000 7f21667fc640 1 -- 172.21.15.31:0/2689924963 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f216800ab90 con 0x7f2170159970 2023-10-30T13:56:39.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.126+0000 7f21667fc640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f21540664d0 0x7f2154068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:39.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.126+0000 7f21667fc640 1 -- 172.21.15.31:0/2689924963 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f21680840e0 con 0x7f2170159970 2023-10-30T13:56:39.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.126+0000 7f21759e2640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f21540664d0 0x7f2154068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:39.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.126+0000 7f21759e2640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f21540664d0 0x7f2154068990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f216c0011f0 tx=0x7f216c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:39.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.130+0000 7f21667fc640 1 -- 172.21.15.31:0/2689924963 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f21680510d0 con 0x7f2170159970 2023-10-30T13:56:39.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.322+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21701f5c80 con 0x7f2170159970 2023-10-30T13:56:39.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.322+0000 7f21667fc640 1 -- 172.21.15.31:0/2689924963 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f21680546f0 con 0x7f2170159970 2023-10-30T13:56:39.328 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:56:39.329 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:39.329 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:56:39.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f21540664d0 msgr2=0x7f2154068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:39.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f21540664d0 0x7f2154068990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f216c0011f0 tx=0x7f216c009040 comp rx=0 tx=0).stop 2023-10-30T13:56:39.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2170159970 msgr2=0x7f217013edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:39.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2170159970 0x7f217013edb0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f216800c730 tx=0x7f216800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 shutdown_connections 2023-10-30T13:56:39.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f21540664d0 0x7f2154068990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2170159970 0x7f217013edb0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2170153c10 0x7f21700ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 --2- 172.21.15.31:0/2689924963 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2170071340 0x7f217014a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 >> 172.21.15.31:0/2689924963 conn(0x7f21700c7f60 msgr2=0x7f21700bd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 shutdown_connections 2023-10-30T13:56:39.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:56:39.326+0000 7f2177c46640 1 -- 172.21.15.31:0/2689924963 wait complete. 2023-10-30T13:56:39.349 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:56:39.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2023-10-30T13:56:40.242 INFO:tasks.ceph.mds.d.smithi033.stderr:2023-10-30T13:56:40.243+0000 7fdfc076b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:56:59.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:59.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:28.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:28.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1d14d6640 1 Processor -- start 2023-10-30T13:57:28.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1d14d6640 1 -- start start 2023-10-30T13:57:28.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1d14d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc153c80 0x7fd1cc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1d14d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd1cc0c4a10 0x7fd1cc0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1d14d6640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc159f20 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1d14d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1cc15dec0 con 0x7fd1cc0c4a10 2023-10-30T13:57:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1d14d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc1599a0 con 0x7fd1cc153c80 2023-10-30T13:57:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1c3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd1cc0c4a10 0x7fd1cc0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1cb7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fd1cc15a3b0 0x7fd1cc0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:50252/0 (socket says 172.21.15.31:50252) 2023-10-30T13:57:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1cb7fe640 1 -- 172.21.15.31:0/1862712377 learned_addr learned my addr 172.21.15.31:0/1862712377 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:57:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1caffd640 1 --2- 172.21.15.31:0/1862712377 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc153c80 0x7fd1cc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1c3fff640 1 -- 172.21.15.31:0/1862712377 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc153c80 msgr2=0x7fd1cc15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1c3fff640 1 --2- 172.21.15.31:0/1862712377 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc153c80 0x7fd1cc15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1c3fff640 1 -- 172.21.15.31:0/1862712377 >> v1:172.21.15.31:6789/0 conn(0x7fd1cc15a3b0 legacy=0x7fd1cc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1ca7fc640 1 -- 172.21.15.31:0/1862712377 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613857356 0 0) 0x7fd1cc159f20 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.505+0000 7fd1c3fff640 1 -- 172.21.15.31:0/1862712377 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0c47e0 con 0x7fd1cc0c4a10 2023-10-30T13:57:28.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1c3fff640 1 --2- 172.21.15.31:0/1862712377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc0c4a10 0x7fd1cc0c42a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40097c0 tx=0x7fd1b401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72b51217d0c50d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1ca7fc640 1 -- 172.21.15.31:0/1862712377 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40a8970 con 0x7fd1cc0c4a10 2023-10-30T13:57:28.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1ca7fc640 1 -- 172.21.15.31:0/1862712377 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b40a8ad0 con 0x7fd1cc0c4a10 2023-10-30T13:57:28.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1ca7fc640 1 -- 172.21.15.31:0/1862712377 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40a8e60 con 0x7fd1cc0c4a10 2023-10-30T13:57:28.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 -- 172.21.15.31:0/1862712377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc0c4a10 msgr2=0x7fd1cc0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/1862712377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc0c4a10 0x7fd1cc0c42a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40097c0 tx=0x7fd1b401ec70 comp rx=0 tx=0).stop 2023-10-30T13:57:28.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 -- 172.21.15.31:0/1862712377 shutdown_connections 2023-10-30T13:57:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/1862712377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc0c4a10 0x7fd1cc0c42a0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/1862712377 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc153c80 0x7fd1cc15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 -- 172.21.15.31:0/1862712377 >> 172.21.15.31:0/1862712377 conn(0x7fd1cc0715c0 msgr2=0x7fd1cc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 -- 172.21.15.31:0/1862712377 shutdown_connections 2023-10-30T13:57:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.509+0000 7fd1d14d6640 1 -- 172.21.15.31:0/1862712377 wait complete. 2023-10-30T13:57:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 Processor -- start 2023-10-30T13:57:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 -- start start 2023-10-30T13:57:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc0c4a10 0x7fd1cc0b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1cc153c80 0x7fd1cc15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc15a3b0 0x7fd1cc0ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0b5130 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc15c9c0 con 0x7fd1cc153c80 2023-10-30T13:57:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc157680 con 0x7fd1cc0c4a10 2023-10-30T13:57:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc15a3b0 0x7fd1cc0ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc15a3b0 0x7fd1cc0ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60048/0 (socket says 172.21.15.31:60048) 2023-10-30T13:57:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 -- 172.21.15.31:0/528558578 learned_addr learned my addr 172.21.15.31:0/528558578 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1c3fff640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1cc153c80 0x7fd1cc15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1caffd640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc0c4a10 0x7fd1cc0b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 -- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc0c4a10 msgr2=0x7fd1cc0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc0c4a10 0x7fd1cc0b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 -- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1cc153c80 msgr2=0x7fd1cc15ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1cc153c80 0x7fd1cc15ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 -- 172.21.15.31:0/528558578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b4009420 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1cb7fe640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc15a3b0 0x7fd1cc0ce2f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400c780 tx=0x7fd1c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1c37fe640 1 -- 172.21.15.31:0/528558578 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c400ce90 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1c37fe640 1 -- 172.21.15.31:0/528558578 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c400a640 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1c37fe640 1 -- 172.21.15.31:0/528558578 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c400a9d0 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc0b3b00 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc157940 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.513+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc04e170 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.517+0000 7fd1c37fe640 1 -- 172.21.15.31:0/528558578 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1c400ab30 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.517+0000 7fd1c37fe640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1ac0664d0 0x7fd1ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.517+0000 7fd1c37fe640 1 -- 172.21.15.31:0/528558578 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fd1c4083630 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.517+0000 7fd1caffd640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1ac0664d0 0x7fd1ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.517+0000 7fd1caffd640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1ac0664d0 0x7fd1ac068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00b9d0 tx=0x7fd1bc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.521+0000 7fd1c37fe640 1 -- 172.21.15.31:0/528558578 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1c40506b0 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.713+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1cc15cb00 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.713+0000 7fd1c37fe640 1 -- 172.21.15.31:0/528558578 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fd1c4053cd0 con 0x7fd1cc15a3b0 2023-10-30T13:57:28.716 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:28.716 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:28.717 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:57:28.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1ac0664d0 msgr2=0x7fd1ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1ac0664d0 0x7fd1ac068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00b9d0 tx=0x7fd1bc008040 comp rx=0 tx=0).stop 2023-10-30T13:57:28.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc15a3b0 msgr2=0x7fd1cc0ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc15a3b0 0x7fd1cc0ce2f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400c780 tx=0x7fd1c400cc40 comp rx=0 tx=0).stop 2023-10-30T13:57:28.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 shutdown_connections 2023-10-30T13:57:28.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd1ac0664d0 0x7fd1ac068990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1cc15a3b0 0x7fd1cc0ce2f0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1cc153c80 0x7fd1cc15ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 --2- 172.21.15.31:0/528558578 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd1cc0c4a10 0x7fd1cc0b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 >> 172.21.15.31:0/528558578 conn(0x7fd1cc0715c0 msgr2=0x7fd1cc148a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:28.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 shutdown_connections 2023-10-30T13:57:28.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.717+0000 7fd1d14d6640 1 -- 172.21.15.31:0/528558578 wait complete. 2023-10-30T13:57:28.737 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:57:28.738 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:57:28.738 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:57:28.766 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:57:28.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:28.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 Processor -- start 2023-10-30T13:57:28.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 -- start start 2023-10-30T13:57:28.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd70c153c80 0x7fd70c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c0c4a10 0x7fd70c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd70c15a3b0 0x7fd70c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70c15dec0 con 0x7fd70c15a3b0 2023-10-30T13:57:28.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70c1599a0 con 0x7fd70c153c80 2023-10-30T13:57:28.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd712017640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c159f20 con 0x7fd70c0c4a10 2023-10-30T13:57:28.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd70c15a3b0 0x7fd70c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd70c15a3b0 0x7fd70c0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60050/0 (socket says 172.21.15.31:60050) 2023-10-30T13:57:28.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 -- 172.21.15.31:0/2818720148 learned_addr learned my addr 172.21.15.31:0/2818720148 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:28.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70affd640 1 --2- 172.21.15.31:0/2818720148 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c0c4a10 0x7fd70c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70b7fe640 1 --2- 172.21.15.31:0/2818720148 >> v2:172.21.15.33:3300/0 conn(0x7fd70c153c80 0x7fd70c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 -- 172.21.15.31:0/2818720148 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c0c4a10 msgr2=0x7fd70c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 --2- 172.21.15.31:0/2818720148 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c0c4a10 0x7fd70c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 -- 172.21.15.31:0/2818720148 >> v2:172.21.15.33:3300/0 conn(0x7fd70c153c80 msgr2=0x7fd70c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 --2- 172.21.15.31:0/2818720148 >> v2:172.21.15.33:3300/0 conn(0x7fd70c153c80 0x7fd70c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.909+0000 7fd70bfff640 1 -- 172.21.15.31:0/2818720148 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c0c47e0 con 0x7fd70c15a3b0 2023-10-30T13:57:28.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd70bfff640 1 --2- 172.21.15.31:0/2818720148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c15a3b0 0x7fd70c0ca1e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0d00f0 tx=0x7fd70401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe32dfea834d902a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2818720148 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7040af400 con 0x7fd70c15a3b0 2023-10-30T13:57:28.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2818720148 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7040ab050 con 0x7fd70c15a3b0 2023-10-30T13:57:28.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd70a7fc640 1 -- 172.21.15.31:0/2818720148 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7040af780 con 0x7fd70c15a3b0 2023-10-30T13:57:28.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 -- 172.21.15.31:0/2818720148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c15a3b0 msgr2=0x7fd70c0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 --2- 172.21.15.31:0/2818720148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c15a3b0 0x7fd70c0ca1e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0d00f0 tx=0x7fd70401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 -- 172.21.15.31:0/2818720148 shutdown_connections 2023-10-30T13:57:28.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 --2- 172.21.15.31:0/2818720148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c15a3b0 0x7fd70c0ca1e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 --2- 172.21.15.31:0/2818720148 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c0c4a10 0x7fd70c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 --2- 172.21.15.31:0/2818720148 >> v2:172.21.15.33:3300/0 conn(0x7fd70c153c80 0x7fd70c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 -- 172.21.15.31:0/2818720148 >> 172.21.15.31:0/2818720148 conn(0x7fd70c0715c0 msgr2=0x7fd70c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:28.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 -- 172.21.15.31:0/2818720148 shutdown_connections 2023-10-30T13:57:28.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 -- 172.21.15.31:0/2818720148 wait complete. 2023-10-30T13:57:28.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.913+0000 7fd712017640 1 Processor -- start 2023-10-30T13:57:28.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 -- start start 2023-10-30T13:57:28.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c0c4a10 0x7fd70c1eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd70c153c80 0x7fd70c151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c15a3b0 0x7fd70c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c15afa0 con 0x7fd70c0c4a10 2023-10-30T13:57:28.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c0b3490 con 0x7fd70c153c80 2023-10-30T13:57:28.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c15ac00 con 0x7fd70c15a3b0 2023-10-30T13:57:28.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c0c4a10 0x7fd70c1eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c0c4a10 0x7fd70c1eee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60056/0 (socket says 172.21.15.31:60056) 2023-10-30T13:57:28.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 -- 172.21.15.31:0/2830242766 learned_addr learned my addr 172.21.15.31:0/2830242766 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:28.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70bfff640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c15a3b0 0x7fd70c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70affd640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd70c153c80 0x7fd70c151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 -- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c15a3b0 msgr2=0x7fd70c15b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c15a3b0 0x7fd70c15b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 -- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd70c153c80 msgr2=0x7fd70c151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd70c153c80 0x7fd70c151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 -- 172.21.15.31:0/2830242766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd704009b30 con 0x7fd70c0c4a10 2023-10-30T13:57:28.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd70b7fe640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c0c4a10 0x7fd70c1eee00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00d800 tx=0x7fd6fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd6f3fff640 1 -- 172.21.15.31:0/2830242766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc00b6c0 con 0x7fd70c0c4a10 2023-10-30T13:57:28.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd6f3fff640 1 -- 172.21.15.31:0/2830242766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc00b820 con 0x7fd70c0c4a10 2023-10-30T13:57:28.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c1ef440 con 0x7fd70c0c4a10 2023-10-30T13:57:28.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd6f3fff640 1 -- 172.21.15.31:0/2830242766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc00bb90 con 0x7fd70c0c4a10 2023-10-30T13:57:28.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70c1efa00 con 0x7fd70c0c4a10 2023-10-30T13:57:28.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.917+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70c04e170 con 0x7fd70c0c4a10 2023-10-30T13:57:28.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.921+0000 7fd6f3fff640 1 -- 172.21.15.31:0/2830242766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6fc00bcf0 con 0x7fd70c0c4a10 2023-10-30T13:57:28.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.921+0000 7fd6f3fff640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664d0 0x7fd6f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.921+0000 7fd6f3fff640 1 -- 172.21.15.31:0/2830242766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fd6fc0848c0 con 0x7fd70c0c4a10 2023-10-30T13:57:28.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.921+0000 7fd70affd640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664d0 0x7fd6f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.921+0000 7fd70affd640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664d0 0x7fd6f8068990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd70c1574e0 tx=0x7fd700008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:28.925+0000 7fd6f3fff640 1 -- 172.21.15.31:0/2830242766 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6fc051820 con 0x7fd70c0c4a10 2023-10-30T13:57:29.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.117+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd70c0b5050 con 0x7fd70c0c4a10 2023-10-30T13:57:29.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.117+0000 7fd6f3fff640 1 -- 172.21.15.31:0/2830242766 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fd6fc054e40 con 0x7fd70c0c4a10 2023-10-30T13:57:29.119 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:29.119 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:57:29.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.117+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664d0 msgr2=0x7fd6f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:29.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.117+0000 7fd712017640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664d0 0x7fd6f8068990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd70c1574e0 tx=0x7fd700008040 comp rx=0 tx=0).stop 2023-10-30T13:57:29.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.117+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c0c4a10 msgr2=0x7fd70c1eee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:29.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.117+0000 7fd712017640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c0c4a10 0x7fd70c1eee00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00d800 tx=0x7fd6fc00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:57:29.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 shutdown_connections 2023-10-30T13:57:29.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6f80664d0 0x7fd6f8068990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:29.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd70c15a3b0 0x7fd70c15b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:29.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd70c153c80 0x7fd70c151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:29.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 --2- 172.21.15.31:0/2830242766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd70c0c4a10 0x7fd70c1eee00 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:29.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 >> 172.21.15.31:0/2830242766 conn(0x7fd70c0715c0 msgr2=0x7fd70c152f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:29.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 shutdown_connections 2023-10-30T13:57:29.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:29.121+0000 7fd712017640 1 -- 172.21.15.31:0/2830242766 wait complete. 2023-10-30T13:57:29.140 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:57:29.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:29.230 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:31.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:31.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.293+0000 7f100c10e640 1 Processor -- start 2023-10-30T13:57:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.293+0000 7f100c10e640 1 -- start start 2023-10-30T13:57:31.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100c10e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f100406bd10 0x7f10040c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100c10e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10040c5030 con 0x7f1004151c20 2023-10-30T13:57:31.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100c10e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1004157830 con 0x7f1004151840 2023-10-30T13:57:31.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100c10e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1004157b10 con 0x7f100406bd10 2023-10-30T13:57:31.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100a6ab640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f100406bd10 0x7f10040c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f10096a9640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f1004151c20 0x7f100406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:50256/0 (socket says 172.21.15.31:50256) 2023-10-30T13:57:31.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f10096a9640 1 -- 172.21.15.31:0/3071516763 learned_addr learned my addr 172.21.15.31:0/3071516763 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:57:31.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100a6ab640 1 -- 172.21.15.31:0/3071516763 >> v1:172.21.15.33:6789/0 conn(0x7f1004151840 legacy=0x7f1004072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100a6ab640 1 -- 172.21.15.31:0/3071516763 >> v1:172.21.15.31:6789/0 conn(0x7f1004151c20 legacy=0x7f100406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f100a6ab640 1 -- 172.21.15.31:0/3071516763 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004157df0 con 0x7f100406bd10 2023-10-30T13:57:31.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.297+0000 7f1008ea8640 1 -- 172.21.15.31:0/3071516763 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685308250 0 0) 0x7f10040c5030 con 0x7f1004151c20 2023-10-30T13:57:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100a6ab640 1 --2- 172.21.15.31:0/3071516763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040c49a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009d90 tx=0x7f0ffc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d8c2824a315428 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f1008ea8640 1 -- 172.21.15.31:0/3071516763 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0af600 con 0x7f100406bd10 2023-10-30T13:57:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f1008ea8640 1 -- 172.21.15.31:0/3071516763 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc0ab050 con 0x7f100406bd10 2023-10-30T13:57:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f1008ea8640 1 -- 172.21.15.31:0/3071516763 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0af980 con 0x7f100406bd10 2023-10-30T13:57:31.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- 172.21.15.31:0/3071516763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 msgr2=0x7f10040c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 --2- 172.21.15.31:0/3071516763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040c49a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009d90 tx=0x7f0ffc01f130 comp rx=0 tx=0).stop 2023-10-30T13:57:31.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- 172.21.15.31:0/3071516763 shutdown_connections 2023-10-30T13:57:31.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 --2- 172.21.15.31:0/3071516763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040c49a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- 172.21.15.31:0/3071516763 >> 172.21.15.31:0/3071516763 conn(0x7f10040cac70 msgr2=0x7f10040cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:31.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- 172.21.15.31:0/3071516763 shutdown_connections 2023-10-30T13:57:31.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- 172.21.15.31:0/3071516763 wait complete. 2023-10-30T13:57:31.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 Processor -- start 2023-10-30T13:57:31.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- start start 2023-10-30T13:57:31.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004151840 0x7f10040c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1004151c20 0x7f10041e4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10041eedb0 con 0x7f100406bd10 2023-10-30T13:57:31.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10041ef5f0 con 0x7f1004151840 2023-10-30T13:57:31.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.301+0000 7f100c10e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10040c8f90 con 0x7f1004151c20 2023-10-30T13:57:31.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60080/0 (socket says 172.21.15.31:60080) 2023-10-30T13:57:31.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 -- 172.21.15.31:0/2195591510 learned_addr learned my addr 172.21.15.31:0/2195591510 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:31.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f10096a9640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004151840 0x7f10040c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f100a6ab640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1004151c20 0x7f10041e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 -- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1004151c20 msgr2=0x7f10041e4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1004151c20 0x7f10041e4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 -- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004151840 msgr2=0x7f10040c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004151840 0x7f10040c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 -- 172.21.15.31:0/2195591510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc009a70 con 0x7f100406bd10 2023-10-30T13:57:31.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f1009eaa640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040ca5d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f100000c530 tx=0x7f100000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:31.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f0ffa7fc640 1 -- 172.21.15.31:0/2195591510 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000005220 con 0x7f100406bd10 2023-10-30T13:57:31.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f0ffa7fc640 1 -- 172.21.15.31:0/2195591510 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1000005380 con 0x7f100406bd10 2023-10-30T13:57:31.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f0ffa7fc640 1 -- 172.21.15.31:0/2195591510 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000007270 con 0x7f100406bd10 2023-10-30T13:57:31.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1004071450 con 0x7f100406bd10 2023-10-30T13:57:31.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1004071cd0 con 0x7f100406bd10 2023-10-30T13:57:31.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100404e170 con 0x7f100406bd10 2023-10-30T13:57:31.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f0ffa7fc640 1 -- 172.21.15.31:0/2195591510 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10000073d0 con 0x7f100406bd10 2023-10-30T13:57:31.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.305+0000 7f0ffa7fc640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fe80664a0 0x7f0fe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.309+0000 7f0ffa7fc640 1 -- 172.21.15.31:0/2195591510 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f1000083b80 con 0x7f100406bd10 2023-10-30T13:57:31.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.309+0000 7f10096a9640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fe80664a0 0x7f0fe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.309+0000 7f10096a9640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fe80664a0 0x7f0fe8068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f10041e4a30 tx=0x7f0ff4007d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:31.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.309+0000 7f0ffa7fc640 1 -- 172.21.15.31:0/2195591510 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1000050c00 con 0x7f100406bd10 2023-10-30T13:57:31.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.501+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1004155060 con 0x7f100406bd10 2023-10-30T13:57:31.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.501+0000 7f0ffa7fc640 1 -- 172.21.15.31:0/2195591510 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f1000054220 con 0x7f100406bd10 2023-10-30T13:57:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:31.504 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:31.505 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:57:31.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fe80664a0 msgr2=0x7f0fe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fe80664a0 0x7f0fe8068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f10041e4a30 tx=0x7f0ff4007d10 comp rx=0 tx=0).stop 2023-10-30T13:57:31.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 msgr2=0x7f10040ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040ca5d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f100000c530 tx=0x7f100000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 shutdown_connections 2023-10-30T13:57:31.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0fe80664a0 0x7f0fe8068960 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f1004151c20 0x7f10041e4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004151840 0x7f10040c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 --2- 172.21.15.31:0/2195591510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100406bd10 0x7f10040ca5d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 >> 172.21.15.31:0/2195591510 conn(0x7f10040cac70 msgr2=0x7f10040cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:31.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 shutdown_connections 2023-10-30T13:57:31.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:31.505+0000 7f100c10e640 1 -- 172.21.15.31:0/2195591510 wait complete. 2023-10-30T13:57:31.525 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:57:33.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.669+0000 7f26ec5aa640 1 Processor -- start 2023-10-30T13:57:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.669+0000 7f26ec5aa640 1 -- start start 2023-10-30T13:57:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ec5aa640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f26e4150830 0x7f26e406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ec5aa640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f26e40cb7d0 0x7f26e40c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ec5aa640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e40c3e40 0x7f26e4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ec5aa640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e40c2310 con 0x7f26e4150830 2023-10-30T13:57:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ec5aa640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e4156710 con 0x7f26e40cb7d0 2023-10-30T13:57:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ec5aa640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40c4360 con 0x7f26e40c3e40 2023-10-30T13:57:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f26e4150830 0x7f26e406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f26e4150830 0x7f26e406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60094/0 (socket says 172.21.15.31:60094) 2023-10-30T13:57:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 -- 172.21.15.31:0/1038249602 learned_addr learned my addr 172.21.15.31:0/1038249602 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26eab47640 1 --2- 172.21.15.31:0/1038249602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e40c3e40 0x7f26e4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26e9b45640 1 --2- 172.21.15.31:0/1038249602 >> v2:172.21.15.33:3300/0 conn(0x7f26e40cb7d0 0x7f26e40c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 -- 172.21.15.31:0/1038249602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e40c3e40 msgr2=0x7f26e4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 --2- 172.21.15.31:0/1038249602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e40c3e40 0x7f26e4070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 -- 172.21.15.31:0/1038249602 >> v2:172.21.15.33:3300/0 conn(0x7f26e40cb7d0 msgr2=0x7f26e40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 --2- 172.21.15.31:0/1038249602 >> v2:172.21.15.33:3300/0 conn(0x7f26e40cb7d0 0x7f26e40c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26ea346640 1 -- 172.21.15.31:0/1038249602 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e41569b0 con 0x7f26e4150830 2023-10-30T13:57:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26e9b45640 1 --2- 172.21.15.31:0/1038249602 >> v2:172.21.15.33:3300/0 conn(0x7f26e40cb7d0 0x7f26e40c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:57:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.673+0000 7f26eab47640 1 --2- 172.21.15.31:0/1038249602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e40c3e40 0x7f26e4070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:57:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ea346640 1 --2- 172.21.15.31:0/1038249602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e4150830 0x7f26e406b500 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f26e406f9f0 tx=0x7f26e00213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77853103b497bf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26e9344640 1 -- 172.21.15.31:0/1038249602 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00b1600 con 0x7f26e4150830 2023-10-30T13:57:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26e9344640 1 -- 172.21.15.31:0/1038249602 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e00ad050 con 0x7f26e4150830 2023-10-30T13:57:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26e9344640 1 -- 172.21.15.31:0/1038249602 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00b1980 con 0x7f26e4150830 2023-10-30T13:57:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- 172.21.15.31:0/1038249602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e4150830 msgr2=0x7f26e406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/1038249602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e4150830 0x7f26e406b500 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f26e406f9f0 tx=0x7f26e00213c0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- 172.21.15.31:0/1038249602 shutdown_connections 2023-10-30T13:57:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/1038249602 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e40c3e40 0x7f26e4070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/1038249602 >> v2:172.21.15.33:3300/0 conn(0x7f26e40cb7d0 0x7f26e40c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/1038249602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e4150830 0x7f26e406b500 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- 172.21.15.31:0/1038249602 >> 172.21.15.31:0/1038249602 conn(0x7f26e40cac70 msgr2=0x7f26e40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- 172.21.15.31:0/1038249602 shutdown_connections 2023-10-30T13:57:33.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- 172.21.15.31:0/1038249602 wait complete. 2023-10-30T13:57:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 Processor -- start 2023-10-30T13:57:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- start start 2023-10-30T13:57:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e40c3e40 0x7f26e4145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e40cb7d0 0x7f26e4146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e4150830 0x7f26e41ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e41545d0 con 0x7f26e40c3e40 2023-10-30T13:57:33.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e41eac90 con 0x7f26e40cb7d0 2023-10-30T13:57:33.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ec5aa640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e41eaf70 con 0x7f26e4150830 2023-10-30T13:57:33.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.677+0000 7f26ea346640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e40c3e40 0x7f26e4145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e40c3e40 0x7f26e4145390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60100/0 (socket says 172.21.15.31:60100) 2023-10-30T13:57:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 -- 172.21.15.31:0/3134468221 learned_addr learned my addr 172.21.15.31:0/3134468221 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26e9b45640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e40cb7d0 0x7f26e4146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26eab47640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e4150830 0x7f26e41ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 -- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e4150830 msgr2=0x7f26e41ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e4150830 0x7f26e41ee450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 -- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e40cb7d0 msgr2=0x7f26e4146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e40cb7d0 0x7f26e4146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 -- 172.21.15.31:0/3134468221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e000b770 con 0x7f26e40c3e40 2023-10-30T13:57:33.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ea346640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e40c3e40 0x7f26e4145390 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f26e001fa70 tx=0x7f26e001e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:33.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26daffd640 1 -- 172.21.15.31:0/3134468221 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00ba810 con 0x7f26e40c3e40 2023-10-30T13:57:33.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26daffd640 1 -- 172.21.15.31:0/3134468221 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e00ba990 con 0x7f26e40c3e40 2023-10-30T13:57:33.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26daffd640 1 -- 172.21.15.31:0/3134468221 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00bad40 con 0x7f26e40c3e40 2023-10-30T13:57:33.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e41eb250 con 0x7f26e40c3e40 2023-10-30T13:57:33.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e41edc50 con 0x7f26e40c3e40 2023-10-30T13:57:33.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4066510 con 0x7f26e40c3e40 2023-10-30T13:57:33.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26daffd640 1 -- 172.21.15.31:0/3134468221 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26e00b8040 con 0x7f26e40c3e40 2023-10-30T13:57:33.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26daffd640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26c80664a0 0x7f26c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.681+0000 7f26daffd640 1 -- 172.21.15.31:0/3134468221 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f26e01248f0 con 0x7f26e40c3e40 2023-10-30T13:57:33.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.685+0000 7f26e9b45640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26c80664a0 0x7f26c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.685+0000 7f26e9b45640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26c80664a0 0x7f26c8068960 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f26e40717b0 tx=0x7f26d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.689+0000 7f26daffd640 1 -- 172.21.15.31:0/3134468221 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26e00f1a20 con 0x7f26e40c3e40 2023-10-30T13:57:33.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.877+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e404e170 con 0x7f26e40c3e40 2023-10-30T13:57:33.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.877+0000 7f26daffd640 1 -- 172.21.15.31:0/3134468221 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f26e00b6070 con 0x7f26e40c3e40 2023-10-30T13:57:33.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:33.880 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:33.881 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:57:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26c80664a0 msgr2=0x7f26c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26c80664a0 0x7f26c8068960 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f26e40717b0 tx=0x7f26d4008040 comp rx=0 tx=0).stop 2023-10-30T13:57:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e40c3e40 msgr2=0x7f26e4145390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e40c3e40 0x7f26e4145390 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f26e001fa70 tx=0x7f26e001e160 comp rx=0 tx=0).stop 2023-10-30T13:57:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 shutdown_connections 2023-10-30T13:57:33.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f26c80664a0 0x7f26c8068960 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f26e4150830 0x7f26e41ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e40cb7d0 0x7f26e4146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 --2- 172.21.15.31:0/3134468221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26e40c3e40 0x7f26e4145390 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 >> 172.21.15.31:0/3134468221 conn(0x7f26e40cac70 msgr2=0x7f26e40c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:33.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 shutdown_connections 2023-10-30T13:57:33.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:33.881+0000 7f26ec5aa640 1 -- 172.21.15.31:0/3134468221 wait complete. 2023-10-30T13:57:33.901 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:57:35.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:36.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.053+0000 7fe4d2503640 1 Processor -- start 2023-10-30T13:57:36.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d2503640 1 -- start start 2023-10-30T13:57:36.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d2503640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe4cc06dff0 0x7fe4cc15f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:36.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d2503640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc0c67d0 con 0x7fe4cc157c80 2023-10-30T13:57:36.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d2503640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc0c58e0 con 0x7fe4cc0c5460 2023-10-30T13:57:36.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d2503640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4cc0c5bc0 con 0x7fe4cc06dff0 2023-10-30T13:57:36.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d1d02640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe4cc06dff0 0x7fe4cc15f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:36.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d1d02640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe4cc06dff0 0x7fe4cc15f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40790/0 (socket says 172.21.15.31:40790) 2023-10-30T13:57:36.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d1d02640 1 -- 172.21.15.31:0/2656546746 learned_addr learned my addr 172.21.15.31:0/2656546746 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:36.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d1d02640 1 -- 172.21.15.31:0/2656546746 >> v1:172.21.15.33:6789/0 conn(0x7fe4cc0c5460 legacy=0x7fe4cc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:36.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d1d02640 1 -- 172.21.15.31:0/2656546746 >> v1:172.21.15.31:6789/0 conn(0x7fe4cc157c80 legacy=0x7fe4cc15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4d1d02640 1 -- 172.21.15.31:0/2656546746 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc0c5ea0 con 0x7fe4cc06dff0 2023-10-30T13:57:36.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.057+0000 7fe4c3fff640 1 -- 172.21.15.31:0/2656546746 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078150130 0 0) 0x7fe4cc0c67d0 con 0x7fe4cc157c80 2023-10-30T13:57:36.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d1d02640 1 --2- 172.21.15.31:0/2656546746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc06dff0 0x7fe4cc15f310 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009e50 tx=0x7fe4c401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6452aa39cd025045 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4c3fff640 1 -- 172.21.15.31:0/2656546746 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40a5860 con 0x7fe4cc06dff0 2023-10-30T13:57:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4c3fff640 1 -- 172.21.15.31:0/2656546746 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c40a59c0 con 0x7fe4cc06dff0 2023-10-30T13:57:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4c3fff640 1 -- 172.21.15.31:0/2656546746 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40a5d50 con 0x7fe4cc06dff0 2023-10-30T13:57:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 -- 172.21.15.31:0/2656546746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc06dff0 msgr2=0x7fe4cc15f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 --2- 172.21.15.31:0/2656546746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc06dff0 0x7fe4cc15f310 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009e50 tx=0x7fe4c401c9a0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 -- 172.21.15.31:0/2656546746 shutdown_connections 2023-10-30T13:57:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 --2- 172.21.15.31:0/2656546746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc06dff0 0x7fe4cc15f310 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 -- 172.21.15.31:0/2656546746 >> 172.21.15.31:0/2656546746 conn(0x7fe4cc0caae0 msgr2=0x7fe4cc0caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:36.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 -- 172.21.15.31:0/2656546746 shutdown_connections 2023-10-30T13:57:36.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 -- 172.21.15.31:0/2656546746 wait complete. 2023-10-30T13:57:36.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 Processor -- start 2023-10-30T13:57:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.061+0000 7fe4d2503640 1 -- start start 2023-10-30T13:57:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4cc06dff0 0x7fe4cc1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc0c5460 0x7fe4cc072830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe4cc157c80 0x7fe4cc0c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc154910 con 0x7fe4cc0c5460 2023-10-30T13:57:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc155040 con 0x7fe4cc06dff0 2023-10-30T13:57:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc151150 con 0x7fe4cc157c80 2023-10-30T13:57:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc0c5460 0x7fe4cc072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc0c5460 0x7fe4cc072830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40796/0 (socket says 172.21.15.31:40796) 2023-10-30T13:57:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 -- 172.21.15.31:0/1343978069 learned_addr learned my addr 172.21.15.31:0/1343978069 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d1501640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4cc06dff0 0x7fe4cc1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d1d02640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe4cc157c80 0x7fe4cc0c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 -- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe4cc157c80 msgr2=0x7fe4cc0c35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe4cc157c80 0x7fe4cc0c35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 -- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4cc06dff0 msgr2=0x7fe4cc1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4cc06dff0 0x7fe4cc1541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 -- 172.21.15.31:0/1343978069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c4009b30 con 0x7fe4cc0c5460 2023-10-30T13:57:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d1d02640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe4cc157c80 0x7fe4cc0c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d0d00640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc0c5460 0x7fe4cc072830 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00c450 tx=0x7fe4bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4c1ffb640 1 -- 172.21.15.31:0/1343978069 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc012830 con 0x7fe4cc0c5460 2023-10-30T13:57:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4c1ffb640 1 -- 172.21.15.31:0/1343978069 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0129b0 con 0x7fe4cc0c5460 2023-10-30T13:57:36.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4cc1f6750 con 0x7fe4cc0c5460 2023-10-30T13:57:36.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4c1ffb640 1 -- 172.21.15.31:0/1343978069 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc012d60 con 0x7fe4cc0c5460 2023-10-30T13:57:36.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4cc154b20 con 0x7fe4cc0c5460 2023-10-30T13:57:36.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.065+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4cc04e170 con 0x7fe4cc0c5460 2023-10-30T13:57:36.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.069+0000 7fe4c1ffb640 1 -- 172.21.15.31:0/1343978069 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4bc01c460 con 0x7fe4cc0c5460 2023-10-30T13:57:36.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.069+0000 7fe4c1ffb640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4b00664a0 0x7fe4b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:36.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.069+0000 7fe4c1ffb640 1 -- 172.21.15.31:0/1343978069 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fe4bc083ff0 con 0x7fe4cc0c5460 2023-10-30T13:57:36.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.069+0000 7fe4d1501640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4b00664a0 0x7fe4b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:36.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.069+0000 7fe4d1501640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4b00664a0 0x7fe4b0068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8007fd0 tx=0x7fe4c8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:36.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.073+0000 7fe4c1ffb640 1 -- 172.21.15.31:0/1343978069 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4bc0510d0 con 0x7fe4cc0c5460 2023-10-30T13:57:36.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4cc06e4b0 con 0x7fe4cc0c5460 2023-10-30T13:57:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4c1ffb640 1 -- 172.21.15.31:0/1343978069 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fe4bc0546f0 con 0x7fe4cc0c5460 2023-10-30T13:57:36.262 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:36.262 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:36.263 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:57:36.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4b00664a0 msgr2=0x7fe4b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4b00664a0 0x7fe4b0068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8007fd0 tx=0x7fe4c8009040 comp rx=0 tx=0).stop 2023-10-30T13:57:36.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc0c5460 msgr2=0x7fe4cc072830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc0c5460 0x7fe4cc072830 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00c450 tx=0x7fe4bc00c910 comp rx=0 tx=0).stop 2023-10-30T13:57:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 shutdown_connections 2023-10-30T13:57:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe4b00664a0 0x7fe4b0068960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe4cc157c80 0x7fe4cc0c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4cc0c5460 0x7fe4cc072830 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 --2- 172.21.15.31:0/1343978069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4cc06dff0 0x7fe4cc1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 >> 172.21.15.31:0/1343978069 conn(0x7fe4cc0caae0 msgr2=0x7fe4cc0c7480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 shutdown_connections 2023-10-30T13:57:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:36.261+0000 7fe4d2503640 1 -- 172.21.15.31:0/1343978069 wait complete. 2023-10-30T13:57:36.283 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:57:38.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd4013e640 1 Processor -- start 2023-10-30T13:57:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd4013e640 1 -- start start 2023-10-30T13:57:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd4013e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd3815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd4013e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7efd380c42a0 0x7efd380ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd4013e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd38159f20 con 0x7efd380c4a10 2023-10-30T13:57:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd4013e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3815dec0 con 0x7efd380c42a0 2023-10-30T13:57:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd4013e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd381599a0 con 0x7efd38153c80 2023-10-30T13:57:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3deda640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd3815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3deda640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd3815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:54154/0 (socket says 172.21.15.31:54154) 2023-10-30T13:57:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3e6db640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7efd380c42a0 0x7efd380ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3d6d9640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7efd380c4a10 0x7efd3815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:58822/0 (socket says 172.21.15.31:58822) 2023-10-30T13:57:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3deda640 1 -- 172.21.15.31:0/2443231734 learned_addr learned my addr 172.21.15.31:0/2443231734 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3deda640 1 -- 172.21.15.31:0/2443231734 >> v2:172.21.15.33:3300/0 conn(0x7efd380c42a0 msgr2=0x7efd380ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3deda640 1 --2- 172.21.15.31:0/2443231734 >> v2:172.21.15.33:3300/0 conn(0x7efd380c42a0 0x7efd380ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3deda640 1 -- 172.21.15.31:0/2443231734 >> v1:172.21.15.33:6789/0 conn(0x7efd380c4a10 legacy=0x7efd3815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.521+0000 7efd3deda640 1 -- 172.21.15.31:0/2443231734 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3815d620 con 0x7efd38153c80 2023-10-30T13:57:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd3deda640 1 --2- 172.21.15.31:0/2443231734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd3815cf90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efd34009870 tx=0x7efd3401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44316b7a0049cbd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd3ced8640 1 -- 172.21.15.31:0/2443231734 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340a8a30 con 0x7efd38153c80 2023-10-30T13:57:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd3ced8640 1 -- 172.21.15.31:0/2443231734 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd340a8b90 con 0x7efd38153c80 2023-10-30T13:57:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd3ced8640 1 -- 172.21.15.31:0/2443231734 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340b0440 con 0x7efd38153c80 2023-10-30T13:57:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 -- 172.21.15.31:0/2443231734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 msgr2=0x7efd3815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 --2- 172.21.15.31:0/2443231734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd3815cf90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efd34009870 tx=0x7efd3401ed30 comp rx=0 tx=0).stop 2023-10-30T13:57:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 -- 172.21.15.31:0/2443231734 shutdown_connections 2023-10-30T13:57:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 --2- 172.21.15.31:0/2443231734 >> v2:172.21.15.33:3300/0 conn(0x7efd380c42a0 0x7efd380ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 --2- 172.21.15.31:0/2443231734 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd3815cf90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 -- 172.21.15.31:0/2443231734 >> 172.21.15.31:0/2443231734 conn(0x7efd380715c0 msgr2=0x7efd380719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:38.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 -- 172.21.15.31:0/2443231734 shutdown_connections 2023-10-30T13:57:38.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.525+0000 7efd4013e640 1 -- 172.21.15.31:0/2443231734 wait complete. 2023-10-30T13:57:38.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 Processor -- start 2023-10-30T13:57:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 -- start start 2023-10-30T13:57:38.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd380c42a0 0x7efd380ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd380c4a10 0x7efd381eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd38149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380c8ae0 con 0x7efd380c4a10 2023-10-30T13:57:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd38155710 con 0x7efd380c42a0 2023-10-30T13:57:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd381ee760 con 0x7efd38153c80 2023-10-30T13:57:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd380c42a0 0x7efd380ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3d6d9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd380c4a10 0x7efd381eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd380c42a0 0x7efd380ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:59692/0 (socket says 172.21.15.31:59692) 2023-10-30T13:57:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 -- 172.21.15.31:0/69938795 learned_addr learned my addr 172.21.15.31:0/69938795 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3e6db640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd38149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 -- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 msgr2=0x7efd38149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd38149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 -- 172.21.15.31:0/69938795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd380c4a10 msgr2=0x7efd381eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd380c4a10 0x7efd381eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 -- 172.21.15.31:0/69938795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd340094d0 con 0x7efd380c42a0 2023-10-30T13:57:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd3deda640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd380c42a0 0x7efd380ce420 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efd340b3040 tx=0x7efd3401bad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd2e7fc640 1 -- 172.21.15.31:0/69938795 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340b9620 con 0x7efd380c42a0 2023-10-30T13:57:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd2e7fc640 1 -- 172.21.15.31:0/69938795 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd340b97a0 con 0x7efd380c42a0 2023-10-30T13:57:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd2e7fc640 1 -- 172.21.15.31:0/69938795 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340b9b50 con 0x7efd380c42a0 2023-10-30T13:57:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd381f2a20 con 0x7efd380c42a0 2023-10-30T13:57:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd380c8800 con 0x7efd380c42a0 2023-10-30T13:57:38.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.529+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3804e170 con 0x7efd380c42a0 2023-10-30T13:57:38.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.533+0000 7efd2e7fc640 1 -- 172.21.15.31:0/69938795 <== mon.1 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd340b7070 con 0x7efd380c42a0 2023-10-30T13:57:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.533+0000 7efd2e7fc640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.533+0000 7efd2e7fc640 1 -- 172.21.15.31:0/69938795 <== mon.1 v2:172.21.15.33:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7efd34122060 con 0x7efd380c42a0 2023-10-30T13:57:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.533+0000 7efd3d6d9640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.533+0000 7efd3d6d9640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efd38150020 tx=0x7efd28007e50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.537+0000 7efd2e7fc640 1 -- 172.21.15.31:0/69938795 <== mon.1 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd340ef100 con 0x7efd380c42a0 2023-10-30T13:57:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd380cad40 con 0x7efd380c42a0 2023-10-30T13:57:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd2e7fc640 1 -- 172.21.15.31:0/69938795 <== mon.1 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7efd340f2720 con 0x7efd380c42a0 2023-10-30T13:57:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:38.731 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:57:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 msgr2=0x7efd1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efd38150020 tx=0x7efd28007e50 comp rx=0 tx=0).stop 2023-10-30T13:57:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd380c42a0 msgr2=0x7efd380ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd380c42a0 0x7efd380ce420 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efd340b3040 tx=0x7efd3401bad0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 shutdown_connections 2023-10-30T13:57:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7efd1c0664a0 0x7efd1c068960 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7efd38153c80 0x7efd38149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd380c4a10 0x7efd381eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.729+0000 7efd4013e640 1 --2- 172.21.15.31:0/69938795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd380c42a0 0x7efd380ce420 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.733+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 >> 172.21.15.31:0/69938795 conn(0x7efd380715c0 msgr2=0x7efd38158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.733+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 shutdown_connections 2023-10-30T13:57:38.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:38.733+0000 7efd4013e640 1 -- 172.21.15.31:0/69938795 wait complete. 2023-10-30T13:57:38.752 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:57:40.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:40.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd68a2dc640 1 Processor -- start 2023-10-30T13:57:40.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd68a2dc640 1 -- start start 2023-10-30T13:57:40.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd68a2dc640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd6840c8eb0 0x7fd68415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd68a2dc640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd68a2dc640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68415dcc0 con 0x7fd68415ac70 2023-10-30T13:57:40.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd68a2dc640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd684159800 con 0x7fd6840c8eb0 2023-10-30T13:57:40.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd68a2dc640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd684159d80 con 0x7fd68406aeb0 2023-10-30T13:57:40.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd689adb640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd6892da640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd6840c8eb0 0x7fd68415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd6892da640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd6840c8eb0 0x7fd68415cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:59706/0 (socket says 172.21.15.31:59706) 2023-10-30T13:57:40.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd689adb640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:54158/0 (socket says 172.21.15.31:54158) 2023-10-30T13:57:40.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd6892da640 1 -- 172.21.15.31:0/1429987136 learned_addr learned my addr 172.21.15.31:0/1429987136 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:40.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd688ad9640 1 --1- 172.21.15.31:0/1429987136 >> v1:172.21.15.33:6789/0 conn(0x7fd68415ac70 0x7fd68415a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:58826/0 (socket says 172.21.15.31:58826) 2023-10-30T13:57:40.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd6892da640 1 -- 172.21.15.31:0/1429987136 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 msgr2=0x7fd684069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:40.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd6892da640 1 --2- 172.21.15.31:0/1429987136 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd6892da640 1 -- 172.21.15.31:0/1429987136 >> v1:172.21.15.33:6789/0 conn(0x7fd68415ac70 legacy=0x7fd68415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.909+0000 7fd6892da640 1 -- 172.21.15.31:0/1429987136 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68415d420 con 0x7fd6840c8eb0 2023-10-30T13:57:40.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd6892da640 1 --2- 172.21.15.31:0/1429987136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6840c8eb0 0x7fd68415cd70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd680009870 tx=0x7fd68001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdd7e1efd908f5e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:40.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd67bfff640 1 -- 172.21.15.31:0/1429987136 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6800a8a30 con 0x7fd6840c8eb0 2023-10-30T13:57:40.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd67bfff640 1 -- 172.21.15.31:0/1429987136 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6800a8b90 con 0x7fd6840c8eb0 2023-10-30T13:57:40.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd67bfff640 1 -- 172.21.15.31:0/1429987136 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6800b0440 con 0x7fd6840c8eb0 2023-10-30T13:57:40.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 -- 172.21.15.31:0/1429987136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6840c8eb0 msgr2=0x7fd68415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/1429987136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6840c8eb0 0x7fd68415cd70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd680009870 tx=0x7fd68001ed30 comp rx=0 tx=0).stop 2023-10-30T13:57:40.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 -- 172.21.15.31:0/1429987136 shutdown_connections 2023-10-30T13:57:40.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/1429987136 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/1429987136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6840c8eb0 0x7fd68415cd70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 -- 172.21.15.31:0/1429987136 >> 172.21.15.31:0/1429987136 conn(0x7fd6840cef00 msgr2=0x7fd6840cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:40.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 -- 172.21.15.31:0/1429987136 shutdown_connections 2023-10-30T13:57:40.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.913+0000 7fd68a2dc640 1 -- 172.21.15.31:0/1429987136 wait complete. 2023-10-30T13:57:40.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 Processor -- start 2023-10-30T13:57:40.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 -- start start 2023-10-30T13:57:40.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6840c8eb0 0x7fd68414cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd68415ac70 0x7fd684152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6840c3b30 con 0x7fd6840c8eb0 2023-10-30T13:57:40.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684151070 con 0x7fd68415ac70 2023-10-30T13:57:40.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6840c2f20 con 0x7fd68406aeb0 2023-10-30T13:57:40.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6840c8eb0 0x7fd68414cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6840c8eb0 0x7fd68414cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40814/0 (socket says 172.21.15.31:40814) 2023-10-30T13:57:40.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 -- 172.21.15.31:0/115015560 learned_addr learned my addr 172.21.15.31:0/115015560 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:40.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd6892da640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd689adb640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd68415ac70 0x7fd684152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 -- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 msgr2=0x7fd684150a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684150a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 -- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd68415ac70 msgr2=0x7fd684152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd68415ac70 0x7fd684152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 -- 172.21.15.31:0/115015560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6800094d0 con 0x7fd6840c8eb0 2023-10-30T13:57:40.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd6892da640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684150a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:40.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd688ad9640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6840c8eb0 0x7fd68414cbc0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd67400c430 tx=0x7fd67400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:40.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd679ffb640 1 -- 172.21.15.31:0/115015560 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6740074e0 con 0x7fd6840c8eb0 2023-10-30T13:57:40.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd679ffb640 1 -- 172.21.15.31:0/115015560 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd674007640 con 0x7fd6840c8eb0 2023-10-30T13:57:40.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd679ffb640 1 -- 172.21.15.31:0/115015560 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd674004a10 con 0x7fd6840c8eb0 2023-10-30T13:57:40.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6840c3200 con 0x7fd6840c8eb0 2023-10-30T13:57:40.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.917+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6840c37c0 con 0x7fd6840c8eb0 2023-10-30T13:57:40.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.921+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6840c3980 con 0x7fd6840c8eb0 2023-10-30T13:57:40.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.921+0000 7fd679ffb640 1 -- 172.21.15.31:0/115015560 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd674004b70 con 0x7fd6840c8eb0 2023-10-30T13:57:40.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.921+0000 7fd679ffb640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6680664d0 0x7fd668068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.921+0000 7fd679ffb640 1 -- 172.21.15.31:0/115015560 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fd674083ab0 con 0x7fd6840c8eb0 2023-10-30T13:57:40.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.921+0000 7fd6892da640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6680664d0 0x7fd668068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.921+0000 7fd6892da640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6680664d0 0x7fd668068990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd6800b2000 tx=0x7fd680008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:40.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:40.925+0000 7fd679ffb640 1 -- 172.21.15.31:0/115015560 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd674050b30 con 0x7fd6840c8eb0 2023-10-30T13:57:41.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd68415dcc0 con 0x7fd6840c8eb0 2023-10-30T13:57:41.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd679ffb640 1 -- 172.21.15.31:0/115015560 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fd68415dcc0 con 0x7fd6840c8eb0 2023-10-30T13:57:41.114 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:41.114 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6549,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:56:30.776806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5813":{"gid":5813,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":35,"addr":"172.21.15.33:6836/2281022644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2281022644},{"type":"v1","addr":"172.21.15.33:6836","nonce":2281022644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:41.114 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2023-10-30T13:57:41.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6680664d0 msgr2=0x7fd668068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:41.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6680664d0 0x7fd668068990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd6800b2000 tx=0x7fd680008040 comp rx=0 tx=0).stop 2023-10-30T13:57:41.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6840c8eb0 msgr2=0x7fd68414cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:41.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6840c8eb0 0x7fd68414cbc0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd67400c430 tx=0x7fd67400c8f0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 shutdown_connections 2023-10-30T13:57:41.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6680664d0 0x7fd668068990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd68415ac70 0x7fd684152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6840c8eb0 0x7fd68414cbc0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 --2- 172.21.15.31:0/115015560 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd68406aeb0 0x7fd684150a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 >> 172.21.15.31:0/115015560 conn(0x7fd6840cef00 msgr2=0x7fd684157b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 shutdown_connections 2023-10-30T13:57:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:41.113+0000 7fd68a2dc640 1 -- 172.21.15.31:0/115015560 wait complete. 2023-10-30T13:57:41.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:57:43.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:43.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f80811c4640 1 Processor -- start 2023-10-30T13:57:43.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f80811c4640 1 -- start start 2023-10-30T13:57:43.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f80811c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f80811c4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f807c0cb8b0 0x7f807c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f80811c4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c0c5030 con 0x7f807c151840 2023-10-30T13:57:43.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f80811c4640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c157830 con 0x7f807c0cb8b0 2023-10-30T13:57:43.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f80811c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c157b10 con 0x7f807c151c20 2023-10-30T13:57:43.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f807b59e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f807c0cb8b0 0x7f807c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f807ad9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f807c151840 0x7f807c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58522/0 (socket says 172.21.15.31:58522) 2023-10-30T13:57:43.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.289+0000 7f807ad9d640 1 -- 172.21.15.31:0/3378499561 learned_addr learned my addr 172.21.15.31:0/3378499561 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:57:43.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f807a59c640 1 --2- 172.21.15.31:0/3378499561 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f807b59e640 1 -- 172.21.15.31:0/3378499561 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 msgr2=0x7f807c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:43.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f807b59e640 1 --2- 172.21.15.31:0/3378499561 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f807b59e640 1 -- 172.21.15.31:0/3378499561 >> v1:172.21.15.31:6789/0 conn(0x7f807c151840 legacy=0x7f807c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f807b59e640 1 -- 172.21.15.31:0/3378499561 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c157df0 con 0x7f807c0cb8b0 2023-10-30T13:57:43.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f807b59e640 1 --2- 172.21.15.31:0/3378499561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0cb8b0 0x7f807c06b600 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8074009d90 tx=0x7f807401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46626548b246af40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f8079d9b640 1 -- 172.21.15.31:0/3378499561 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80740af600 con 0x7f807c0cb8b0 2023-10-30T13:57:43.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f8079d9b640 1 -- 172.21.15.31:0/3378499561 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80740ab050 con 0x7f807c0cb8b0 2023-10-30T13:57:43.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f8079d9b640 1 -- 172.21.15.31:0/3378499561 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80740af980 con 0x7f807c0cb8b0 2023-10-30T13:57:43.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 -- 172.21.15.31:0/3378499561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0cb8b0 msgr2=0x7f807c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 --2- 172.21.15.31:0/3378499561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0cb8b0 0x7f807c06b600 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8074009d90 tx=0x7f807401f130 comp rx=0 tx=0).stop 2023-10-30T13:57:43.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 -- 172.21.15.31:0/3378499561 shutdown_connections 2023-10-30T13:57:43.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 --2- 172.21.15.31:0/3378499561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c0cb8b0 0x7f807c06b600 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 --2- 172.21.15.31:0/3378499561 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 -- 172.21.15.31:0/3378499561 >> 172.21.15.31:0/3378499561 conn(0x7f807c0cac70 msgr2=0x7f807c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:43.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 -- 172.21.15.31:0/3378499561 shutdown_connections 2023-10-30T13:57:43.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.293+0000 7f80811c4640 1 -- 172.21.15.31:0/3378499561 wait complete. 2023-10-30T13:57:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 Processor -- start 2023-10-30T13:57:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 -- start start 2023-10-30T13:57:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c0cb8b0 0x7f807c071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c151840 0x7f807c145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c1f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c1f26f0 con 0x7f807c151840 2023-10-30T13:57:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c1561b0 con 0x7f807c0cb8b0 2023-10-30T13:57:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f80811c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0722a0 con 0x7f807c151c20 2023-10-30T13:57:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c151840 0x7f807c145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c151840 0x7f807c145060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40826/0 (socket says 172.21.15.31:40826) 2023-10-30T13:57:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 -- 172.21.15.31:0/2701361455 learned_addr learned my addr 172.21.15.31:0/2701361455 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807ad9d640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c0cb8b0 0x7f807c071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807b59e640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c1f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 -- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 msgr2=0x7f807c1f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c1f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 -- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c0cb8b0 msgr2=0x7f807c071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c0cb8b0 0x7f807c071c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 -- 172.21.15.31:0/2701361455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074009a70 con 0x7f807c151840 2023-10-30T13:57:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.297+0000 7f807a59c640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c151840 0x7f807c145060 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f807000d7d0 tx=0x7f807000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80637fe640 1 -- 172.21.15.31:0/2701361455 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807000b6c0 con 0x7f807c151840 2023-10-30T13:57:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80637fe640 1 -- 172.21.15.31:0/2701361455 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807000b820 con 0x7f807c151840 2023-10-30T13:57:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c155630 con 0x7f807c151840 2023-10-30T13:57:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80637fe640 1 -- 172.21.15.31:0/2701361455 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807000bbb0 con 0x7f807c151840 2023-10-30T13:57:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c155bf0 con 0x7f807c151840 2023-10-30T13:57:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c0c4230 con 0x7f807c151840 2023-10-30T13:57:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80637fe640 1 -- 172.21.15.31:0/2701361455 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f807000bd10 con 0x7f807c151840 2023-10-30T13:57:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80637fe640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80680664d0 0x7f8068068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f80637fe640 1 -- 172.21.15.31:0/2701361455 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f80700847b0 con 0x7f807c151840 2023-10-30T13:57:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.301+0000 7f807ad9d640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80680664d0 0x7f8068068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.305+0000 7f807ad9d640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80680664d0 0x7f8068068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f806c005890 tx=0x7f806c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.305+0000 7f80637fe640 1 -- 172.21.15.31:0/2701361455 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80700516f0 con 0x7f807c151840 2023-10-30T13:57:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.493+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807c1f24c0 con 0x7f807c151840 2023-10-30T13:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80637fe640 1 -- 172.21.15.31:0/2701361455 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f8070054d10 con 0x7f807c151840 2023-10-30T13:57:43.497 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:43.497 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:42.156930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2023-10-30T13:57:43.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80680664d0 msgr2=0x7f8068068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80680664d0 0x7f8068068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f806c005890 tx=0x7f806c009040 comp rx=0 tx=0).stop 2023-10-30T13:57:43.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c151840 msgr2=0x7f807c145060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c151840 0x7f807c145060 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f807000d7d0 tx=0x7f807000dc90 comp rx=0 tx=0).stop 2023-10-30T13:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 shutdown_connections 2023-10-30T13:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f80680664d0 0x7f8068068990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f807c151c20 0x7f807c1f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f807c151840 0x7f807c145060 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 --2- 172.21.15.31:0/2701361455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c0cb8b0 0x7f807c071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 >> 172.21.15.31:0/2701361455 conn(0x7f807c0cac70 msgr2=0x7f807c1e49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 shutdown_connections 2023-10-30T13:57:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.497+0000 7f80811c4640 1 -- 172.21.15.31:0/2701361455 wait complete. 2023-10-30T13:57:43.518 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:57:43.518 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:57:43.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:43.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.661+0000 7fab9ed9d640 1 Processor -- start 2023-10-30T13:57:43.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.661+0000 7fab9ed9d640 1 -- start start 2023-10-30T13:57:43.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 0x7fab980c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9ed9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fab980c3e40 0x7fab98070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9ed9d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab980c2310 con 0x7fab98150830 2023-10-30T13:57:43.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9ed9d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9806bc10 con 0x7fab980c3e40 2023-10-30T13:57:43.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9ed9d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab98156710 con 0x7fab980cb7d0 2023-10-30T13:57:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9cb39640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fab98150830 0x7fab9806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:58524/0 (socket says 172.21.15.31:58524) 2023-10-30T13:57:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9cb39640 1 -- 172.21.15.31:0/1551393822 learned_addr learned my addr 172.21.15.31:0/1551393822 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:57:43.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9d33a640 1 --2- 172.21.15.31:0/1551393822 >> v2:172.21.15.33:3300/0 conn(0x7fab980c3e40 0x7fab98070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab97fff640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 0x7fab980c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654384924 0 0) 0x7fab980c2310 con 0x7fab98150830 2023-10-30T13:57:43.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab88003530 con 0x7fab98150830 2023-10-30T13:57:43.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 866842715 0 0) 0x7fab88003530 con 0x7fab98150830 2023-10-30T13:57:43.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 msgr2=0x7fab980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:43.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 0x7fab980c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 >> v2:172.21.15.33:3300/0 conn(0x7fab980c3e40 msgr2=0x7fab98070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:43.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 --2- 172.21.15.31:0/1551393822 >> v2:172.21.15.33:3300/0 conn(0x7fab980c3e40 0x7fab98070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab980c4360 con 0x7fab98150830 2023-10-30T13:57:43.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fab8c001680 con 0x7fab98150830 2023-10-30T13:57:43.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 >> v1:172.21.15.31:6789/0 conn(0x7fab98150830 legacy=0x7fab9806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 0x7fab88003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab88003f00 0x7fab880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab88005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9806bc10 con 0x7fab88004d10 2023-10-30T13:57:43.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab98156710 con 0x7fab88003f00 2023-10-30T13:57:43.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8800b9c0 con 0x7fab980cb7d0 2023-10-30T13:57:43.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9d33a640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab88005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9cb39640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 0x7fab88003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab97fff640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab88003f00 0x7fab880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9d33a640 1 -- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 msgr2=0x7fab88003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:43.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9d33a640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980cb7d0 0x7fab88003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9d33a640 1 -- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab88003f00 msgr2=0x7fab880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:43.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9d33a640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab88003f00 0x7fab880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.665+0000 7fab9d33a640 1 -- 172.21.15.31:0/1551393822 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88003530 con 0x7fab88004d10 2023-10-30T13:57:43.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9d33a640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab88005190 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fab90009f90 tx=0x7fab9001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2258157280707120 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab900a5c70 con 0x7fab88004d10 2023-10-30T13:57:43.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9000f040 con 0x7fab88004d10 2023-10-30T13:57:43.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab977fe640 1 -- 172.21.15.31:0/1551393822 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab900ad440 con 0x7fab88004d10 2023-10-30T13:57:43.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 -- 172.21.15.31:0/1551393822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 msgr2=0x7fab88005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 --2- 172.21.15.31:0/1551393822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab88005190 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fab90009f90 tx=0x7fab9001cc60 comp rx=0 tx=0).stop 2023-10-30T13:57:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9d33a640 1 -- 172.21.15.31:0/1551393822 reap_dead start 2023-10-30T13:57:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 -- 172.21.15.31:0/1551393822 shutdown_connections 2023-10-30T13:57:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 -- 172.21.15.31:0/1551393822 >> 172.21.15.31:0/1551393822 conn(0x7fab980cac70 msgr2=0x7fab980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 -- 172.21.15.31:0/1551393822 shutdown_connections 2023-10-30T13:57:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 -- 172.21.15.31:0/1551393822 wait complete. 2023-10-30T13:57:43.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 Processor -- start 2023-10-30T13:57:43.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.669+0000 7fab9ed9d640 1 -- start start 2023-10-30T13:57:43.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab980c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980c8ae0 0x7fab9815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981585e0 0x7fab9815a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab98158fe0 con 0x7fab88004d10 2023-10-30T13:57:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9815ab60 con 0x7fab981585e0 2023-10-30T13:57:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab98157860 con 0x7fab980c8ae0 2023-10-30T13:57:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab980c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab980c9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40854/0 (socket says 172.21.15.31:40854) 2023-10-30T13:57:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 -- 172.21.15.31:0/4130139621 learned_addr learned my addr 172.21.15.31:0/4130139621 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab97fff640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980c8ae0 0x7fab9815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9d33a640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981585e0 0x7fab9815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 -- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980c8ae0 msgr2=0x7fab9815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980c8ae0 0x7fab9815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 -- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981585e0 msgr2=0x7fab9815a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981585e0 0x7fab9815a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 -- 172.21.15.31:0/4130139621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab981ef090 con 0x7fab88004d10 2023-10-30T13:57:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9cb39640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab980c9920 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fab8c000fc0 tx=0x7fab8c007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab957fa640 1 -- 172.21.15.31:0/4130139621 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c00f9c0 con 0x7fab88004d10 2023-10-30T13:57:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab957fa640 1 -- 172.21.15.31:0/4130139621 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8c00fb40 con 0x7fab88004d10 2023-10-30T13:57:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab98157b00 con 0x7fab88004d10 2023-10-30T13:57:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab957fa640 1 -- 172.21.15.31:0/4130139621 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c0133f0 con 0x7fab88004d10 2023-10-30T13:57:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab981580c0 con 0x7fab88004d10 2023-10-30T13:57:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9804e170 con 0x7fab88004d10 2023-10-30T13:57:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.673+0000 7fab957fa640 1 -- 172.21.15.31:0/4130139621 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab8c013550 con 0x7fab88004d10 2023-10-30T13:57:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.677+0000 7fab957fa640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab7c0664a0 0x7fab7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.677+0000 7fab957fa640 1 -- 172.21.15.31:0/4130139621 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fab8c0845d0 con 0x7fab88004d10 2023-10-30T13:57:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.677+0000 7fab97fff640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab7c0664a0 0x7fab7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.677+0000 7fab97fff640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab7c0664a0 0x7fab7c068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fab98157710 tx=0x7fab84008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.681+0000 7fab957fa640 1 -- 172.21.15.31:0/4130139621 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab8c051590 con 0x7fab88004d10 2023-10-30T13:57:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.869+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab98156c60 con 0x7fab88004d10 2023-10-30T13:57:43.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.869+0000 7fab957fa640 1 -- 172.21.15.31:0/4130139621 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fab8c054bb0 con 0x7fab88004d10 2023-10-30T13:57:43.872 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:43.872 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:42.156930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:43.873 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2023-10-30T13:57:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab7c0664a0 msgr2=0x7fab7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab7c0664a0 0x7fab7c068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fab98157710 tx=0x7fab84008040 comp rx=0 tx=0).stop 2023-10-30T13:57:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 msgr2=0x7fab980c9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab980c9920 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fab8c000fc0 tx=0x7fab8c007720 comp rx=0 tx=0).stop 2023-10-30T13:57:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 shutdown_connections 2023-10-30T13:57:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fab7c0664a0 0x7fab7c068960 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981585e0 0x7fab9815a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fab980c8ae0 0x7fab9815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 --2- 172.21.15.31:0/4130139621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab88004d10 0x7fab980c9920 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 >> 172.21.15.31:0/4130139621 conn(0x7fab980cac70 msgr2=0x7fab980c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:43.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 shutdown_connections 2023-10-30T13:57:43.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:43.873+0000 7fab9ed9d640 1 -- 172.21.15.31:0/4130139621 wait complete. 2023-10-30T13:57:43.893 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 90, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 90, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6549": { "addr": "172.21.15.31:6837/1195492284", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1195492284, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1195492284, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6549, "incarnation": 90, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:57:42.156930+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6549 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:57:45.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:46.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 Processor -- start 2023-10-30T13:57:46.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 -- start start 2023-10-30T13:57:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faa40153b50 0x7faa401597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7faa400c62b0 0x7faa400ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400cb180 0x7faa4006ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa40159cf0 con 0x7faa40153b50 2023-10-30T13:57:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa4006aa30 con 0x7faa400c62b0 2023-10-30T13:57:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa47556640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4015a770 con 0x7faa400cb180 2023-10-30T13:57:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faa40153b50 0x7faa401597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7faa40153b50 0x7faa401597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44998/0 (socket says 172.21.15.31:44998) 2023-10-30T13:57:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 -- 172.21.15.31:0/3750277751 learned_addr learned my addr 172.21.15.31:0/3750277751 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa45d53640 1 --2- 172.21.15.31:0/3750277751 >> v2:172.21.15.33:3300/0 conn(0x7faa400c62b0 0x7faa400ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46d55640 1 --2- 172.21.15.31:0/3750277751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400cb180 0x7faa4006ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 -- 172.21.15.31:0/3750277751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400cb180 msgr2=0x7faa4006ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 --2- 172.21.15.31:0/3750277751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400cb180 0x7faa4006ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 -- 172.21.15.31:0/3750277751 >> v2:172.21.15.33:3300/0 conn(0x7faa400c62b0 msgr2=0x7faa400ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 --2- 172.21.15.31:0/3750277751 >> v2:172.21.15.33:3300/0 conn(0x7faa400c62b0 0x7faa400ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.049+0000 7faa46554640 1 -- 172.21.15.31:0/3750277751 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400c9d40 con 0x7faa40153b50 2023-10-30T13:57:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa46554640 1 --2- 172.21.15.31:0/3750277751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa40153b50 0x7faa401597b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7faa4006d180 tx=0x7faa3c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5f91c7d93ce104 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa45552640 1 -- 172.21.15.31:0/3750277751 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c0a89b0 con 0x7faa40153b50 2023-10-30T13:57:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa45552640 1 -- 172.21.15.31:0/3750277751 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c0a8b10 con 0x7faa40153b50 2023-10-30T13:57:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa45552640 1 -- 172.21.15.31:0/3750277751 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c0a8ea0 con 0x7faa40153b50 2023-10-30T13:57:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 -- 172.21.15.31:0/3750277751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa40153b50 msgr2=0x7faa401597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 --2- 172.21.15.31:0/3750277751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa40153b50 0x7faa401597b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7faa4006d180 tx=0x7faa3c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 -- 172.21.15.31:0/3750277751 shutdown_connections 2023-10-30T13:57:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 --2- 172.21.15.31:0/3750277751 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400cb180 0x7faa4006ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 --2- 172.21.15.31:0/3750277751 >> v2:172.21.15.33:3300/0 conn(0x7faa400c62b0 0x7faa400ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 --2- 172.21.15.31:0/3750277751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa40153b50 0x7faa401597b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 -- 172.21.15.31:0/3750277751 >> 172.21.15.31:0/3750277751 conn(0x7faa40065a80 msgr2=0x7faa40065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:46.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 -- 172.21.15.31:0/3750277751 shutdown_connections 2023-10-30T13:57:46.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.053+0000 7faa47556640 1 -- 172.21.15.31:0/3750277751 wait complete. 2023-10-30T13:57:46.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 Processor -- start 2023-10-30T13:57:46.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 -- start start 2023-10-30T13:57:46.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400c62b0 0x7faa400c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:46.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa400cb180 0x7faa401f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:46.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa40153b50 0x7faa401f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:46.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401f2fd0 con 0x7faa400cb180 2023-10-30T13:57:46.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401597e0 con 0x7faa40153b50 2023-10-30T13:57:46.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa401f2cf0 con 0x7faa400c62b0 2023-10-30T13:57:46.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa400cb180 0x7faa401f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:46.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa400cb180 0x7faa401f2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45004/0 (socket says 172.21.15.31:45004) 2023-10-30T13:57:46.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 -- 172.21.15.31:0/66872135 learned_addr learned my addr 172.21.15.31:0/66872135 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:46.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa46554640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400c62b0 0x7faa400c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:46.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa46d55640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa40153b50 0x7faa401f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:46.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 -- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400c62b0 msgr2=0x7faa400c23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:46.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400c62b0 0x7faa400c23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 -- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa40153b50 msgr2=0x7faa401f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:46.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa40153b50 0x7faa401f0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 -- 172.21.15.31:0/66872135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c0094d0 con 0x7faa400cb180 2023-10-30T13:57:46.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa45d53640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa400cb180 0x7faa401f2290 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7faa3000d7d0 tx=0x7faa3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:46.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa36ffd640 1 -- 172.21.15.31:0/66872135 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30004020 con 0x7faa400cb180 2023-10-30T13:57:46.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa36ffd640 1 -- 172.21.15.31:0/66872135 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30004180 con 0x7faa400cb180 2023-10-30T13:57:46.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa40159500 con 0x7faa400cb180 2023-10-30T13:57:46.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa36ffd640 1 -- 172.21.15.31:0/66872135 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30004330 con 0x7faa400cb180 2023-10-30T13:57:46.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4014cf90 con 0x7faa400cb180 2023-10-30T13:57:46.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.057+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa40067540 con 0x7faa400cb180 2023-10-30T13:57:46.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.061+0000 7faa36ffd640 1 -- 172.21.15.31:0/66872135 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa300096a0 con 0x7faa400cb180 2023-10-30T13:57:46.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.061+0000 7faa36ffd640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faa240664a0 0x7faa24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:46.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.061+0000 7faa36ffd640 1 -- 172.21.15.31:0/66872135 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7faa300844e0 con 0x7faa400cb180 2023-10-30T13:57:46.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.061+0000 7faa46554640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faa240664a0 0x7faa24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:46.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.061+0000 7faa46554640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faa240664a0 0x7faa24068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0b2000 tx=0x7faa3c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:46.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.065+0000 7faa36ffd640 1 -- 172.21.15.31:0/66872135 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa30051550 con 0x7faa400cb180 2023-10-30T13:57:46.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.257+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa400ca980 con 0x7faa400cb180 2023-10-30T13:57:46.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.257+0000 7faa36ffd640 1 -- 172.21.15.31:0/66872135 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5701 (secure 0 0 0) 0x7faa30054b70 con 0x7faa400cb180 2023-10-30T13:57:46.262 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:46.262 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:42.156930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:46.263 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2023-10-30T13:57:46.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faa240664a0 msgr2=0x7faa24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:46.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faa240664a0 0x7faa24068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0b2000 tx=0x7faa3c008040 comp rx=0 tx=0).stop 2023-10-30T13:57:46.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa400cb180 msgr2=0x7faa401f2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:46.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa400cb180 0x7faa401f2290 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7faa3000d7d0 tx=0x7faa3000dc90 comp rx=0 tx=0).stop 2023-10-30T13:57:46.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 shutdown_connections 2023-10-30T13:57:46.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7faa240664a0 0x7faa24068960 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa40153b50 0x7faa401f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa400cb180 0x7faa401f2290 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 --2- 172.21.15.31:0/66872135 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7faa400c62b0 0x7faa400c23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 >> 172.21.15.31:0/66872135 conn(0x7faa40065a80 msgr2=0x7faa400c7180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 shutdown_connections 2023-10-30T13:57:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:46.261+0000 7faa47556640 1 -- 172.21.15.31:0/66872135 wait complete. 2023-10-30T13:57:46.280 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:48.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:48.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb357491640 1 Processor -- start 2023-10-30T13:57:48.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb357491640 1 -- start start 2023-10-30T13:57:48.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb357491640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3500c4a10 0x7fb35015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb357491640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fb3500c42a0 0x7fb3500ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb357491640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb350159f20 con 0x7fb350153c80 2023-10-30T13:57:48.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb357491640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35015dec0 con 0x7fb3500c4a10 2023-10-30T13:57:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb357491640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3501599a0 con 0x7fb3500c42a0 2023-10-30T13:57:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb354a2c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3500c4a10 0x7fb35015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb354a2c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb3500c4a10 0x7fb35015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45008/0 (socket says 172.21.15.31:45008) 2023-10-30T13:57:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb354a2c640 1 -- 172.21.15.31:0/1844894193 learned_addr learned my addr 172.21.15.31:0/1844894193 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.429+0000 7fb355a2e640 1 --2- 172.21.15.31:0/1844894193 >> v2:172.21.15.33:3300/0 conn(0x7fb3500c42a0 0x7fb3500ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.433+0000 7fb354a2c640 1 -- 172.21.15.31:0/1844894193 >> v2:172.21.15.33:3300/0 conn(0x7fb3500c42a0 msgr2=0x7fb3500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:48.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.433+0000 7fb354a2c640 1 --2- 172.21.15.31:0/1844894193 >> v2:172.21.15.33:3300/0 conn(0x7fb3500c42a0 0x7fb3500ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.433+0000 7fb354a2c640 1 -- 172.21.15.31:0/1844894193 >> v1:172.21.15.33:6789/0 conn(0x7fb350153c80 legacy=0x7fb35015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.433+0000 7fb354a2c640 1 -- 172.21.15.31:0/1844894193 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35015d620 con 0x7fb3500c4a10 2023-10-30T13:57:48.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb354a2c640 1 --2- 172.21.15.31:0/1844894193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c4a10 0x7fb35015cf90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb3500b1d40 tx=0x7fb34001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e90a72c3b5e347f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:48.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb347fff640 1 -- 172.21.15.31:0/1844894193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3400a89f0 con 0x7fb3500c4a10 2023-10-30T13:57:48.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb347fff640 1 -- 172.21.15.31:0/1844894193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3400a8b50 con 0x7fb3500c4a10 2023-10-30T13:57:48.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb347fff640 1 -- 172.21.15.31:0/1844894193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3400b0440 con 0x7fb3500c4a10 2023-10-30T13:57:48.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 -- 172.21.15.31:0/1844894193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c4a10 msgr2=0x7fb35015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 --2- 172.21.15.31:0/1844894193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c4a10 0x7fb35015cf90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb3500b1d40 tx=0x7fb34001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 -- 172.21.15.31:0/1844894193 shutdown_connections 2023-10-30T13:57:48.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 --2- 172.21.15.31:0/1844894193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c4a10 0x7fb35015cf90 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 --2- 172.21.15.31:0/1844894193 >> v2:172.21.15.33:3300/0 conn(0x7fb3500c42a0 0x7fb3500ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 -- 172.21.15.31:0/1844894193 >> 172.21.15.31:0/1844894193 conn(0x7fb3500715c0 msgr2=0x7fb3500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 -- 172.21.15.31:0/1844894193 shutdown_connections 2023-10-30T13:57:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.437+0000 7fb357491640 1 -- 172.21.15.31:0/1844894193 wait complete. 2023-10-30T13:57:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 Processor -- start 2023-10-30T13:57:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 -- start start 2023-10-30T13:57:48.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c42a0 0x7fb350150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3500c4a10 0x7fb3500cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb350153c80 0x7fb350149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350151940 con 0x7fb3500c42a0 2023-10-30T13:57:48.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35014fd40 con 0x7fb3500c4a10 2023-10-30T13:57:48.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb350150020 con 0x7fb350153c80 2023-10-30T13:57:48.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c42a0 0x7fb350150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c42a0 0x7fb350150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45012/0 (socket says 172.21.15.31:45012) 2023-10-30T13:57:48.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 -- 172.21.15.31:0/3878038231 learned_addr learned my addr 172.21.15.31:0/3878038231 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:48.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb354a2c640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3500c4a10 0x7fb3500cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb355a2e640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb350153c80 0x7fb350149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 -- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb350153c80 msgr2=0x7fb350149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb350153c80 0x7fb350149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 -- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3500c4a10 msgr2=0x7fb3500cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3500c4a10 0x7fb3500cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 -- 172.21.15.31:0/3878038231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400094d0 con 0x7fb3500c42a0 2023-10-30T13:57:48.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb355a2e640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb350153c80 0x7fb350149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:48.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb35522d640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c42a0 0x7fb350150ff0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c3c0 tx=0x7fb34c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:48.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb345ffb640 1 -- 172.21.15.31:0/3878038231 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0052d0 con 0x7fb3500c42a0 2023-10-30T13:57:48.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb345ffb640 1 -- 172.21.15.31:0/3878038231 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c005430 con 0x7fb3500c42a0 2023-10-30T13:57:48.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb345ffb640 1 -- 172.21.15.31:0/3878038231 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c003840 con 0x7fb3500c42a0 2023-10-30T13:57:48.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3501f1f10 con 0x7fb3500c42a0 2023-10-30T13:57:48.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.441+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3501f24d0 con 0x7fb3500c42a0 2023-10-30T13:57:48.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.445+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35004e170 con 0x7fb3500c42a0 2023-10-30T13:57:48.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.445+0000 7fb345ffb640 1 -- 172.21.15.31:0/3878038231 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb34c0039a0 con 0x7fb3500c42a0 2023-10-30T13:57:48.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.449+0000 7fb345ffb640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3340664a0 0x7fb334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.449+0000 7fb345ffb640 1 -- 172.21.15.31:0/3878038231 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fb34c0837e0 con 0x7fb3500c42a0 2023-10-30T13:57:48.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.449+0000 7fb354a2c640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3340664a0 0x7fb334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.449+0000 7fb354a2c640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3340664a0 0x7fb334068960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb3400b2000 tx=0x7fb340008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:48.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.453+0000 7fb345ffb640 1 -- 172.21.15.31:0/3878038231 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb34c050840 con 0x7fb3500c42a0 2023-10-30T13:57:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.641+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3500cad40 con 0x7fb3500c42a0 2023-10-30T13:57:48.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb345ffb640 1 -- 172.21.15.31:0/3878038231 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fb34c017090 con 0x7fb3500c42a0 2023-10-30T13:57:48.645 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:48.646 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:42.156930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:48.646 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2023-10-30T13:57:48.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3340664a0 msgr2=0x7fb334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3340664a0 0x7fb334068960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb3400b2000 tx=0x7fb340008040 comp rx=0 tx=0).stop 2023-10-30T13:57:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c42a0 msgr2=0x7fb350150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c42a0 0x7fb350150ff0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c3c0 tx=0x7fb34c00c880 comp rx=0 tx=0).stop 2023-10-30T13:57:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 shutdown_connections 2023-10-30T13:57:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb3340664a0 0x7fb334068960 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb350153c80 0x7fb350149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3500c4a10 0x7fb3500cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 --2- 172.21.15.31:0/3878038231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3500c42a0 0x7fb350150ff0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 >> 172.21.15.31:0/3878038231 conn(0x7fb3500715c0 msgr2=0x7fb3500c4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 shutdown_connections 2023-10-30T13:57:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:48.645+0000 7fb357491640 1 -- 172.21.15.31:0/3878038231 wait complete. 2023-10-30T13:57:48.668 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:50.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:50.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.813+0000 7f8a07091640 1 Processor -- start 2023-10-30T13:57:50.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a07091640 1 -- start start 2023-10-30T13:57:50.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a07091640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8a00153c80 0x7f8a0015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a07091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a000c4a10 0x7f8a000c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a07091640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a00159f20 con 0x7f8a0015a3b0 2023-10-30T13:57:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a07091640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0015dec0 con 0x7f8a00153c80 2023-10-30T13:57:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a07091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a001599a0 con 0x7f8a000c4a10 2023-10-30T13:57:50.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a04e2d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8a00153c80 0x7f8a0015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f89fffff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a000c4a10 0x7f8a000c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a04e2d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8a00153c80 0x7f8a0015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:37008/0 (socket says 172.21.15.31:37008) 2023-10-30T13:57:50.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f8a04e2d640 1 -- 172.21.15.31:0/3728387119 learned_addr learned my addr 172.21.15.31:0/3728387119 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:50.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f89fffff640 1 -- 172.21.15.31:0/3728387119 >> v2:172.21.15.33:3300/0 conn(0x7f8a00153c80 msgr2=0x7f8a0015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:50.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f89fffff640 1 --2- 172.21.15.31:0/3728387119 >> v2:172.21.15.33:3300/0 conn(0x7f8a00153c80 0x7f8a0015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f89fffff640 1 -- 172.21.15.31:0/3728387119 >> v1:172.21.15.33:6789/0 conn(0x7f8a0015a3b0 legacy=0x7f8a000ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.817+0000 7f89fffff640 1 -- 172.21.15.31:0/3728387119 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a000c47e0 con 0x7f8a000c4a10 2023-10-30T13:57:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f89fffff640 1 --2- 172.21.15.31:0/3728387119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a000c4a10 0x7f8a000c42a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0097c0 tx=0x7f89ec01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a86c11e67c68da14 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f89ff7fe640 1 -- 172.21.15.31:0/3728387119 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0a8970 con 0x7f8a000c4a10 2023-10-30T13:57:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f89ff7fe640 1 -- 172.21.15.31:0/3728387119 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec0a8ad0 con 0x7f8a000c4a10 2023-10-30T13:57:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f89ff7fe640 1 -- 172.21.15.31:0/3728387119 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0a8e60 con 0x7f8a000c4a10 2023-10-30T13:57:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f8a07091640 1 -- 172.21.15.31:0/3728387119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a000c4a10 msgr2=0x7f8a000c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f8a07091640 1 --2- 172.21.15.31:0/3728387119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a000c4a10 0x7f8a000c42a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0097c0 tx=0x7f89ec01ec70 comp rx=0 tx=0).stop 2023-10-30T13:57:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f8a07091640 1 -- 172.21.15.31:0/3728387119 shutdown_connections 2023-10-30T13:57:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f8a07091640 1 --2- 172.21.15.31:0/3728387119 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a000c4a10 0x7f8a000c42a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f8a07091640 1 --2- 172.21.15.31:0/3728387119 >> v2:172.21.15.33:3300/0 conn(0x7f8a00153c80 0x7f8a0015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.821+0000 7f8a07091640 1 -- 172.21.15.31:0/3728387119 >> 172.21.15.31:0/3728387119 conn(0x7f8a000715c0 msgr2=0x7f8a000719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.825+0000 7f8a07091640 1 -- 172.21.15.31:0/3728387119 shutdown_connections 2023-10-30T13:57:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.825+0000 7f8a07091640 1 -- 172.21.15.31:0/3728387119 wait complete. 2023-10-30T13:57:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 Processor -- start 2023-10-30T13:57:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 -- start start 2023-10-30T13:57:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a000c4a10 0x7f8a001ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a00153c80 0x7f8a0015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a0015a3b0 0x7f8a0006c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a001efd60 con 0x7f8a00153c80 2023-10-30T13:57:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0014b820 con 0x7f8a000c4a10 2023-10-30T13:57:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.829+0000 7f8a07091640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0015b0a0 con 0x7f8a0015a3b0 2023-10-30T13:57:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a00153c80 0x7f8a0015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a00153c80 0x7f8a0015ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45022/0 (socket says 172.21.15.31:45022) 2023-10-30T13:57:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 -- 172.21.15.31:0/2213131010 learned_addr learned my addr 172.21.15.31:0/2213131010 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f8a04e2d640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a000c4a10 0x7f8a001ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f8a0562e640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a0015a3b0 0x7f8a0006c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 -- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a0015a3b0 msgr2=0x7f8a0006c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a0015a3b0 0x7f8a0006c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 -- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a000c4a10 msgr2=0x7f8a001ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a000c4a10 0x7f8a001ef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 -- 172.21.15.31:0/2213131010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec009420 con 0x7f8a00153c80 2023-10-30T13:57:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fffff640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a00153c80 0x7f8a0015ab60 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009710 tx=0x7f89ec01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fd7fa640 1 -- 172.21.15.31:0/2213131010 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0b1e20 con 0x7f8a00153c80 2023-10-30T13:57:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fd7fa640 1 -- 172.21.15.31:0/2213131010 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec0b9410 con 0x7f8a00153c80 2023-10-30T13:57:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fd7fa640 1 -- 172.21.15.31:0/2213131010 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0b97c0 con 0x7f8a00153c80 2023-10-30T13:57:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0015cf10 con 0x7f8a00153c80 2023-10-30T13:57:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0014b1d0 con 0x7f8a00153c80 2023-10-30T13:57:50.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0004e170 con 0x7f8a00153c80 2023-10-30T13:57:50.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.833+0000 7f89fd7fa640 1 -- 172.21.15.31:0/2213131010 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89ec0b7040 con 0x7f8a00153c80 2023-10-30T13:57:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.837+0000 7f89fd7fa640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89e40664a0 0x7f89e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.837+0000 7f89fd7fa640 1 -- 172.21.15.31:0/2213131010 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f89ec121e70 con 0x7f8a00153c80 2023-10-30T13:57:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.837+0000 7f8a04e2d640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89e40664a0 0x7f89e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.837+0000 7f8a04e2d640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89e40664a0 0x7f89e4068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009b50 tx=0x7f89f4009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:50.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:50.841+0000 7f89fd7fa640 1 -- 172.21.15.31:0/2213131010 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f89ec0eee50 con 0x7f8a00153c80 2023-10-30T13:57:51.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a0014b960 con 0x7f8a00153c80 2023-10-30T13:57:51.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f89fd7fa640 1 -- 172.21.15.31:0/2213131010 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5701 (secure 0 0 0) 0x7f89ec0b5030 con 0x7f8a00153c80 2023-10-30T13:57:51.234 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:51.234 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:42.156930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:51.235 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2023-10-30T13:57:51.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89e40664a0 msgr2=0x7f89e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:51.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89e40664a0 0x7f89e4068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009b50 tx=0x7f89f4009240 comp rx=0 tx=0).stop 2023-10-30T13:57:51.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a00153c80 msgr2=0x7f8a0015ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:51.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a00153c80 0x7f8a0015ab60 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009710 tx=0x7f89ec01ba10 comp rx=0 tx=0).stop 2023-10-30T13:57:51.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 shutdown_connections 2023-10-30T13:57:51.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f89e40664a0 0x7f89e4068960 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:51.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8a0015a3b0 0x7f8a0006c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:51.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a00153c80 0x7f8a0015ab60 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:51.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 --2- 172.21.15.31:0/2213131010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a000c4a10 0x7f8a001ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:51.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 >> 172.21.15.31:0/2213131010 conn(0x7f8a000715c0 msgr2=0x7f8a000719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:51.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 shutdown_connections 2023-10-30T13:57:51.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:51.233+0000 7f8a07091640 1 -- 172.21.15.31:0/2213131010 wait complete. 2023-10-30T13:57:51.258 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:53.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:53.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.401+0000 7fd2793d3640 1 Processor -- start 2023-10-30T13:57:53.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2793d3640 1 -- start start 2023-10-30T13:57:53.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2793d3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd274153c80 0x7fd27415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2793d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2740c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2793d3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd27415a3b0 0x7fd2740ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2793d3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27415dec0 con 0x7fd27415a3b0 2023-10-30T13:57:53.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2793d3640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2741599a0 con 0x7fd274153c80 2023-10-30T13:57:53.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2793d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd274159f20 con 0x7fd2740c4a10 2023-10-30T13:57:53.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd27415a3b0 0x7fd2740ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd27415a3b0 0x7fd2740ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45038/0 (socket says 172.21.15.31:45038) 2023-10-30T13:57:53.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 -- 172.21.15.31:0/4240034694 learned_addr learned my addr 172.21.15.31:0/4240034694 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:53.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd272ffd640 1 --2- 172.21.15.31:0/4240034694 >> v2:172.21.15.33:3300/0 conn(0x7fd274153c80 0x7fd27415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2727fc640 1 --2- 172.21.15.31:0/4240034694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2740c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 -- 172.21.15.31:0/4240034694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 msgr2=0x7fd2740c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:53.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 --2- 172.21.15.31:0/4240034694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2740c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 -- 172.21.15.31:0/4240034694 >> v2:172.21.15.33:3300/0 conn(0x7fd274153c80 msgr2=0x7fd27415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:53.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 --2- 172.21.15.31:0/4240034694 >> v2:172.21.15.33:3300/0 conn(0x7fd274153c80 0x7fd27415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2737fe640 1 -- 172.21.15.31:0/4240034694 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740c47e0 con 0x7fd27415a3b0 2023-10-30T13:57:53.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.405+0000 7fd2727fc640 1 --2- 172.21.15.31:0/4240034694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2740c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2737fe640 1 --2- 172.21.15.31:0/4240034694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd27415a3b0 0x7fd2740ca1e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd2740d00f0 tx=0x7fd26c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa7147ca5ca5271 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd271ffb640 1 -- 172.21.15.31:0/4240034694 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0af400 con 0x7fd27415a3b0 2023-10-30T13:57:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd271ffb640 1 -- 172.21.15.31:0/4240034694 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c0ab050 con 0x7fd27415a3b0 2023-10-30T13:57:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd271ffb640 1 -- 172.21.15.31:0/4240034694 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0af780 con 0x7fd27415a3b0 2023-10-30T13:57:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 -- 172.21.15.31:0/4240034694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd27415a3b0 msgr2=0x7fd2740ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 --2- 172.21.15.31:0/4240034694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd27415a3b0 0x7fd2740ca1e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd2740d00f0 tx=0x7fd26c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 -- 172.21.15.31:0/4240034694 shutdown_connections 2023-10-30T13:57:53.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 --2- 172.21.15.31:0/4240034694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd27415a3b0 0x7fd2740ca1e0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 --2- 172.21.15.31:0/4240034694 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2740c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 --2- 172.21.15.31:0/4240034694 >> v2:172.21.15.33:3300/0 conn(0x7fd274153c80 0x7fd27415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 -- 172.21.15.31:0/4240034694 >> 172.21.15.31:0/4240034694 conn(0x7fd2740715c0 msgr2=0x7fd2740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:53.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 -- 172.21.15.31:0/4240034694 shutdown_connections 2023-10-30T13:57:53.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.409+0000 7fd2793d3640 1 -- 172.21.15.31:0/4240034694 wait complete. 2023-10-30T13:57:53.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 Processor -- start 2023-10-30T13:57:53.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 -- start start 2023-10-30T13:57:53.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2741eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd274153c80 0x7fd274151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd27415a3b0 0x7fd27415b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27415afa0 con 0x7fd274153c80 2023-10-30T13:57:53.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740b3490 con 0x7fd27415a3b0 2023-10-30T13:57:53.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2793d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27415ac00 con 0x7fd2740c4a10 2023-10-30T13:57:53.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd274153c80 0x7fd274151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd274153c80 0x7fd274151d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45050/0 (socket says 172.21.15.31:45050) 2023-10-30T13:57:53.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 -- 172.21.15.31:0/119331009 learned_addr learned my addr 172.21.15.31:0/119331009 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:53.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2737fe640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd27415a3b0 0x7fd27415b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd272ffd640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2741eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 -- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 msgr2=0x7fd2741eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2741eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 -- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd27415a3b0 msgr2=0x7fd27415b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd27415a3b0 0x7fd27415b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd2727fc640 1 -- 172.21.15.31:0/119331009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c009b30 con 0x7fd274153c80 2023-10-30T13:57:53.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.413+0000 7fd272ffd640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2741eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:53.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd2727fc640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd274153c80 0x7fd274151d10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fd26800e7e0 tx=0x7fd26800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:53.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd25b7fe640 1 -- 172.21.15.31:0/119331009 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268004030 con 0x7fd274153c80 2023-10-30T13:57:53.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd25b7fe640 1 -- 172.21.15.31:0/119331009 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd268004190 con 0x7fd274153c80 2023-10-30T13:57:53.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2741ef440 con 0x7fd274153c80 2023-10-30T13:57:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd25b7fe640 1 -- 172.21.15.31:0/119331009 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268004500 con 0x7fd274153c80 2023-10-30T13:57:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2741efa00 con 0x7fd274153c80 2023-10-30T13:57:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27415b0e0 con 0x7fd274153c80 2023-10-30T13:57:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd25b7fe640 1 -- 172.21.15.31:0/119331009 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd26800c6d0 con 0x7fd274153c80 2023-10-30T13:57:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd25b7fe640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd2600664d0 0x7fd260068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.417+0000 7fd25b7fe640 1 -- 172.21.15.31:0/119331009 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fd268083da0 con 0x7fd274153c80 2023-10-30T13:57:53.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.421+0000 7fd272ffd640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd2600664d0 0x7fd260068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.421+0000 7fd272ffd640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd2600664d0 0x7fd260068990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd264009bb0 tx=0x7fd264008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:53.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.421+0000 7fd25b7fe640 1 -- 172.21.15.31:0/119331009 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd27415b0e0 con 0x7fd274153c80 2023-10-30T13:57:53.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.613+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd27415b0e0 con 0x7fd274153c80 2023-10-30T13:57:53.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.613+0000 7fd25b7fe640 1 -- 172.21.15.31:0/119331009 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5701 (secure 0 0 0) 0x7fd27415b0e0 con 0x7fd274153c80 2023-10-30T13:57:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:53.616 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:42.156930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:53.617 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2023-10-30T13:57:53.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd2600664d0 msgr2=0x7fd260068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd2600664d0 0x7fd260068990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd264009bb0 tx=0x7fd264008040 comp rx=0 tx=0).stop 2023-10-30T13:57:53.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd274153c80 msgr2=0x7fd274151d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd274153c80 0x7fd274151d10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fd26800e7e0 tx=0x7fd26800eca0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 shutdown_connections 2023-10-30T13:57:53.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd2600664d0 0x7fd260068990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd27415a3b0 0x7fd27415b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd274153c80 0x7fd274151d10 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 --2- 172.21.15.31:0/119331009 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd2740c4a10 0x7fd2741eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 >> 172.21.15.31:0/119331009 conn(0x7fd2740715c0 msgr2=0x7fd274152f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:53.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 shutdown_connections 2023-10-30T13:57:53.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:53.617+0000 7fd2793d3640 1 -- 172.21.15.31:0/119331009 wait complete. 2023-10-30T13:57:53.646 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:55.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26e446640 1 Processor -- start 2023-10-30T13:57:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26e446640 1 -- start start 2023-10-30T13:57:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26e446640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa268070520 0x7fa268070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26e446640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2680c2310 con 0x7fa2680cb7d0 2023-10-30T13:57:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26e446640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa268156710 con 0x7fa268150830 2023-10-30T13:57:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26e446640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26806bc10 con 0x7fa268070520 2023-10-30T13:57:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26c9e3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa268070520 0x7fa268070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26c9e3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa268070520 0x7fa268070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43948/0 (socket says 172.21.15.31:43948) 2023-10-30T13:57:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26c9e3640 1 -- 172.21.15.31:0/832269862 learned_addr learned my addr 172.21.15.31:0/832269862 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:55.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26c9e3640 1 -- 172.21.15.31:0/832269862 >> v1:172.21.15.33:6789/0 conn(0x7fa268150830 legacy=0x7fa26806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26c9e3640 1 -- 172.21.15.31:0/832269862 >> v1:172.21.15.31:6789/0 conn(0x7fa2680cb7d0 legacy=0x7fa2680c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa26c9e3640 1 -- 172.21.15.31:0/832269862 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2680c4360 con 0x7fa268070520 2023-10-30T13:57:55.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.805+0000 7fa266ffd640 1 -- 172.21.15.31:0/832269862 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998009748 0 0) 0x7fa2680c2310 con 0x7fa2680cb7d0 2023-10-30T13:57:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26c9e3640 1 --2- 172.21.15.31:0/832269862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa268070990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa260009e50 tx=0x7fa26001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd217edb4f9cd3bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa266ffd640 1 -- 172.21.15.31:0/832269862 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600af400 con 0x7fa268070520 2023-10-30T13:57:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa266ffd640 1 -- 172.21.15.31:0/832269862 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2600ab050 con 0x7fa268070520 2023-10-30T13:57:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa266ffd640 1 -- 172.21.15.31:0/832269862 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600af780 con 0x7fa268070520 2023-10-30T13:57:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26e446640 1 -- 172.21.15.31:0/832269862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 msgr2=0x7fa268070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26e446640 1 --2- 172.21.15.31:0/832269862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa268070990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa260009e50 tx=0x7fa26001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26e446640 1 -- 172.21.15.31:0/832269862 shutdown_connections 2023-10-30T13:57:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26e446640 1 --2- 172.21.15.31:0/832269862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa268070990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26e446640 1 -- 172.21.15.31:0/832269862 >> 172.21.15.31:0/832269862 conn(0x7fa2680cac70 msgr2=0x7fa2680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26e446640 1 -- 172.21.15.31:0/832269862 shutdown_connections 2023-10-30T13:57:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.809+0000 7fa26e446640 1 -- 172.21.15.31:0/832269862 wait complete. 2023-10-30T13:57:55.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 Processor -- start 2023-10-30T13:57:55.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 -- start start 2023-10-30T13:57:55.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa26815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2680cb7d0 0x7fa268149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa268150830 0x7fa26814dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681ee830 con 0x7fa268070520 2023-10-30T13:57:55.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681f1d70 con 0x7fa2680cb7d0 2023-10-30T13:57:55.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681eea40 con 0x7fa268150830 2023-10-30T13:57:55.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa26815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa26815a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43962/0 (socket says 172.21.15.31:43962) 2023-10-30T13:57:55.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 -- 172.21.15.31:0/3690483293 learned_addr learned my addr 172.21.15.31:0/3690483293 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:55.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26c9e3640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa268150830 0x7fa26814dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa2677fe640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2680cb7d0 0x7fa268149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 -- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa268150830 msgr2=0x7fa26814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa268150830 0x7fa26814dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 -- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2680cb7d0 msgr2=0x7fa268149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2680cb7d0 0x7fa268149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 -- 172.21.15.31:0/3690483293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa260009b30 con 0x7fa268070520 2023-10-30T13:57:55.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26c9e3640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa268150830 0x7fa26814dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:55.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa267fff640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa26815a660 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa25800c3c0 tx=0x7fa25800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:55.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa264ff9640 1 -- 172.21.15.31:0/3690483293 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2580052d0 con 0x7fa268070520 2023-10-30T13:57:55.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa264ff9640 1 -- 172.21.15.31:0/3690483293 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa258005430 con 0x7fa268070520 2023-10-30T13:57:55.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2681eed20 con 0x7fa268070520 2023-10-30T13:57:55.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa264ff9640 1 -- 172.21.15.31:0/3690483293 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258003840 con 0x7fa268070520 2023-10-30T13:57:55.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2681ef2e0 con 0x7fa268070520 2023-10-30T13:57:55.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.813+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26804e170 con 0x7fa268070520 2023-10-30T13:57:55.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.817+0000 7fa264ff9640 1 -- 172.21.15.31:0/3690483293 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2580039a0 con 0x7fa268070520 2023-10-30T13:57:55.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.817+0000 7fa264ff9640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa2540664a0 0x7fa254068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.817+0000 7fa264ff9640 1 -- 172.21.15.31:0/3690483293 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fa2580838a0 con 0x7fa268070520 2023-10-30T13:57:55.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.817+0000 7fa2677fe640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa2540664a0 0x7fa254068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.817+0000 7fa2677fe640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa2540664a0 0x7fa254068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa26814d280 tx=0x7fa25c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:55.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:55.821+0000 7fa264ff9640 1 -- 172.21.15.31:0/3690483293 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa258050880 con 0x7fa268070520 2023-10-30T13:57:56.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.009+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa268070300 con 0x7fa268070520 2023-10-30T13:57:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.009+0000 7fa264ff9640 1 -- 172.21.15.31:0/3690483293 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5705 (secure 0 0 0) 0x7fa258017090 con 0x7fa268070520 2023-10-30T13:57:56.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:56.012 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:54.034478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 91 2023-10-30T13:57:56.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa2540664a0 msgr2=0x7fa254068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:56.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa2540664a0 0x7fa254068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa26814d280 tx=0x7fa25c009040 comp rx=0 tx=0).stop 2023-10-30T13:57:56.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 msgr2=0x7fa26815a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:56.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa26815a660 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa25800c3c0 tx=0x7fa25800c880 comp rx=0 tx=0).stop 2023-10-30T13:57:56.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 shutdown_connections 2023-10-30T13:57:56.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa2540664a0 0x7fa254068960 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:56.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa268150830 0x7fa26814dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:56.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2680cb7d0 0x7fa268149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:56.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 --2- 172.21.15.31:0/3690483293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa268070520 0x7fa26815a660 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:56.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 >> 172.21.15.31:0/3690483293 conn(0x7fa2680cac70 msgr2=0x7fa26814ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:56.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 shutdown_connections 2023-10-30T13:57:56.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:56.013+0000 7fa26e446640 1 -- 172.21.15.31:0/3690483293 wait complete. 2023-10-30T13:57:56.034 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6549": { "addr": "172.21.15.31:6837/1195492284", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1195492284, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1195492284, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6549, "incarnation": 90, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:57:54.034478+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6549 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:57:58.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f45a1b3e640 1 Processor -- start 2023-10-30T13:57:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f45a1b3e640 1 -- start start 2023-10-30T13:57:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f45a1b3e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f459c0c4a10 0x7f459c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f45a1b3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 0x7f459c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f45a1b3e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c159f20 con 0x7f459c153c80 2023-10-30T13:57:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f45a1b3e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c15dec0 con 0x7f459c0c4a10 2023-10-30T13:57:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f45a1b3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1599a0 con 0x7f459c0c42a0 2023-10-30T13:57:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f459c153c80 0x7f459c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51516/0 (socket says 172.21.15.31:51516) 2023-10-30T13:57:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459b7fe640 1 -- 172.21.15.31:0/3586796243 learned_addr learned my addr 172.21.15.31:0/3586796243 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:57:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459affd640 1 --2- 172.21.15.31:0/3586796243 >> v2:172.21.15.33:3300/0 conn(0x7f459c0c4a10 0x7f459c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459bfff640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 0x7f459c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604234163 0 0) 0x7f459c159f20 con 0x7f459c153c80 2023-10-30T13:57:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4584003530 con 0x7f459c153c80 2023-10-30T13:57:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902494136 0 0) 0x7f4584003530 con 0x7f459c153c80 2023-10-30T13:57:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 msgr2=0x7f459c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 0x7f459c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 >> v2:172.21.15.33:3300/0 conn(0x7f459c0c4a10 msgr2=0x7f459c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 --2- 172.21.15.31:0/3586796243 >> v2:172.21.15.33:3300/0 conn(0x7f459c0c4a10 0x7f459c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c15d620 con 0x7f459c153c80 2023-10-30T13:57:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f458c001680 con 0x7f459c153c80 2023-10-30T13:57:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 >> v1:172.21.15.31:6789/0 conn(0x7f459c153c80 legacy=0x7f459c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 0x7f4584003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.189+0000 7f459a7fc640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f45840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459a7fc640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4584004d10 0x7f4584005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c15dec0 con 0x7f4584003f00 2023-10-30T13:57:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1599a0 con 0x7f4584004d10 2023-10-30T13:57:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458400b9c0 con 0x7f459c0c42a0 2023-10-30T13:57:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459affd640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f45840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459b7fe640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 0x7f4584003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459bfff640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4584004d10 0x7f4584005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459affd640 1 -- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 msgr2=0x7f4584003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459affd640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c0c42a0 0x7f4584003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459affd640 1 -- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4584004d10 msgr2=0x7f4584005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459affd640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4584004d10 0x7f4584005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459affd640 1 -- 172.21.15.31:0/3586796243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584003530 con 0x7f4584003f00 2023-10-30T13:57:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459bfff640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4584004d10 0x7f4584005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459affd640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f45840047d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f4590009ab0 tx=0x7f459001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3699913e5db7b2df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45900a5660 con 0x7f4584003f00 2023-10-30T13:57:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45900a57c0 con 0x7f4584003f00 2023-10-30T13:57:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.193+0000 7f459a7fc640 1 -- 172.21.15.31:0/3586796243 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45900a5bb0 con 0x7f4584003f00 2023-10-30T13:57:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- 172.21.15.31:0/3586796243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 msgr2=0x7f45840047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 --2- 172.21.15.31:0/3586796243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f45840047d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f4590009ab0 tx=0x7f459001ca70 comp rx=0 tx=0).stop 2023-10-30T13:57:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f459bfff640 1 -- 172.21.15.31:0/3586796243 reap_dead start 2023-10-30T13:57:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- 172.21.15.31:0/3586796243 shutdown_connections 2023-10-30T13:57:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- 172.21.15.31:0/3586796243 >> 172.21.15.31:0/3586796243 conn(0x7f459c0715c0 msgr2=0x7f459c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- 172.21.15.31:0/3586796243 shutdown_connections 2023-10-30T13:57:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- 172.21.15.31:0/3586796243 wait complete. 2023-10-30T13:57:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 Processor -- start 2023-10-30T13:57:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- start start 2023-10-30T13:57:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f459c0b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c15c030 0x7f459c15c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f459c15b360 0x7f459c15b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c146a30 con 0x7f4584003f00 2023-10-30T13:57:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0b38a0 con 0x7f459c15b360 2023-10-30T13:57:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f45a1b3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c14f020 con 0x7f459c15c030 2023-10-30T13:57:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f459c0b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f459c0b29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43970/0 (socket says 172.21.15.31:43970) 2023-10-30T13:57:58.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f459b7fe640 1 -- 172.21.15.31:0/657812381 learned_addr learned my addr 172.21.15.31:0/657812381 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:57:58.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.197+0000 7f459bfff640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f459c15b360 0x7f459c15b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f459affd640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c15c030 0x7f459c15c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f459b7fe640 1 -- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c15c030 msgr2=0x7f459c15c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f459b7fe640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c15c030 0x7f459c15c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f459b7fe640 1 -- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f459c15b360 msgr2=0x7f459c15b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f459b7fe640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f459c15b360 0x7f459c15b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f459b7fe640 1 -- 172.21.15.31:0/657812381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c14f300 con 0x7f4584003f00 2023-10-30T13:57:58.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f459b7fe640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f459c0b29e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f458c00e880 tx=0x7f458c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:58.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f4583fff640 1 -- 172.21.15.31:0/657812381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c00c8d0 con 0x7f4584003f00 2023-10-30T13:57:58.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f4583fff640 1 -- 172.21.15.31:0/657812381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c00ca50 con 0x7f4584003f00 2023-10-30T13:57:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f4583fff640 1 -- 172.21.15.31:0/657812381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c00ce00 con 0x7f4584003f00 2023-10-30T13:57:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c14f5e0 con 0x7f4584003f00 2023-10-30T13:57:58.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c15ab60 con 0x7f4584003f00 2023-10-30T13:57:58.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c04e170 con 0x7f4584003f00 2023-10-30T13:57:58.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f4583fff640 1 -- 172.21.15.31:0/657812381 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f458c01d460 con 0x7f4584003f00 2023-10-30T13:57:58.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f4583fff640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f45880664a0 0x7f4588068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:58.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.201+0000 7f4583fff640 1 -- 172.21.15.31:0/657812381 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f458c084540 con 0x7f4584003f00 2023-10-30T13:57:58.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.205+0000 7f459affd640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f45880664a0 0x7f4588068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:58.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.205+0000 7f459affd640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f45880664a0 0x7f4588068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45900af000 tx=0x7f45900046d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:58.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.205+0000 7f4583fff640 1 -- 172.21.15.31:0/657812381 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f458c051500 con 0x7f4584003f00 2023-10-30T13:57:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.401+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f459c1595f0 con 0x7f4584003f00 2023-10-30T13:57:58.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.401+0000 7f4583fff640 1 -- 172.21.15.31:0/657812381 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5705 (secure 0 0 0) 0x7f458c054b20 con 0x7f4584003f00 2023-10-30T13:57:58.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:57:58.403 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:54.034478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:58.404 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 91 2023-10-30T13:57:58.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f45880664a0 msgr2=0x7f4588068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f45880664a0 0x7f4588068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45900af000 tx=0x7f45900046d0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 msgr2=0x7f459c0b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f459c0b29e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f458c00e880 tx=0x7f458c00ed40 comp rx=0 tx=0).stop 2023-10-30T13:57:58.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 shutdown_connections 2023-10-30T13:57:58.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f45880664a0 0x7f4588068960 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f459c15b360 0x7f459c15b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f459c15c030 0x7f459c15c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 --2- 172.21.15.31:0/657812381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4584003f00 0x7f459c0b29e0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 >> 172.21.15.31:0/657812381 conn(0x7f459c0715c0 msgr2=0x7f459c157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:58.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 shutdown_connections 2023-10-30T13:57:58.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:57:58.405+0000 7f45a1b3e640 1 -- 172.21.15.31:0/657812381 wait complete. 2023-10-30T13:57:58.424 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:59.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:59.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:00.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:00.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6efcc9640 1 Processor -- start 2023-10-30T13:58:00.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6efcc9640 1 -- start start 2023-10-30T13:58:00.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6efcc9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd6e8070520 0x7fd6e8070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6efcc9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e80c2310 con 0x7fd6e8150830 2023-10-30T13:58:00.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6efcc9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e8156710 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6efcc9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e806bc10 con 0x7fd6e8070520 2023-10-30T13:58:00.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6ee266640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd6e8070520 0x7fd6e8070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6ee266640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fd6e8070520 0x7fd6e8070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43984/0 (socket says 172.21.15.31:43984) 2023-10-30T13:58:00.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6eda65640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fd6e8150830 0x7fd6e806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51526/0 (socket says 172.21.15.31:51526) 2023-10-30T13:58:00.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6ee266640 1 -- 172.21.15.31:0/3234981093 learned_addr learned my addr 172.21.15.31:0/3234981093 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:00.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6ee266640 1 -- 172.21.15.31:0/3234981093 >> v1:172.21.15.33:6789/0 conn(0x7fd6e80cb7d0 legacy=0x7fd6e80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6ee266640 1 -- 172.21.15.31:0/3234981093 >> v1:172.21.15.31:6789/0 conn(0x7fd6e8150830 legacy=0x7fd6e806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6ee266640 1 -- 172.21.15.31:0/3234981093 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e80c4360 con 0x7fd6e8070520 2023-10-30T13:58:00.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.577+0000 7fd6eca63640 1 -- 172.21.15.31:0/3234981093 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184075032 0 0) 0x7fd6e80c2310 con 0x7fd6e8150830 2023-10-30T13:58:00.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6ee266640 1 --2- 172.21.15.31:0/3234981093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8070520 0x7fd6e8070990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009e50 tx=0x7fd6e001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f8e82046be1b25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:00.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6eca63640 1 -- 172.21.15.31:0/3234981093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00af400 con 0x7fd6e8070520 2023-10-30T13:58:00.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6eca63640 1 -- 172.21.15.31:0/3234981093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e00ab050 con 0x7fd6e8070520 2023-10-30T13:58:00.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6eca63640 1 -- 172.21.15.31:0/3234981093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00af780 con 0x7fd6e8070520 2023-10-30T13:58:00.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3234981093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8070520 msgr2=0x7fd6e8070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3234981093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8070520 0x7fd6e8070990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009e50 tx=0x7fd6e001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3234981093 shutdown_connections 2023-10-30T13:58:00.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3234981093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8070520 0x7fd6e8070990 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3234981093 >> 172.21.15.31:0/3234981093 conn(0x7fd6e80cac70 msgr2=0x7fd6e80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:00.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3234981093 shutdown_connections 2023-10-30T13:58:00.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.581+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3234981093 wait complete. 2023-10-30T13:58:00.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 Processor -- start 2023-10-30T13:58:00.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 -- start start 2023-10-30T13:58:00.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e8070520 0x7fd6e8153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e80cb7d0 0x7fd6e81ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6e8150830 0x7fd6e806bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e81ef6f0 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e81f1ce0 con 0x7fd6e8070520 2023-10-30T13:58:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e8153ed0 con 0x7fd6e8150830 2023-10-30T13:58:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e80cb7d0 0x7fd6e81ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e80cb7d0 0x7fd6e81ef1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43988/0 (socket says 172.21.15.31:43988) 2023-10-30T13:58:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 -- 172.21.15.31:0/3730234760 learned_addr learned my addr 172.21.15.31:0/3730234760 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6eda65640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e8070520 0x7fd6e8153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ee266640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6e8150830 0x7fd6e806bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 -- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6e8150830 msgr2=0x7fd6e806bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6e8150830 0x7fd6e806bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 -- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e8070520 msgr2=0x7fd6e8153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e8070520 0x7fd6e8153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 -- 172.21.15.31:0/3730234760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e0009b30 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6eda65640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e8070520 0x7fd6e8153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:00.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6ed264640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e80cb7d0 0x7fd6e81ef1b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800c550 tx=0x7fd6d800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:00.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6de7fc640 1 -- 172.21.15.31:0/3730234760 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8012830 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6de7fc640 1 -- 172.21.15.31:0/3730234760 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d8012990 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6de7fc640 1 -- 172.21.15.31:0/3730234760 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8012d40 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e81f1f80 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.585+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e81f2540 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.589+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e804e170 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.589+0000 7fd6de7fc640 1 -- 172.21.15.31:0/3730234760 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6d801b400 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.589+0000 7fd6de7fc640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6cc0664a0 0x7fd6cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.589+0000 7fd6de7fc640 1 -- 172.21.15.31:0/3730234760 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fd6d8084640 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.589+0000 7fd6eda65640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6cc0664a0 0x7fd6cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.593+0000 7fd6eda65640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6cc0664a0 0x7fd6cc068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4000f70 tx=0x7fd6e4007df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.593+0000 7fd6de7fc640 1 -- 172.21.15.31:0/3730234760 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6d8051570 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.785+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6e80c6c60 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.785+0000 7fd6de7fc640 1 -- 172.21.15.31:0/3730234760 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fd6d8016020 con 0x7fd6e80cb7d0 2023-10-30T13:58:00.789 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:00.789 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:59.198129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:00.789 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 92 2023-10-30T13:58:00.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6cc0664a0 msgr2=0x7fd6cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6cc0664a0 0x7fd6cc068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4000f70 tx=0x7fd6e4007df0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e80cb7d0 msgr2=0x7fd6e81ef1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e80cb7d0 0x7fd6e81ef1b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800c550 tx=0x7fd6d800ca10 comp rx=0 tx=0).stop 2023-10-30T13:58:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 shutdown_connections 2023-10-30T13:58:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd6cc0664a0 0x7fd6cc068960 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd6e8150830 0x7fd6e806bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e80cb7d0 0x7fd6e81ef1b0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 --2- 172.21.15.31:0/3730234760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e8070520 0x7fd6e8153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 >> 172.21.15.31:0/3730234760 conn(0x7fd6e80cac70 msgr2=0x7fd6e80c7680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 shutdown_connections 2023-10-30T13:58:00.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:00.789+0000 7fd6efcc9640 1 -- 172.21.15.31:0/3730234760 wait complete. 2023-10-30T13:58:00.810 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:02.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f0892c09640 1 Processor -- start 2023-10-30T13:58:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f0892c09640 1 -- start start 2023-10-30T13:58:02.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f0892c09640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f088c153c80 0x7f088c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f0892c09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c0c4a10 0x7f088c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f0892c09640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088c159f20 con 0x7f088c15a3b0 2023-10-30T13:58:02.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f0892c09640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c15dec0 con 0x7f088c153c80 2023-10-30T13:58:02.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f0892c09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c1599a0 con 0x7f088c0c4a10 2023-10-30T13:58:02.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f08909a5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f088c153c80 0x7f088c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f08909a5640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f088c153c80 0x7f088c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43998/0 (socket says 172.21.15.31:43998) 2023-10-30T13:58:02.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f08909a5640 1 -- 172.21.15.31:0/1844042014 learned_addr learned my addr 172.21.15.31:0/1844042014 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:02.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f088bfff640 1 --2- 172.21.15.31:0/1844042014 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c0c4a10 0x7f088c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f08909a5640 1 -- 172.21.15.31:0/1844042014 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c0c4a10 msgr2=0x7f088c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:02.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f08909a5640 1 --2- 172.21.15.31:0/1844042014 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c0c4a10 0x7f088c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f08909a5640 1 -- 172.21.15.31:0/1844042014 >> v1:172.21.15.33:6789/0 conn(0x7f088c15a3b0 legacy=0x7f088c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.965+0000 7f08909a5640 1 -- 172.21.15.31:0/1844042014 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c0c47e0 con 0x7f088c153c80 2023-10-30T13:58:02.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f08909a5640 1 --2- 172.21.15.31:0/1844042014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c153c80 0x7f088c15cf90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0880009870 tx=0x7f088001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868caa775fe680f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f088b7fe640 1 -- 172.21.15.31:0/1844042014 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800a8a30 con 0x7f088c153c80 2023-10-30T13:58:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f088b7fe640 1 -- 172.21.15.31:0/1844042014 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08800a8b90 con 0x7f088c153c80 2023-10-30T13:58:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f088b7fe640 1 -- 172.21.15.31:0/1844042014 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800b0440 con 0x7f088c153c80 2023-10-30T13:58:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 -- 172.21.15.31:0/1844042014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c153c80 msgr2=0x7f088c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 --2- 172.21.15.31:0/1844042014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c153c80 0x7f088c15cf90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0880009870 tx=0x7f088001ed30 comp rx=0 tx=0).stop 2023-10-30T13:58:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 -- 172.21.15.31:0/1844042014 shutdown_connections 2023-10-30T13:58:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 --2- 172.21.15.31:0/1844042014 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c0c4a10 0x7f088c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 --2- 172.21.15.31:0/1844042014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c153c80 0x7f088c15cf90 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 -- 172.21.15.31:0/1844042014 >> 172.21.15.31:0/1844042014 conn(0x7f088c0715c0 msgr2=0x7f088c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:02.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 -- 172.21.15.31:0/1844042014 shutdown_connections 2023-10-30T13:58:02.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.969+0000 7f0892c09640 1 -- 172.21.15.31:0/1844042014 wait complete. 2023-10-30T13:58:02.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 Processor -- start 2023-10-30T13:58:02.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 -- start start 2023-10-30T13:58:02.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f088c0c4a10 0x7f088c149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c153c80 0x7f088c0c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c15a3b0 0x7f088c0cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c0b2870 con 0x7f088c15a3b0 2023-10-30T13:58:02.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c0b20b0 con 0x7f088c0c4a10 2023-10-30T13:58:02.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c1527a0 con 0x7f088c153c80 2023-10-30T13:58:02.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c15a3b0 0x7f088c0cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c15a3b0 0x7f088c0cf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44010/0 (socket says 172.21.15.31:44010) 2023-10-30T13:58:02.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 -- 172.21.15.31:0/1231785350 learned_addr learned my addr 172.21.15.31:0/1231785350 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:02.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f088bfff640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c153c80 0x7f088c0c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08909a5640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f088c0c4a10 0x7f088c149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 -- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c153c80 msgr2=0x7f088c0c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c153c80 0x7f088c0c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 -- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f088c0c4a10 msgr2=0x7f088c149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f088c0c4a10 0x7f088c149fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 -- 172.21.15.31:0/1231785350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800094d0 con 0x7f088c15a3b0 2023-10-30T13:58:02.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08911a6640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c15a3b0 0x7f088c0cf270 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f088400c730 tx=0x7f088400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:02.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08897fa640 1 -- 172.21.15.31:0/1231785350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884003b80 con 0x7f088c15a3b0 2023-10-30T13:58:02.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08897fa640 1 -- 172.21.15.31:0/1231785350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0884003ce0 con 0x7f088c15a3b0 2023-10-30T13:58:02.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f08897fa640 1 -- 172.21.15.31:0/1231785350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088400aa30 con 0x7f088c15a3b0 2023-10-30T13:58:02.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c151b90 con 0x7f088c15a3b0 2023-10-30T13:58:02.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.973+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c152150 con 0x7f088c15a3b0 2023-10-30T13:58:02.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.977+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c04e170 con 0x7f088c15a3b0 2023-10-30T13:58:02.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.977+0000 7f08897fa640 1 -- 172.21.15.31:0/1231785350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f088400ab90 con 0x7f088c15a3b0 2023-10-30T13:58:02.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.977+0000 7f08897fa640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08700664a0 0x7f0870068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.977+0000 7f08897fa640 1 -- 172.21.15.31:0/1231785350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f0884083f30 con 0x7f088c15a3b0 2023-10-30T13:58:02.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.977+0000 7f08909a5640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08700664a0 0x7f0870068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.977+0000 7f08909a5640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08700664a0 0x7f0870068960 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f08800b2000 tx=0x7f0880009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:02.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:02.981+0000 7f08897fa640 1 -- 172.21.15.31:0/1231785350 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0884051020 con 0x7f088c15a3b0 2023-10-30T13:58:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f088c1509b0 con 0x7f088c15a3b0 2023-10-30T13:58:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f08897fa640 1 -- 172.21.15.31:0/1231785350 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f0884054640 con 0x7f088c15a3b0 2023-10-30T13:58:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:03.174 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:57:59.198129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 92 2023-10-30T13:58:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08700664a0 msgr2=0x7f0870068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08700664a0 0x7f0870068960 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f08800b2000 tx=0x7f0880009040 comp rx=0 tx=0).stop 2023-10-30T13:58:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c15a3b0 msgr2=0x7f088c0cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c15a3b0 0x7f088c0cf270 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f088400c730 tx=0x7f088400cbf0 comp rx=0 tx=0).stop 2023-10-30T13:58:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 shutdown_connections 2023-10-30T13:58:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f08700664a0 0x7f0870068960 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c15a3b0 0x7f088c0cf270 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f088c153c80 0x7f088c0c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 --2- 172.21.15.31:0/1231785350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f088c0c4a10 0x7f088c149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 >> 172.21.15.31:0/1231785350 conn(0x7f088c0715c0 msgr2=0x7f088c0cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 shutdown_connections 2023-10-30T13:58:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:03.173+0000 7f0892c09640 1 -- 172.21.15.31:0/1231785350 wait complete. 2023-10-30T13:58:03.195 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:05.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.349+0000 7fc861e17640 1 Processor -- start 2023-10-30T13:58:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.349+0000 7fc861e17640 1 -- start start 2023-10-30T13:58:05.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc861e17640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc85c06c420 0x7fc85c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc861e17640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c0c5f80 con 0x7fc85c0c7170 2023-10-30T13:58:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc861e17640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c0c5690 con 0x7fc85c153bb0 2023-10-30T13:58:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc861e17640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85c0c7580 con 0x7fc85c06c420 2023-10-30T13:58:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85affd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc85c06c420 0x7fc85c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc85c0c7170 0x7fc85c159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:38524/0 (socket says 172.21.15.31:38524) 2023-10-30T13:58:05.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85bfff640 1 -- 172.21.15.31:0/439947562 learned_addr learned my addr 172.21.15.31:0/439947562 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:58:05.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85affd640 1 --2- 172.21.15.31:0/439947562 >> v2:172.21.15.31:3300/0 conn(0x7fc85c06c420 0x7fc85c0c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56878/0 (socket says 172.21.15.31:56878) 2023-10-30T13:58:05.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85affd640 1 -- 172.21.15.31:0/439947562 >> v1:172.21.15.33:6789/0 conn(0x7fc85c153bb0 legacy=0x7fc85c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85affd640 1 -- 172.21.15.31:0/439947562 >> v1:172.21.15.31:6789/0 conn(0x7fc85c0c7170 legacy=0x7fc85c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85affd640 1 -- 172.21.15.31:0/439947562 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c15a580 con 0x7fc85c06c420 2023-10-30T13:58:05.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85a7fc640 1 -- 172.21.15.31:0/439947562 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485359552 0 0) 0x7fc85c0c5f80 con 0x7fc85c0c7170 2023-10-30T13:58:05.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.353+0000 7fc85affd640 1 --2- 172.21.15.31:0/439947562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c06c420 0x7fc85c0c6c30 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc8500097c0 tx=0x7fc85001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad6bebdc5afd11c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:05.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc85a7fc640 1 -- 172.21.15.31:0/439947562 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500a8970 con 0x7fc85c06c420 2023-10-30T13:58:05.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc85a7fc640 1 -- 172.21.15.31:0/439947562 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8500a8ad0 con 0x7fc85c06c420 2023-10-30T13:58:05.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc85a7fc640 1 -- 172.21.15.31:0/439947562 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500a8e60 con 0x7fc85c06c420 2023-10-30T13:58:05.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- 172.21.15.31:0/439947562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c06c420 msgr2=0x7fc85c0c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 --2- 172.21.15.31:0/439947562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c06c420 0x7fc85c0c6c30 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc8500097c0 tx=0x7fc85001ec70 comp rx=0 tx=0).stop 2023-10-30T13:58:05.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- 172.21.15.31:0/439947562 shutdown_connections 2023-10-30T13:58:05.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 --2- 172.21.15.31:0/439947562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c06c420 0x7fc85c0c6c30 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- 172.21.15.31:0/439947562 >> 172.21.15.31:0/439947562 conn(0x7fc85c0cac70 msgr2=0x7fc85c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:05.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- 172.21.15.31:0/439947562 shutdown_connections 2023-10-30T13:58:05.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- 172.21.15.31:0/439947562 wait complete. 2023-10-30T13:58:05.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 Processor -- start 2023-10-30T13:58:05.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- start start 2023-10-30T13:58:05.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc85c06c420 0x7fc85c0bf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c0c7170 0x7fc85c0bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc85c153bb0 0x7fc85c14f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0c04a0 con 0x7fc85c0c7170 2023-10-30T13:58:05.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0c0780 con 0x7fc85c153bb0 2023-10-30T13:58:05.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc861e17640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0c0a60 con 0x7fc85c06c420 2023-10-30T13:58:05.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc85affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c0c7170 0x7fc85c0bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc85affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c0c7170 0x7fc85c0bfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56882/0 (socket says 172.21.15.31:56882) 2023-10-30T13:58:05.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.357+0000 7fc85affd640 1 -- 172.21.15.31:0/144124918 learned_addr learned my addr 172.21.15.31:0/144124918 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:05.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85bfff640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc85c153bb0 0x7fc85c14f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85b7fe640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc85c06c420 0x7fc85c0bf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85affd640 1 -- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc85c06c420 msgr2=0x7fc85c0bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85affd640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc85c06c420 0x7fc85c0bf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85affd640 1 -- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc85c153bb0 msgr2=0x7fc85c14f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85affd640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc85c153bb0 0x7fc85c14f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85affd640 1 -- 172.21.15.31:0/144124918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc850009420 con 0x7fc85c0c7170 2023-10-30T13:58:05.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc85affd640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c0c7170 0x7fc85c0bfba0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc85001b880 tx=0x7fc85001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:05.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc843fff640 1 -- 172.21.15.31:0/144124918 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500b0ea0 con 0x7fc85c0c7170 2023-10-30T13:58:05.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc843fff640 1 -- 172.21.15.31:0/144124918 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8500b9410 con 0x7fc85c0c7170 2023-10-30T13:58:05.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc843fff640 1 -- 172.21.15.31:0/144124918 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500b97a0 con 0x7fc85c0c7170 2023-10-30T13:58:05.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c0c0d40 con 0x7fc85c0c7170 2023-10-30T13:58:05.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c0c1300 con 0x7fc85c0c7170 2023-10-30T13:58:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85c154b50 con 0x7fc85c0c7170 2023-10-30T13:58:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.361+0000 7fc843fff640 1 -- 172.21.15.31:0/144124918 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8500b7040 con 0x7fc85c0c7170 2023-10-30T13:58:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.365+0000 7fc843fff640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8480664d0 0x7fc848068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.365+0000 7fc843fff640 1 -- 172.21.15.31:0/144124918 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fc850121fc0 con 0x7fc85c0c7170 2023-10-30T13:58:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.365+0000 7fc85b7fe640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8480664d0 0x7fc848068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.365+0000 7fc85b7fe640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8480664d0 0x7fc848068990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc84c000f70 tx=0x7fc84c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:05.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.369+0000 7fc843fff640 1 -- 172.21.15.31:0/144124918 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc85c154b50 con 0x7fc85c0c7170 2023-10-30T13:58:05.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc85c154b50 con 0x7fc85c0c7170 2023-10-30T13:58:05.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc843fff640 1 -- 172.21.15.31:0/144124918 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5708 (secure 0 0 0) 0x7fc85c154b50 con 0x7fc85c0c7170 2023-10-30T13:58:05.563 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:05.563 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:04.382419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:05.563 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 93 2023-10-30T13:58:05.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8480664d0 msgr2=0x7fc848068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8480664d0 0x7fc848068990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc84c000f70 tx=0x7fc84c009040 comp rx=0 tx=0).stop 2023-10-30T13:58:05.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c0c7170 msgr2=0x7fc85c0bfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c0c7170 0x7fc85c0bfba0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc85001b880 tx=0x7fc85001ba10 comp rx=0 tx=0).stop 2023-10-30T13:58:05.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 shutdown_connections 2023-10-30T13:58:05.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc8480664d0 0x7fc848068990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc85c153bb0 0x7fc85c14f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc85c0c7170 0x7fc85c0bfba0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 --2- 172.21.15.31:0/144124918 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc85c06c420 0x7fc85c0bf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 >> 172.21.15.31:0/144124918 conn(0x7fc85c0cac70 msgr2=0x7fc85c0c8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:05.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.561+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 shutdown_connections 2023-10-30T13:58:05.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:05.565+0000 7fc861e17640 1 -- 172.21.15.31:0/144124918 wait complete. 2023-10-30T13:58:05.584 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:07.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:07.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.736+0000 7f2e60e6b640 1 Processor -- start 2023-10-30T13:58:07.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.736+0000 7f2e60e6b640 1 -- start start 2023-10-30T13:58:07.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e60e6b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2e5c151c20 0x7f2e5c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e60e6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c0cb8b0 0x7f2e5c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e60e6b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c0c5030 con 0x7f2e5c151840 2023-10-30T13:58:07.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e60e6b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e5c157830 con 0x7f2e5c151c20 2023-10-30T13:58:07.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e60e6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c157b10 con 0x7f2e5c0cb8b0 2023-10-30T13:58:07.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e53fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2e5c151c20 0x7f2e5c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e53fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2e5c151c20 0x7f2e5c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56884/0 (socket says 172.21.15.31:56884) 2023-10-30T13:58:07.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e53fff640 1 -- 172.21.15.31:0/236691500 learned_addr learned my addr 172.21.15.31:0/236691500 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:07.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e5ad9d640 1 --2- 172.21.15.31:0/236691500 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c0cb8b0 0x7f2e5c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e53fff640 1 -- 172.21.15.31:0/236691500 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c0cb8b0 msgr2=0x7f2e5c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:07.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e53fff640 1 --2- 172.21.15.31:0/236691500 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c0cb8b0 0x7f2e5c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e53fff640 1 -- 172.21.15.31:0/236691500 >> v1:172.21.15.33:6789/0 conn(0x7f2e5c151840 legacy=0x7f2e5c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e53fff640 1 -- 172.21.15.31:0/236691500 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c157df0 con 0x7f2e5c151c20 2023-10-30T13:58:07.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.740+0000 7f2e5ad9d640 1 --2- 172.21.15.31:0/236691500 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c0cb8b0 0x7f2e5c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:07.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e53fff640 1 --2- 172.21.15.31:0/236691500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c0c49a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009870 tx=0x7f2e4401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbacdec235f8b96c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:07.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e59d9b640 1 -- 172.21.15.31:0/236691500 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e440a8a30 con 0x7f2e5c151c20 2023-10-30T13:58:07.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e59d9b640 1 -- 172.21.15.31:0/236691500 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e440a8b90 con 0x7f2e5c151c20 2023-10-30T13:58:07.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e59d9b640 1 -- 172.21.15.31:0/236691500 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e440b0440 con 0x7f2e5c151c20 2023-10-30T13:58:07.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 -- 172.21.15.31:0/236691500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 msgr2=0x7f2e5c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/236691500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c0c49a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009870 tx=0x7f2e4401ed30 comp rx=0 tx=0).stop 2023-10-30T13:58:07.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 -- 172.21.15.31:0/236691500 shutdown_connections 2023-10-30T13:58:07.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/236691500 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c0cb8b0 0x7f2e5c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/236691500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c0c49a0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 -- 172.21.15.31:0/236691500 >> 172.21.15.31:0/236691500 conn(0x7f2e5c0cac70 msgr2=0x7f2e5c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:07.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 -- 172.21.15.31:0/236691500 shutdown_connections 2023-10-30T13:58:07.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 -- 172.21.15.31:0/236691500 wait complete. 2023-10-30T13:58:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 Processor -- start 2023-10-30T13:58:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 -- start start 2023-10-30T13:58:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e5c0cb8b0 0x7f2e5c1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.744+0000 7f2e60e6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c151840 0x7f2e5c1f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e60e6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e60e6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c1483a0 con 0x7f2e5c151c20 2023-10-30T13:58:07.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e60e6b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c1485d0 con 0x7f2e5c0cb8b0 2023-10-30T13:58:07.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e60e6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c1f2de0 con 0x7f2e5c151840 2023-10-30T13:58:07.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56900/0 (socket says 172.21.15.31:56900) 2023-10-30T13:58:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 -- 172.21.15.31:0/634488055 learned_addr learned my addr 172.21.15.31:0/634488055 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5a59c640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e5c0cb8b0 0x7f2e5c1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e53fff640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c151840 0x7f2e5c1f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 -- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c151840 msgr2=0x7f2e5c1f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c151840 0x7f2e5c1f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 -- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e5c0cb8b0 msgr2=0x7f2e5c1494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e5c0cb8b0 0x7f2e5c1494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 -- 172.21.15.31:0/634488055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e440094d0 con 0x7f2e5c151c20 2023-10-30T13:58:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e5ad9d640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c1535a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2e54007ef0 tx=0x7f2e5400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e52ffd640 1 -- 172.21.15.31:0/634488055 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54003ba0 con 0x7f2e5c151c20 2023-10-30T13:58:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e52ffd640 1 -- 172.21.15.31:0/634488055 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5400f040 con 0x7f2e5c151c20 2023-10-30T13:58:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e52ffd640 1 -- 172.21.15.31:0/634488055 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5400de80 con 0x7f2e5c151c20 2023-10-30T13:58:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c071050 con 0x7f2e5c151c20 2023-10-30T13:58:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c071610 con 0x7f2e5c151c20 2023-10-30T13:58:07.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c04e170 con 0x7f2e5c151c20 2023-10-30T13:58:07.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.748+0000 7f2e52ffd640 1 -- 172.21.15.31:0/634488055 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e54004030 con 0x7f2e5c151c20 2023-10-30T13:58:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.752+0000 7f2e52ffd640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e3c0664a0 0x7f2e3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.752+0000 7f2e52ffd640 1 -- 172.21.15.31:0/634488055 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f2e54084520 con 0x7f2e5c151c20 2023-10-30T13:58:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.752+0000 7f2e5a59c640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e3c0664a0 0x7f2e3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.752+0000 7f2e5a59c640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e3c0664a0 0x7f2e3c068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c007fd0 tx=0x7f2e4c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:07.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.756+0000 7f2e52ffd640 1 -- 172.21.15.31:0/634488055 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e54051620 con 0x7f2e5c151c20 2023-10-30T13:58:07.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.944+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e5c1580a0 con 0x7f2e5c151c20 2023-10-30T13:58:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.944+0000 7f2e52ffd640 1 -- 172.21.15.31:0/634488055 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5708 (secure 0 0 0) 0x7f2e54054c40 con 0x7f2e5c151c20 2023-10-30T13:58:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:07.948 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:04.382419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:07.948 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 93 2023-10-30T13:58:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e3c0664a0 msgr2=0x7f2e3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e3c0664a0 0x7f2e3c068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c007fd0 tx=0x7f2e4c009040 comp rx=0 tx=0).stop 2023-10-30T13:58:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 msgr2=0x7f2e5c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c1535a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2e54007ef0 tx=0x7f2e5400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 shutdown_connections 2023-10-30T13:58:07.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2e3c0664a0 0x7f2e3c068960 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c151c20 0x7f2e5c1535a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2e5c151840 0x7f2e5c1f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 --2- 172.21.15.31:0/634488055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e5c0cb8b0 0x7f2e5c1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 >> 172.21.15.31:0/634488055 conn(0x7f2e5c0cac70 msgr2=0x7f2e5c06b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:07.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 shutdown_connections 2023-10-30T13:58:07.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:07.948+0000 7f2e60e6b640 1 -- 172.21.15.31:0/634488055 wait complete. 2023-10-30T13:58:07.969 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:35:07.654733+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6549": { "addr": "172.21.15.31:6837/1195492284", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6836", "nonce": 1195492284, "type": "v2" }, { "addr": "172.21.15.31:6837", "nonce": 1195492284, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6549, "incarnation": 90, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:58:04.382419+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6549 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/818491485", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 818491485, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 818491485, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.31:6835/3532148251", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3532148251, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3532148251, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6839/331154977", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6838", "nonce": 331154977, "type": "v2" }, { "addr": "172.21.15.33:6839", "nonce": 331154977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 53, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5240, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.33:6837/2118974990", "addrs": { "addrvec": [ { "addr": "172.21.15.33:6835", "nonce": 2118974990, "type": "v2" }, { "addr": "172.21.15.33:6837", "nonce": 2118974990, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5308, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:58:09.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:10.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.124+0000 7fe12aac9640 1 Processor -- start 2023-10-30T13:58:10.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.124+0000 7fe12aac9640 1 -- start start 2023-10-30T13:58:10.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.124+0000 7fe12aac9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe124150830 0x7fe12406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:10.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.124+0000 7fe12aac9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe124070520 0x7fe1240709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:10.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.124+0000 7fe12aac9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1240c2310 con 0x7fe1240cb7d0 2023-10-30T13:58:10.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.124+0000 7fe12aac9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe124156710 con 0x7fe124070520 2023-10-30T13:58:10.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.124+0000 7fe12aac9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1240c4360 con 0x7fe124150830 2023-10-30T13:58:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe128865640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe124150830 0x7fe12406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe124070520 0x7fe1240709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe124070520 0x7fe1240709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:56414/0 (socket says 172.21.15.31:56414) 2023-10-30T13:58:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 -- 172.21.15.31:0/691678576 learned_addr learned my addr 172.21.15.31:0/691678576 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe128865640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe124150830 0x7fe12406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3301/0 says I am v2:172.21.15.31:55320/0 (socket says 172.21.15.31:55320) 2023-10-30T13:58:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe123fff640 1 --1- 172.21.15.31:0/691678576 >> v1:172.21.15.33:6789/0 conn(0x7fe1240cb7d0 0x7fe1240c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:49236/0 (socket says 172.21.15.31:49236) 2023-10-30T13:58:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 -- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe124150830 msgr2=0x7fe12406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 --2- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe124150830 0x7fe12406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 -- 172.21.15.31:0/691678576 >> v1:172.21.15.33:6789/0 conn(0x7fe1240cb7d0 legacy=0x7fe1240c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 -- 172.21.15.31:0/691678576 --> v2:172.21.15.33:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1241569b0 con 0x7fe124070520 2023-10-30T13:58:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe128865640 1 --2- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe124150830 0x7fe12406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:58:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.128+0000 7fe129066640 1 --2- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 0x7fe1240709b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009e50 tx=0x7fe11c021400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39572846acceb7a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe1237fe640 1 -- 172.21.15.31:0/691678576 <== mon.1 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0b1400 con 0x7fe124070520 2023-10-30T13:58:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe1237fe640 1 -- 172.21.15.31:0/691678576 <== mon.1 v2:172.21.15.33:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c0ad050 con 0x7fe124070520 2023-10-30T13:58:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe1237fe640 1 -- 172.21.15.31:0/691678576 <== mon.1 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0b1780 con 0x7fe124070520 2023-10-30T13:58:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 msgr2=0x7fe1240709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 --2- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 0x7fe1240709b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009e50 tx=0x7fe11c021400 comp rx=0 tx=0).stop 2023-10-30T13:58:10.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- 172.21.15.31:0/691678576 shutdown_connections 2023-10-30T13:58:10.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 --2- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 0x7fe1240709b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 --2- 172.21.15.31:0/691678576 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe124150830 0x7fe12406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- 172.21.15.31:0/691678576 >> 172.21.15.31:0/691678576 conn(0x7fe1240cac70 msgr2=0x7fe1240cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:10.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- 172.21.15.31:0/691678576 shutdown_connections 2023-10-30T13:58:10.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- 172.21.15.31:0/691678576 wait complete. 2023-10-30T13:58:10.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 Processor -- start 2023-10-30T13:58:10.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- start start 2023-10-30T13:58:10.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 0x7fe1240c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe1240cb7d0 0x7fe1240c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124150830 0x7fe1240c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1240c9800 con 0x7fe124150830 2023-10-30T13:58:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1240c8bf0 con 0x7fe124070520 2023-10-30T13:58:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.132+0000 7fe12aac9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1240c8ed0 con 0x7fe1240cb7d0 2023-10-30T13:58:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124150830 0x7fe1240c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124150830 0x7fe1240c5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56908/0 (socket says 172.21.15.31:56908) 2023-10-30T13:58:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 -- 172.21.15.31:0/3260473865 learned_addr learned my addr 172.21.15.31:0/3260473865 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe123fff640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe1240cb7d0 0x7fe1240c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe128865640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 0x7fe1240c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 -- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe1240cb7d0 msgr2=0x7fe1240c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe1240cb7d0 0x7fe1240c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 -- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 msgr2=0x7fe1240c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 0x7fe1240c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 -- 172.21.15.31:0/3260473865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118009420 con 0x7fe124150830 2023-10-30T13:58:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe129066640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124150830 0x7fe1240c5a30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe11c01f030 tx=0x7fe11c01e1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe1217fa640 1 -- 172.21.15.31:0/3260473865 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0b1d30 con 0x7fe124150830 2023-10-30T13:58:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe1217fa640 1 -- 172.21.15.31:0/3260473865 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c0b1e90 con 0x7fe124150830 2023-10-30T13:58:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c009b30 con 0x7fe124150830 2023-10-30T13:58:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe1217fa640 1 -- 172.21.15.31:0/3260473865 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0bc660 con 0x7fe124150830 2023-10-30T13:58:10.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1240c9490 con 0x7fe124150830 2023-10-30T13:58:10.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1240ca1d0 con 0x7fe124150830 2023-10-30T13:58:10.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.136+0000 7fe1217fa640 1 -- 172.21.15.31:0/3260473865 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe11c0b8040 con 0x7fe124150830 2023-10-30T13:58:10.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.140+0000 7fe1217fa640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe1080664d0 0x7fe108068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:10.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.140+0000 7fe1217fa640 1 -- 172.21.15.31:0/3260473865 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fe11c124330 con 0x7fe124150830 2023-10-30T13:58:10.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.140+0000 7fe128865640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe1080664d0 0x7fe108068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:10.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.140+0000 7fe128865640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe1080664d0 0x7fe108068990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe118009c50 tx=0x7fe118009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:10.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.144+0000 7fe1217fa640 1 -- 172.21.15.31:0/3260473865 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe11c0f1580 con 0x7fe124150830 2023-10-30T13:58:10.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.360+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe124066510 con 0x7fe124150830 2023-10-30T13:58:10.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.360+0000 7fe1217fa640 1 -- 172.21.15.31:0/3260473865 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5708 (secure 0 0 0) 0x7fe11c009510 con 0x7fe124150830 2023-10-30T13:58:10.363 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:10.363 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:04.382419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:10.363 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 93 2023-10-30T13:58:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe1080664d0 msgr2=0x7fe108068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe1080664d0 0x7fe108068990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe118009c50 tx=0x7fe118009190 comp rx=0 tx=0).stop 2023-10-30T13:58:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124150830 msgr2=0x7fe1240c5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124150830 0x7fe1240c5a30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe11c01f030 tx=0x7fe11c01e1a0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 shutdown_connections 2023-10-30T13:58:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe1080664d0 0x7fe108068990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124150830 0x7fe1240c5a30 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe1240cb7d0 0x7fe1240c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 --2- 172.21.15.31:0/3260473865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe124070520 0x7fe1240c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 >> 172.21.15.31:0/3260473865 conn(0x7fe1240cac70 msgr2=0x7fe1240c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 shutdown_connections 2023-10-30T13:58:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:10.364+0000 7fe12aac9640 1 -- 172.21.15.31:0/3260473865 wait complete. 2023-10-30T13:58:10.391 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:12.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.544+0000 7fa33e817640 1 Processor -- start 2023-10-30T13:58:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.544+0000 7fa33e817640 1 -- start start 2023-10-30T13:58:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa33e817640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa33814fc50 0x7fa33806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa33e817640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3380c6320 0x7fa338070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa33e817640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380c55f0 con 0x7fa33806be90 2023-10-30T13:58:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa33e817640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3380c5f00 con 0x7fa33814fc50 2023-10-30T13:58:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa33e817640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa338071350 con 0x7fa3380c6320 2023-10-30T13:58:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa337fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa33814fc50 0x7fa33806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa33cdb4640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fa33806be90 0x7fa338155c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:38536/0 (socket says 172.21.15.31:38536) 2023-10-30T13:58:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa33cdb4640 1 -- 172.21.15.31:0/80108268 learned_addr learned my addr 172.21.15.31:0/80108268 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:58:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa3377fe640 1 --2- 172.21.15.31:0/80108268 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3380c6320 0x7fa338070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa337fff640 1 -- 172.21.15.31:0/80108268 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3380c6320 msgr2=0x7fa338070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa337fff640 1 --2- 172.21.15.31:0/80108268 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3380c6320 0x7fa338070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa337fff640 1 -- 172.21.15.31:0/80108268 >> v1:172.21.15.31:6789/0 conn(0x7fa33806be90 legacy=0x7fa338155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa337fff640 1 -- 172.21.15.31:0/80108268 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3381567b0 con 0x7fa33814fc50 2023-10-30T13:58:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa336ffd640 1 -- 172.21.15.31:0/80108268 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147337098 0 0) 0x7fa3380c55f0 con 0x7fa33806be90 2023-10-30T13:58:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.548+0000 7fa3377fe640 1 --2- 172.21.15.31:0/80108268 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3380c6320 0x7fa338070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa337fff640 1 --2- 172.21.15.31:0/80108268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa33806fdf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fa3280097c0 tx=0x7fa32801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab628ec8f990de17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa336ffd640 1 -- 172.21.15.31:0/80108268 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280ad640 con 0x7fa33814fc50 2023-10-30T13:58:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa336ffd640 1 -- 172.21.15.31:0/80108268 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3280ad7a0 con 0x7fa33814fc50 2023-10-30T13:58:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa336ffd640 1 -- 172.21.15.31:0/80108268 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280adb30 con 0x7fa33814fc50 2023-10-30T13:58:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 -- 172.21.15.31:0/80108268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 msgr2=0x7fa33806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 --2- 172.21.15.31:0/80108268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa33806fdf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fa3280097c0 tx=0x7fa32801c3f0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 -- 172.21.15.31:0/80108268 shutdown_connections 2023-10-30T13:58:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 --2- 172.21.15.31:0/80108268 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa3380c6320 0x7fa338070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 --2- 172.21.15.31:0/80108268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa33806fdf0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 -- 172.21.15.31:0/80108268 >> 172.21.15.31:0/80108268 conn(0x7fa3380cac70 msgr2=0x7fa3380cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 -- 172.21.15.31:0/80108268 shutdown_connections 2023-10-30T13:58:12.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 -- 172.21.15.31:0/80108268 wait complete. 2023-10-30T13:58:12.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 Processor -- start 2023-10-30T13:58:12.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.552+0000 7fa33e817640 1 -- start start 2023-10-30T13:58:12.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa33806be90 0x7fa338152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3380c6320 0x7fa3380c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa3380c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa338151560 con 0x7fa33814fc50 2023-10-30T13:58:12.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa338157950 con 0x7fa3380c6320 2023-10-30T13:58:12.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3381587e0 con 0x7fa33806be90 2023-10-30T13:58:12.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa3380c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa3380c9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56920/0 (socket says 172.21.15.31:56920) 2023-10-30T13:58:12.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 -- 172.21.15.31:0/207388613 learned_addr learned my addr 172.21.15.31:0/207388613 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:12.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa337fff640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa33806be90 0x7fa338152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa3377fe640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3380c6320 0x7fa3380c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 -- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa33806be90 msgr2=0x7fa338152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa33806be90 0x7fa338152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 -- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3380c6320 msgr2=0x7fa3380c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3380c6320 0x7fa3380c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 -- 172.21.15.31:0/207388613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa328009420 con 0x7fa33814fc50 2023-10-30T13:58:12.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa3377fe640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3380c6320 0x7fa3380c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:12.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33cdb4640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa3380c9550 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa33000c7a0 tx=0x7fa33000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:12.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa334ff9640 1 -- 172.21.15.31:0/207388613 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33000a640 con 0x7fa33814fc50 2023-10-30T13:58:12.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa334ff9640 1 -- 172.21.15.31:0/207388613 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33000a7a0 con 0x7fa33814fc50 2023-10-30T13:58:12.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa338157300 con 0x7fa33814fc50 2023-10-30T13:58:12.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa334ff9640 1 -- 172.21.15.31:0/207388613 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33000abd0 con 0x7fa33814fc50 2023-10-30T13:58:12.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33814efd0 con 0x7fa33814fc50 2023-10-30T13:58:12.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.556+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa338157160 con 0x7fa33814fc50 2023-10-30T13:58:12.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.560+0000 7fa334ff9640 1 -- 172.21.15.31:0/207388613 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa33000ad30 con 0x7fa33814fc50 2023-10-30T13:58:12.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.560+0000 7fa334ff9640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3240664d0 0x7fa324068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.560+0000 7fa334ff9640 1 -- 172.21.15.31:0/207388613 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fa330083620 con 0x7fa33814fc50 2023-10-30T13:58:12.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.560+0000 7fa337fff640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3240664d0 0x7fa324068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.560+0000 7fa337fff640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3240664d0 0x7fa324068990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3280af000 tx=0x7fa3280a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:12.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.564+0000 7fa334ff9640 1 -- 172.21.15.31:0/207388613 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa338157160 con 0x7fa33814fc50 2023-10-30T13:58:12.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa338157160 con 0x7fa33814fc50 2023-10-30T13:58:12.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa334ff9640 1 -- 172.21.15.31:0/207388613 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5702 (secure 0 0 0) 0x7fa338157160 con 0x7fa33814fc50 2023-10-30T13:58:12.758 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:12.758 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:11.104156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:12.758 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 94 2023-10-30T13:58:12.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3240664d0 msgr2=0x7fa324068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3240664d0 0x7fa324068990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3280af000 tx=0x7fa3280a7070 comp rx=0 tx=0).stop 2023-10-30T13:58:12.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 msgr2=0x7fa3380c9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa3380c9550 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa33000c7a0 tx=0x7fa33000cc60 comp rx=0 tx=0).stop 2023-10-30T13:58:12.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 shutdown_connections 2023-10-30T13:58:12.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa3240664d0 0x7fa324068990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa33814fc50 0x7fa3380c9550 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3380c6320 0x7fa3380c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 --2- 172.21.15.31:0/207388613 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa33806be90 0x7fa338152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 >> 172.21.15.31:0/207388613 conn(0x7fa3380cac70 msgr2=0x7fa338160c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:12.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 shutdown_connections 2023-10-30T13:58:12.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:12.756+0000 7fa33e817640 1 -- 172.21.15.31:0/207388613 wait complete. 2023-10-30T13:58:12.779 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5813 2023-10-30T13:58:12.780 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.d 2023-10-30T13:58:13.780 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:58:13.781 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:58:13.781 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:58:13.784 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:58:13.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:13.863 INFO:tasks.ceph.mds.d.smithi033.stdout:starting mds.d at 2023-10-30T13:58:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.936+0000 7f0d72172640 1 Processor -- start 2023-10-30T13:58:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.936+0000 7f0d72172640 1 -- start start 2023-10-30T13:58:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d72172640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0d6c150830 0x7f0d6c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d72172640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d72172640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0c2310 con 0x7f0d6c0cb7d0 2023-10-30T13:58:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d72172640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c156710 con 0x7f0d6c150830 2023-10-30T13:58:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d72172640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0c4360 con 0x7f0d6c070520 2023-10-30T13:58:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0d6c150830 0x7f0d6c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6b7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f0d6c150830 0x7f0d6c06b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56922/0 (socket says 172.21.15.31:56922) 2023-10-30T13:58:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6affd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f0d6c0cb7d0 0x7f0d6c0c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:38538/0 (socket says 172.21.15.31:38538) 2023-10-30T13:58:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/2027983077 learned_addr learned my addr 172.21.15.31:0/2027983077 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6bfff640 1 --2- 172.21.15.31:0/2027983077 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/2027983077 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 msgr2=0x7f0d6c0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6b7fe640 1 --2- 172.21.15.31:0/2027983077 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c0709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/2027983077 >> v1:172.21.15.31:6789/0 conn(0x7f0d6c0cb7d0 legacy=0x7f0d6c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6b7fe640 1 -- 172.21.15.31:0/2027983077 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c1569b0 con 0x7f0d6c150830 2023-10-30T13:58:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.940+0000 7f0d6a7fc640 1 -- 172.21.15.31:0/2027983077 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052472043 0 0) 0x7f0d6c0c2310 con 0x7f0d6c0cb7d0 2023-10-30T13:58:13.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d6b7fe640 1 --2- 172.21.15.31:0/2027983077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c06b500 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009870 tx=0x7f0d5c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc9e612e318fc25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:13.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d6a7fc640 1 -- 172.21.15.31:0/2027983077 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0a8a30 con 0x7f0d6c150830 2023-10-30T13:58:13.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d6a7fc640 1 -- 172.21.15.31:0/2027983077 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c0a8b90 con 0x7f0d6c150830 2023-10-30T13:58:13.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d6a7fc640 1 -- 172.21.15.31:0/2027983077 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0b0440 con 0x7f0d6c150830 2023-10-30T13:58:13.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 -- 172.21.15.31:0/2027983077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 msgr2=0x7f0d6c06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:13.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 --2- 172.21.15.31:0/2027983077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c06b500 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009870 tx=0x7f0d5c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:58:13.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 -- 172.21.15.31:0/2027983077 shutdown_connections 2023-10-30T13:58:13.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 --2- 172.21.15.31:0/2027983077 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c0709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:13.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 --2- 172.21.15.31:0/2027983077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c06b500 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:13.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 -- 172.21.15.31:0/2027983077 >> 172.21.15.31:0/2027983077 conn(0x7f0d6c0cac70 msgr2=0x7f0d6c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:13.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 -- 172.21.15.31:0/2027983077 shutdown_connections 2023-10-30T13:58:13.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 -- 172.21.15.31:0/2027983077 wait complete. 2023-10-30T13:58:13.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 Processor -- start 2023-10-30T13:58:13.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.944+0000 7f0d72172640 1 -- start start 2023-10-30T13:58:13.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c1e1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:13.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb7d0 0x7f0d6c14cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:13.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c14b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:13.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c1f2f00 con 0x7f0d6c150830 2023-10-30T13:58:13.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c154430 con 0x7f0d6c0cb7d0 2023-10-30T13:58:13.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0c5870 con 0x7f0d6c070520 2023-10-30T13:58:13.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c14b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:13.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c14b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56932/0 (socket says 172.21.15.31:56932) 2023-10-30T13:58:13.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 -- 172.21.15.31:0/4227493466 learned_addr learned my addr 172.21.15.31:0/4227493466 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:13.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6affd640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb7d0 0x7f0d6c14cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:13.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6b7fe640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c1e1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:13.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 -- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 msgr2=0x7f0d6c1e1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:13.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c1e1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:13.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 -- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb7d0 msgr2=0x7f0d6c14cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:13.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb7d0 0x7f0d6c14cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:13.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 -- 172.21.15.31:0/4227493466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c0094d0 con 0x7f0d6c150830 2023-10-30T13:58:13.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d6bfff640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c14b6a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0d64007e40 tx=0x7f0d6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:13.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d53fff640 1 -- 172.21.15.31:0/4227493466 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64014620 con 0x7f0d6c150830 2023-10-30T13:58:13.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d53fff640 1 -- 172.21.15.31:0/4227493466 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6400ee90 con 0x7f0d6c150830 2023-10-30T13:58:13.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d53fff640 1 -- 172.21.15.31:0/4227493466 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640149e0 con 0x7f0d6c150830 2023-10-30T13:58:13.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c0c5b50 con 0x7f0d6c150830 2023-10-30T13:58:13.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c0c6110 con 0x7f0d6c150830 2023-10-30T13:58:13.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c1f3040 con 0x7f0d6c150830 2023-10-30T13:58:13.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.948+0000 7f0d53fff640 1 -- 172.21.15.31:0/4227493466 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d64014b40 con 0x7f0d6c150830 2023-10-30T13:58:13.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.952+0000 7f0d53fff640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d580664a0 0x7f0d58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:13.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.952+0000 7f0d53fff640 1 -- 172.21.15.31:0/4227493466 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f0d64084de0 con 0x7f0d6c150830 2023-10-30T13:58:13.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.952+0000 7f0d6b7fe640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d580664a0 0x7f0d58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:13.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.952+0000 7f0d6b7fe640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d580664a0 0x7f0d58068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0b2000 tx=0x7f0d5c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:13.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:13.956+0000 7f0d53fff640 1 -- 172.21.15.31:0/4227493466 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d6c1f3040 con 0x7f0d6c150830 2023-10-30T13:58:14.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.144+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d6c1f3040 con 0x7f0d6c150830 2023-10-30T13:58:14.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.144+0000 7f0d53fff640 1 -- 172.21.15.31:0/4227493466 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5702 (secure 0 0 0) 0x7f0d6c1f3040 con 0x7f0d6c150830 2023-10-30T13:58:14.148 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:14.148 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:11.104156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:14.149 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 94 2023-10-30T13:58:14.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d580664a0 msgr2=0x7f0d58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d580664a0 0x7f0d58068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0b2000 tx=0x7f0d5c008040 comp rx=0 tx=0).stop 2023-10-30T13:58:14.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 msgr2=0x7f0d6c14b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c14b6a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0d64007e40 tx=0x7f0d6400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 shutdown_connections 2023-10-30T13:58:14.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f0d580664a0 0x7f0d58068960 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d6c150830 0x7f0d6c14b6a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d6c0cb7d0 0x7f0d6c14cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 --2- 172.21.15.31:0/4227493466 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f0d6c070520 0x7f0d6c1e1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 >> 172.21.15.31:0/4227493466 conn(0x7f0d6c0cac70 msgr2=0x7f0d6c0733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:14.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 shutdown_connections 2023-10-30T13:58:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:14.148+0000 7f0d72172640 1 -- 172.21.15.31:0/4227493466 wait complete. 2023-10-30T13:58:14.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:58:16.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509e7d3640 1 Processor -- start 2023-10-30T13:58:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509e7d3640 1 -- start start 2023-10-30T13:58:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509e7d3640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f5098153c80 0x7f509815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509e7d3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f50980c4a10 0x7f50980c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509e7d3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5098159f20 con 0x7f509815a3b0 2023-10-30T13:58:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509e7d3640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509815dec0 con 0x7f50980c4a10 2023-10-30T13:58:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509e7d3640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50981599a0 con 0x7f5098153c80 2023-10-30T13:58:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f5097fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f5098153c80 0x7f509815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509cd70640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f509815a3b0 0x7f50980ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:33182/0 (socket says 172.21.15.31:33182) 2023-10-30T13:58:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f509cd70640 1 -- 172.21.15.31:0/1189570182 learned_addr learned my addr 172.21.15.31:0/1189570182 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:58:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.336+0000 7f50977fe640 1 --2- 172.21.15.31:0/1189570182 >> v2:172.21.15.31:3300/0 conn(0x7f50980c4a10 0x7f50980c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f50977fe640 1 -- 172.21.15.31:0/1189570182 >> v2:172.21.15.33:3300/0 conn(0x7f5098153c80 msgr2=0x7f509815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f50977fe640 1 --2- 172.21.15.31:0/1189570182 >> v2:172.21.15.33:3300/0 conn(0x7f5098153c80 0x7f509815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f50977fe640 1 -- 172.21.15.31:0/1189570182 >> v1:172.21.15.33:6789/0 conn(0x7f509815a3b0 legacy=0x7f50980ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f50977fe640 1 -- 172.21.15.31:0/1189570182 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50980c47e0 con 0x7f50980c4a10 2023-10-30T13:58:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f50977fe640 1 --2- 172.21.15.31:0/1189570182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50980c4a10 0x7f50980c42a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f50980d00f0 tx=0x7f508c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d681c8746b1f57c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f5096ffd640 1 -- 172.21.15.31:0/1189570182 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c0a8950 con 0x7f50980c4a10 2023-10-30T13:58:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f5096ffd640 1 -- 172.21.15.31:0/1189570182 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c0a8ab0 con 0x7f50980c4a10 2023-10-30T13:58:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.340+0000 7f5096ffd640 1 -- 172.21.15.31:0/1189570182 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c0a8e40 con 0x7f50980c4a10 2023-10-30T13:58:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 -- 172.21.15.31:0/1189570182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50980c4a10 msgr2=0x7f50980c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 --2- 172.21.15.31:0/1189570182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50980c4a10 0x7f50980c42a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f50980d00f0 tx=0x7f508c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:58:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 -- 172.21.15.31:0/1189570182 shutdown_connections 2023-10-30T13:58:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 --2- 172.21.15.31:0/1189570182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50980c4a10 0x7f50980c42a0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 --2- 172.21.15.31:0/1189570182 >> v2:172.21.15.33:3300/0 conn(0x7f5098153c80 0x7f509815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 -- 172.21.15.31:0/1189570182 >> 172.21.15.31:0/1189570182 conn(0x7f50980715c0 msgr2=0x7f50980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 -- 172.21.15.31:0/1189570182 shutdown_connections 2023-10-30T13:58:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 -- 172.21.15.31:0/1189570182 wait complete. 2023-10-30T13:58:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 Processor -- start 2023-10-30T13:58:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.344+0000 7f509e7d3640 1 -- start start 2023-10-30T13:58:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f50980c4a10 0x7f509814e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5098153c80 0x7f5098157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f509815a3b0 0x7f50981f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50981f2ee0 con 0x7f5098153c80 2023-10-30T13:58:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50981ee9e0 con 0x7f509815a3b0 2023-10-30T13:58:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50981f2920 con 0x7f50980c4a10 2023-10-30T13:58:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5098153c80 0x7f5098157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f5097fff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f50980c4a10 0x7f509814e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5098153c80 0x7f5098157fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56594/0 (socket says 172.21.15.31:56594) 2023-10-30T13:58:16.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 -- 172.21.15.31:0/140101419 learned_addr learned my addr 172.21.15.31:0/140101419 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:58:16.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509cd70640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f509815a3b0 0x7f50981f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:16.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 -- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f50980c4a10 msgr2=0x7f509814e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:16.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f50980c4a10 0x7f509814e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 -- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f509815a3b0 msgr2=0x7f50981f1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f509815a3b0 0x7f50981f1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 -- 172.21.15.31:0/140101419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c009420 con 0x7f5098153c80 2023-10-30T13:58:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f50977fe640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5098153c80 0x7f5098157fb0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f508c0b3040 tx=0x7f508c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f5094ff9640 1 -- 172.21.15.31:0/140101419 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c0b0df0 con 0x7f5098153c80 2023-10-30T13:58:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f5094ff9640 1 -- 172.21.15.31:0/140101419 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c0b9410 con 0x7f5098153c80 2023-10-30T13:58:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f5094ff9640 1 -- 172.21.15.31:0/140101419 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c0b97c0 con 0x7f5098153c80 2023-10-30T13:58:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50981f2c00 con 0x7f5098153c80 2023-10-30T13:58:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.348+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50981eeed0 con 0x7f5098153c80 2023-10-30T13:58:16.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.352+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509804e170 con 0x7f5098153c80 2023-10-30T13:58:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.352+0000 7f5094ff9640 1 -- 172.21.15.31:0/140101419 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f508c0b7040 con 0x7f5098153c80 2023-10-30T13:58:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.352+0000 7f5094ff9640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f50840664d0 0x7f5084068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.352+0000 7f5094ff9640 1 -- 172.21.15.31:0/140101419 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f508c121e20 con 0x7f5098153c80 2023-10-30T13:58:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.352+0000 7f5097fff640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f50840664d0 0x7f5084068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.356+0000 7f5097fff640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f50840664d0 0x7f5084068990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5088016f90 tx=0x7f5088018040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.360+0000 7f5094ff9640 1 -- 172.21.15.31:0/140101419 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f508c0eee00 con 0x7f5098153c80 2023-10-30T13:58:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.724+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50980c6270 con 0x7f5098153c80 2023-10-30T13:58:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.724+0000 7f5094ff9640 1 -- 172.21.15.31:0/140101419 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+6493 (secure 0 0 0) 0x7f508c0b5030 con 0x7f5098153c80 2023-10-30T13:58:16.728 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:58:16.728 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/1696211446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":1696211446},{"type":"v1","addr":"172.21.15.33:6836","nonce":1696211446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:11.104156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:16.728 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 95 2023-10-30T13:58:16.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f50840664d0 msgr2=0x7f5084068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:16.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f50840664d0 0x7f5084068990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5088016f90 tx=0x7f5088018040 comp rx=0 tx=0).stop 2023-10-30T13:58:16.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5098153c80 msgr2=0x7f5098157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:16.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5098153c80 0x7f5098157fb0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f508c0b3040 tx=0x7f508c01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 shutdown_connections 2023-10-30T13:58:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f50840664d0 0x7f5084068990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f509815a3b0 0x7f50981f1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5098153c80 0x7f5098157fb0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 --2- 172.21.15.31:0/140101419 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f50980c4a10 0x7f509814e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 >> 172.21.15.31:0/140101419 conn(0x7f50980715c0 msgr2=0x7f5098152f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 shutdown_connections 2023-10-30T13:58:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:58:16.728+0000 7f509e7d3640 1 -- 172.21.15.31:0/140101419 wait complete. 2023-10-30T13:58:16.765 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:58:16.765 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-10-30T13:58:20.233 INFO:tasks.ceph.mds.a.smithi031.stderr:2023-10-30T13:58:20.232+0000 7f90f46b7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:58:29.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:29.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:59.130 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 0% (97/12071) Updating files: 1% (121/12071) Updating files: 2% (242/12071) Updating files: 3% (363/12071) Updating files: 3% (435/12071) Updating files: 4% (483/12071) Updating files: 5% (604/12071) Updating files: 5% (686/12071) Updating files: 6% (725/12071) Updating files: 7% (845/12071) Updating files: 7% (919/12071) Updating files: 8% (966/12071) Updating files: 9% (1087/12071) Updating files: 9% (1128/12071) Updating files: 9% (1151/12071) Updating files: 10% (1208/12071) Updating files: 11% (1328/12071) Updating files: 11% (1370/12071) Updating files: 11% (1399/12071) Updating files: 12% (1449/12071) Updating files: 12% (1466/12071) Updating files: 13% (1570/12071) Updating files: 13% (1644/12071) Updating files: 13% (1655/12071) Updating files: 14% (1690/12071) Updating files: 15% (1811/12071) Updating files: 15% (1873/12071) Updating files: 16% (1932/12071) Updating files: 16% (2022/12071) Updating files: 17% (2053/12071) Updating files: 18% (2173/12071) Updating files: 19% (2294/12071) Updating files: 19% (2333/12071) Updating files: 20% (2415/12071) Updating files: 20% (2503/12071) Updating files: 20% (2528/12071) Updating files: 21% (2535/12071) Updating files: 21% (2555/12071) Updating files: 22% (2656/12071) Updating files: 22% (2714/12071) Updating files: 22% (2770/12071) Updating files: 23% (2777/12071) Updating files: 23% (2851/12071) Updating files: 24% (2898/12071) Updating files: 25% (3018/12071) Updating files: 26% (3139/12071) Updating files: 26% (3204/12071) Updating files: 27% (3260/12071) Updating files: 27% (3283/12071) Updating files: 28% (3380/12071) Updating files: 29% (3501/12071) Updating files: 29% (3562/12071) Updating files: 30% (3622/12071) Updating files: 30% (3629/12071) Updating files: 30% (3702/12071) Updating files: 31% (3743/12071) Updating files: 31% (3748/12071) Updating files: 31% (3833/12071) Updating files: 32% (3863/12071) Updating files: 32% (3900/12071) Updating files: 32% (3909/12071) Updating files: 32% (3914/12071) Updating files: 33% (3984/12071) Updating files: 33% (4048/12071) Updating files: 34% (4105/12071) Updating files: 34% (4107/12071) Updating files: 35% (4225/12071) Updating files: 35% (4274/12071) Updating files: 36% (4346/12071) Updating files: 36% (4405/12071) Updating files: 37% (4467/12071) Updating files: 37% (4534/12071) Updating files: 38% (4587/12071) Updating files: 39% (4708/12071) Updating files: 39% (4735/12071) Updating files: 40% (4829/12071) Updating files: 40% (4869/12071) Updating files: 40% (4880/12071) Updating files: 41% (4950/12071) Updating files: 41% (5014/12071) Updating files: 41% (5052/12071) Updating files: 42% (5070/12071) Updating files: 42% (5075/12071) Updating files: 42% (5117/12071) Updating files: 42% (5180/12071) Updating files: 43% (5191/12071) Updating files: 44% (5312/12071) Updating files: 44% (5319/12071) Updating files: 45% (5432/12071) Updating files: 45% (5455/12071) Updating files: 46% (5553/12071) Updating files: 47% (5674/12071) Updating files: 47% (5678/12071) Updating files: 48% (5795/12071) Updating files: 48% (5832/12071) Updating files: 49% (5915/12071) Updating files: 49% (5975/12071) Updating files: 50% (6036/12071) Updating files: 50% (6097/12071) Updating files: 51% (6157/12071) Updating files: 52% (6277/12071) Updating files: 52% (6284/12071) Updating files: 52% (6340/12071) Updating files: 52% (6383/12071) Updating files: 53% (6398/12071) Updating files: 53% (6511/12071) Updating files: 54% (6519/12071) Updating files: 54% (6601/12071) Updating files: 54% (6607/12071) Updating files: 55% (6640/12071) Updating files: 55% (6721/12071) Updating files: 56% (6760/12071) Updating files: 56% (6805/12071) Updating files: 56% (6841/12071) Updating files: 57% (6881/12071) Updating files: 58% (7002/12071) Updating files: 58% (7035/12071) Updating files: 58% (7075/12071) Updating files: 59% (7122/12071) Updating files: 60% (7243/12071) Updating files: 61% (7364/12071) Updating files: 61% (7403/12071) Updating files: 62% (7485/12071) Updating files: 62% (7553/12071) Updating files: 63% (7605/12071) Updating files: 63% (7701/12071) Updating files: 64% (7726/12071) Updating files: 65% (7847/12071) Updating files: 65% (7878/12071) Updating files: 66% (7967/12071) Updating files: 67% (8088/12071) Updating files: 67% (8147/12071) Updating files: 68% (8209/12071) Updating files: 68% (8273/12071) Updating files: 69% (8329/12071) Updating files: 69% (8345/12071) Updating files: 69% (8415/12071) Updating files: 70% (8450/12071) Updating files: 70% (8493/12071) Updating files: 71% (8571/12071) Updating files: 71% (8623/12071) Updating files: 72% (8692/12071) Updating files: 72% (8712/12071) Updating files: 72% (8800/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 75% (9054/12071) Updating files: 75% (9172/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 77% (9314/12071) Updating files: 77% (9394/12071) Updating files: 78% (9416/12071) Updating files: 78% (9447/12071) Updating files: 79% (9537/12071) Updating files: 79% (9611/12071) Updating files: 80% (9657/12071) Updating files: 80% (9689/12071) Updating files: 80% (9741/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 82% (9926/12071) Updating files: 83% (10019/12071) Updating files: 83% (10047/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 85% (10299/12071) Updating files: 86% (10382/12071) Updating files: 86% (10410/12071) Updating files: 86% (10411/12071) Updating files: 86% (10455/12071) Updating files: 87% (10502/12071) Updating files: 87% (10606/12071) Updating files: 88% (10623/12071) Updating files: 88% (10682/12071) Updating files: 89% (10744/12071) Updating files: 89% (10798/12071) Updating files: 90% (10864/12071) Updating files: 90% (10885/12071) Updating files: 91% (10985/12071) Updating files: 91% (11024/12071) Updating files: 92% (11106/12071) Updating files: 92% (11143/12071) Updating files: 92% (11224/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 94% (11370/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 96% (11603/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 98% (11832/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T13:58:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-10-30T13:58:59.211 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-30T13:58:59.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T13:58:59.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:59.419 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:00.088 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi031 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T13:59:00.154 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=671698dbf0a642c1bda1c3ffdfdd3cdc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi031 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T13:59:00.157 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-30T13:59:00.167 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-30T13:59:00.170 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 1 2023-10-30T13:59:00.170 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-30T13:59:00.170 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 11 2023-10-30T13:59:00.171 INFO:tasks.mds_thrash:done joining 2023-10-30T13:59:00.171 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-30T13:59:00.179 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-30T13:59:00.179 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:59:00.180 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:59:00.192 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-30T13:59:00.192 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:59:00.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:59:00.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 Processor -- start 2023-10-30T13:59:00.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 -- start start 2023-10-30T13:59:00.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a1c150830 0x7f7a1c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f7a1c0cb7d0 0x7f7a1c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0c3e40 0x7f7a1c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1c0c2310 con 0x7f7a1c150830 2023-10-30T13:59:00.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1c156710 con 0x7f7a1c0cb7d0 2023-10-30T13:59:00.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.424+0000 7f7a23db8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0c4360 con 0x7f7a1c0c3e40 2023-10-30T13:59:00.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a1c150830 0x7f7a1c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f7a1c150830 0x7f7a1c06b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38912/0 (socket says 172.21.15.31:38912) 2023-10-30T13:59:00.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 -- 172.21.15.31:0/2504159174 learned_addr learned my addr 172.21.15.31:0/2504159174 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:00.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21353640 1 --2- 172.21.15.31:0/2504159174 >> v2:172.21.15.33:3300/0 conn(0x7f7a1c0cb7d0 0x7f7a1c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a22355640 1 --2- 172.21.15.31:0/2504159174 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0c3e40 0x7f7a1c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 -- 172.21.15.31:0/2504159174 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0c3e40 msgr2=0x7f7a1c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:00.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 --2- 172.21.15.31:0/2504159174 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0c3e40 0x7f7a1c070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 -- 172.21.15.31:0/2504159174 >> v2:172.21.15.33:3300/0 conn(0x7f7a1c0cb7d0 msgr2=0x7f7a1c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:00.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 --2- 172.21.15.31:0/2504159174 >> v2:172.21.15.33:3300/0 conn(0x7f7a1c0cb7d0 0x7f7a1c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 -- 172.21.15.31:0/2504159174 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c1569b0 con 0x7f7a1c150830 2023-10-30T13:59:00.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a22355640 1 --2- 172.21.15.31:0/2504159174 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0c3e40 0x7f7a1c070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a21b54640 1 --2- 172.21.15.31:0/2504159174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c06b500 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c06f9f0 tx=0x7f7a1801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e494fda650e9770 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a20b52640 1 -- 172.21.15.31:0/2504159174 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180a89b0 con 0x7f7a1c150830 2023-10-30T13:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a20b52640 1 -- 172.21.15.31:0/2504159174 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a180a8b10 con 0x7f7a1c150830 2023-10-30T13:59:00.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.428+0000 7f7a20b52640 1 -- 172.21.15.31:0/2504159174 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180a8ea0 con 0x7f7a1c150830 2023-10-30T13:59:00.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- 172.21.15.31:0/2504159174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 msgr2=0x7f7a1c06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 --2- 172.21.15.31:0/2504159174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c06b500 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c06f9f0 tx=0x7f7a1801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- 172.21.15.31:0/2504159174 shutdown_connections 2023-10-30T13:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 --2- 172.21.15.31:0/2504159174 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0c3e40 0x7f7a1c070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 --2- 172.21.15.31:0/2504159174 >> v2:172.21.15.33:3300/0 conn(0x7f7a1c0cb7d0 0x7f7a1c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 --2- 172.21.15.31:0/2504159174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c06b500 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- 172.21.15.31:0/2504159174 >> 172.21.15.31:0/2504159174 conn(0x7f7a1c0cac70 msgr2=0x7f7a1c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- 172.21.15.31:0/2504159174 shutdown_connections 2023-10-30T13:59:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- 172.21.15.31:0/2504159174 wait complete. 2023-10-30T13:59:00.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 Processor -- start 2023-10-30T13:59:00.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- start start 2023-10-30T13:59:00.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a1c0c3e40 0x7f7a1c1ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0cb7d0 0x7f7a1c1f02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c1ed820 con 0x7f7a1c150830 2023-10-30T13:59:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c1f2fd0 con 0x7f7a1c0c3e40 2023-10-30T13:59:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a23db8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c1eda70 con 0x7f7a1c0cb7d0 2023-10-30T13:59:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a22355640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a22355640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c1f25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38926/0 (socket says 172.21.15.31:38926) 2023-10-30T13:59:00.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a22355640 1 -- 172.21.15.31:0/399989737 learned_addr learned my addr 172.21.15.31:0/399989737 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:00.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a21b54640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a1c0c3e40 0x7f7a1c1ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.432+0000 7f7a21353640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0cb7d0 0x7f7a1c1f02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a22355640 1 -- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0cb7d0 msgr2=0x7f7a1c1f02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a22355640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0cb7d0 0x7f7a1c1f02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a22355640 1 -- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a1c0c3e40 msgr2=0x7f7a1c1ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a22355640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a1c0c3e40 0x7f7a1c1ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a22355640 1 -- 172.21.15.31:0/399989737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a180094d0 con 0x7f7a1c150830 2023-10-30T13:59:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a21b54640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a1c0c3e40 0x7f7a1c1ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a22355640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c1f25a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7a14007d80 tx=0x7f7a1400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a127fc640 1 -- 172.21.15.31:0/399989737 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a140039a0 con 0x7f7a1c150830 2023-10-30T13:59:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a127fc640 1 -- 172.21.15.31:0/399989737 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1400f040 con 0x7f7a1c150830 2023-10-30T13:59:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a127fc640 1 -- 172.21.15.31:0/399989737 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14003d60 con 0x7f7a1c150830 2023-10-30T13:59:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1c1edd50 con 0x7f7a1c150830 2023-10-30T13:59:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c1ee310 con 0x7f7a1c150830 2023-10-30T13:59:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c156c70 con 0x7f7a1c150830 2023-10-30T13:59:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a127fc640 1 -- 172.21.15.31:0/399989737 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a14004030 con 0x7f7a1c150830 2023-10-30T13:59:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a127fc640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a000664a0 0x7f7a00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.436+0000 7f7a127fc640 1 -- 172.21.15.31:0/399989737 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f7a14084560 con 0x7f7a1c150830 2023-10-30T13:59:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.440+0000 7f7a21b54640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a000664a0 0x7f7a00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.440+0000 7f7a21b54640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a000664a0 0x7f7a00068960 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7a180b2000 tx=0x7f7a18008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:00.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.444+0000 7f7a127fc640 1 -- 172.21.15.31:0/399989737 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a1c156c70 con 0x7f7a1c150830 2023-10-30T13:59:00.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.600+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a1c156c70 con 0x7f7a1c150830 2023-10-30T13:59:00.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a127fc640 1 -- 172.21.15.31:0/399989737 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18184 (secure 0 0 0) 0x7f7a1c156c70 con 0x7f7a1c150830 2023-10-30T13:59:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:00.606 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":56,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:57:42.156993+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6836/2267972067":"2023-10-31T13:56:02.130126+0000","172.21.15.33:6836/3928996203":"2023-10-31T13:53:47.073133+0000","172.21.15.33:6834/3928996203":"2023-10-31T13:53:47.073133+0000","172.21.15.31:6837/327248983":"2023-10-31T13:51:22.020336+0000","172.21.15.31:6836/327248983":"2023-10-31T13:51:22.020336+0000","172.21.15.33:6834/2281022644":"2023-10-31T13:57:42.156901+0000","172.21.15.31:6837/2267972067":"2023-10-31T13:56:02.130126+0000","172.21.15.33:6836/2545654386":"2023-10-31T13:36:26.753730+0000","172.21.15.31:6836/1956809670":"2023-10-31T13:39:06.794112+0000","172.21.15.33:6836/15857146":"2023-10-31T13:49:36.991529+0000","172.21.15.31:6837/1956809670":"2023-10-31T13:39:06.794112+0000","172.21.15.33:6835/3402459191":"2023-10-31T13:40:56.840815+0000","172.21.15.33:6836/2281022644":"2023-10-31T13:57:42.156901+0000","172.21.15.33:6836/210349556":"2023-10-31T13:44:26.910290+0000","172.21.15.31:6836/349618961":"2023-10-31T13:47:11.952274+0000","172.21.15.33:6838/2419808617":"2023-10-31T13:43:31.895625+0000","172.21.15.33:6834/2545654386":"2023-10-31T13:36:26.753730+0000","172.21.15.33:6834/15857146":"2023-10-31T13:49:36.991529+0000","172.21.15.33:6839/2419808617":"2023-10-31T13:43:31.895625+0000","172.21.15.33:6837/3402459191":"2023-10-31T13:40:56.840815+0000","172.21.15.33:6834/210349556":"2023-10-31T13:44:26.910290+0000","172.21.15.31:6837/349618961":"2023-10-31T13:47:11.952274+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:59:00.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a000664a0 msgr2=0x7f7a00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a000664a0 0x7f7a00068960 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7a180b2000 tx=0x7f7a18008040 comp rx=0 tx=0).stop 2023-10-30T13:59:00.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 msgr2=0x7f7a1c1f25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c1f25a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7a14007d80 tx=0x7f7a1400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 shutdown_connections 2023-10-30T13:59:00.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f7a000664a0 0x7f7a00068960 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a1c150830 0x7f7a1c1f25a0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f7a1c0cb7d0 0x7f7a1c1f02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 --2- 172.21.15.31:0/399989737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a1c0c3e40 0x7f7a1c1ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 >> 172.21.15.31:0/399989737 conn(0x7f7a1c0cac70 msgr2=0x7f7a1c0733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:00.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 shutdown_connections 2023-10-30T13:59:00.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.604+0000 7f7a23db8640 1 -- 172.21.15.31:0/399989737 wait complete. 2023-10-30T13:59:00.627 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:34:53.247214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:35:05.304460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:35:06.652927+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:59:00.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:59:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.772+0000 7fa7b6cd6640 1 Processor -- start 2023-10-30T13:59:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.772+0000 7fa7b6cd6640 1 -- start start 2023-10-30T13:59:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.772+0000 7fa7b6cd6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0153c40 0x7fa7b006d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7b6cd6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa7b006dfc0 0x7fa7b0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7b6cd6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa7b0071ac0 0x7fa7b006a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7b6cd6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b006d8d0 con 0x7fa7b006dfc0 2023-10-30T13:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7b6cd6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b015cf10 con 0x7fa7b0071ac0 2023-10-30T13:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7b6cd6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00722b0 con 0x7fa7b0153c40 2023-10-30T13:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa7b006dfc0 0x7fa7b0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa7b006dfc0 0x7fa7b0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38934/0 (socket says 172.21.15.31:38934) 2023-10-30T13:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 -- 172.21.15.31:0/2814161748 learned_addr learned my addr 172.21.15.31:0/2814161748 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7b4a72640 1 --2- 172.21.15.31:0/2814161748 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0153c40 0x7fa7b006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7b5273640 1 --2- 172.21.15.31:0/2814161748 >> v2:172.21.15.33:3300/0 conn(0x7fa7b0071ac0 0x7fa7b006a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 -- 172.21.15.31:0/2814161748 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0153c40 msgr2=0x7fa7b006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 --2- 172.21.15.31:0/2814161748 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0153c40 0x7fa7b006d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 -- 172.21.15.31:0/2814161748 >> v2:172.21.15.33:3300/0 conn(0x7fa7b0071ac0 msgr2=0x7fa7b006a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 --2- 172.21.15.31:0/2814161748 >> v2:172.21.15.33:3300/0 conn(0x7fa7b0071ac0 0x7fa7b006a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 -- 172.21.15.31:0/2814161748 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0072590 con 0x7fa7b006dfc0 2023-10-30T13:59:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a7fff640 1 --2- 172.21.15.31:0/2814161748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b0069e90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa7b006c040 tx=0x7fa7a001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb7f8fc6f99efec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a77fe640 1 -- 172.21.15.31:0/2814161748 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a00a89b0 con 0x7fa7b006dfc0 2023-10-30T13:59:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a77fe640 1 -- 172.21.15.31:0/2814161748 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a00a8b10 con 0x7fa7b006dfc0 2023-10-30T13:59:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.776+0000 7fa7a77fe640 1 -- 172.21.15.31:0/2814161748 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a00a8ea0 con 0x7fa7b006dfc0 2023-10-30T13:59:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/2814161748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 msgr2=0x7fa7b0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/2814161748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b0069e90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa7b006c040 tx=0x7fa7a001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/2814161748 shutdown_connections 2023-10-30T13:59:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/2814161748 >> v2:172.21.15.33:3300/0 conn(0x7fa7b0071ac0 0x7fa7b006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/2814161748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b0069e90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/2814161748 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0153c40 0x7fa7b006d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/2814161748 >> 172.21.15.31:0/2814161748 conn(0x7fa7b006afa0 msgr2=0x7fa7b006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:00.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/2814161748 shutdown_connections 2023-10-30T13:59:00.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/2814161748 wait complete. 2023-10-30T13:59:00.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 Processor -- start 2023-10-30T13:59:00.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- start start 2023-10-30T13:59:00.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0071ac0 0x7fa7b01e2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7b0153c40 0x7fa7b0152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b013e4b0 con 0x7fa7b006dfc0 2023-10-30T13:59:00.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b01e33e0 con 0x7fa7b0153c40 2023-10-30T13:59:00.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b6cd6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b01f26a0 con 0x7fa7b0071ac0 2023-10-30T13:59:00.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b013df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38938/0 (socket says 172.21.15.31:38938) 2023-10-30T13:59:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 -- 172.21.15.31:0/37024020 learned_addr learned my addr 172.21.15.31:0/37024020 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b5273640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7b0153c40 0x7fa7b0152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7a7fff640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0071ac0 0x7fa7b01e2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 -- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0071ac0 msgr2=0x7fa7b01e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0071ac0 0x7fa7b01e2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 -- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7b0153c40 msgr2=0x7fa7b0152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7b0153c40 0x7fa7b0152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.780+0000 7fa7b4a72640 1 -- 172.21.15.31:0/37024020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a00094d0 con 0x7fa7b006dfc0 2023-10-30T13:59:00.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7b4a72640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b013df70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac007f20 tx=0x7fa7ac00e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:00.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7a57fa640 1 -- 172.21.15.31:0/37024020 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac014620 con 0x7fa7b006dfc0 2023-10-30T13:59:00.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7a57fa640 1 -- 172.21.15.31:0/37024020 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac014780 con 0x7fa7b006dfc0 2023-10-30T13:59:00.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b01e3650 con 0x7fa7b006dfc0 2023-10-30T13:59:00.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7a57fa640 1 -- 172.21.15.31:0/37024020 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac014b30 con 0x7fa7b006dfc0 2023-10-30T13:59:00.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b015c630 con 0x7fa7b006dfc0 2023-10-30T13:59:00.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b004e170 con 0x7fa7b006dfc0 2023-10-30T13:59:00.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7a57fa640 1 -- 172.21.15.31:0/37024020 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7ac014c90 con 0x7fa7b006dfc0 2023-10-30T13:59:00.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7a57fa640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa79c0664a0 0x7fa79c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:00.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.784+0000 7fa7a57fa640 1 -- 172.21.15.31:0/37024020 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fa7ac084d50 con 0x7fa7b006dfc0 2023-10-30T13:59:00.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.788+0000 7fa7a7fff640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa79c0664a0 0x7fa79c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:00.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.788+0000 7fa7a7fff640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa79c0664a0 0x7fa79c068960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00b2000 tx=0x7fa7a0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:00.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.788+0000 7fa7a57fa640 1 -- 172.21.15.31:0/37024020 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7ac051d10 con 0x7fa7b006dfc0 2023-10-30T13:59:00.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.948+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa7b006dda0 con 0x7fa7b006dfc0 2023-10-30T13:59:00.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.948+0000 7fa7a57fa640 1 -- 172.21.15.31:0/37024020 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa7ac055330 con 0x7fa7b006dfc0 2023-10-30T13:59:00.951 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-30T13:59:00.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa79c0664a0 msgr2=0x7fa79c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa79c0664a0 0x7fa79c068960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00b2000 tx=0x7fa7a0008040 comp rx=0 tx=0).stop 2023-10-30T13:59:00.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 msgr2=0x7fa7b013df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:00.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b013df70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac007f20 tx=0x7fa7ac00e570 comp rx=0 tx=0).stop 2023-10-30T13:59:00.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 shutdown_connections 2023-10-30T13:59:00.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa79c0664a0 0x7fa79c068960 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7b0153c40 0x7fa7b0152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa7b0071ac0 0x7fa7b01e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 --2- 172.21.15.31:0/37024020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b006dfc0 0x7fa7b013df70 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:00.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 >> 172.21.15.31:0/37024020 conn(0x7fa7b006afa0 msgr2=0x7fa7b014e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:00.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 shutdown_connections 2023-10-30T13:59:00.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:00.952+0000 7fa7b6cd6640 1 -- 172.21.15.31:0/37024020 wait complete. 2023-10-30T13:59:00.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:59:01.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f904d6640 1 Processor -- start 2023-10-30T13:59:01.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f904d6640 1 -- start start 2023-10-30T13:59:01.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f904d6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f3f88151a20 0x7f3f880cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f904d6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3f880c74a0 0x7f3f880c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f904d6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f881581f0 con 0x7f3f880c1020 2023-10-30T13:59:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f904d6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f881576d0 con 0x7f3f880c74a0 2023-10-30T13:59:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f904d6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f880c7e10 con 0x7f3f88151a20 2023-10-30T13:59:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f8ea73640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3f880c74a0 0x7f3f880c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f8ea73640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3f880c74a0 0x7f3f880c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38950/0 (socket says 172.21.15.31:38950) 2023-10-30T13:59:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f8ea73640 1 -- 172.21.15.31:0/3616408260 learned_addr learned my addr 172.21.15.31:0/3616408260 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:01.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f8e272640 1 --2- 172.21.15.31:0/3616408260 >> v2:172.21.15.33:3300/0 conn(0x7f3f88151a20 0x7f3f880cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f8ea73640 1 -- 172.21.15.31:0/3616408260 >> v2:172.21.15.33:3300/0 conn(0x7f3f88151a20 msgr2=0x7f3f880cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:01.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f8ea73640 1 --2- 172.21.15.31:0/3616408260 >> v2:172.21.15.33:3300/0 conn(0x7f3f88151a20 0x7f3f880cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.116+0000 7f3f8ea73640 1 -- 172.21.15.31:0/3616408260 >> v1:172.21.15.33:6789/0 conn(0x7f3f880c1020 legacy=0x7f3f88072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f8ea73640 1 -- 172.21.15.31:0/3616408260 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880cb9f0 con 0x7f3f880c74a0 2023-10-30T13:59:01.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f8ea73640 1 --2- 172.21.15.31:0/3616408260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f880c74a0 0x7f3f880c78d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3f88072bd0 tx=0x7f3f8001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4707985b6beb46 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f8d270640 1 -- 172.21.15.31:0/3616408260 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f800af400 con 0x7f3f880c74a0 2023-10-30T13:59:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f8d270640 1 -- 172.21.15.31:0/3616408260 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f800ab050 con 0x7f3f880c74a0 2023-10-30T13:59:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f8d270640 1 -- 172.21.15.31:0/3616408260 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f800af780 con 0x7f3f880c74a0 2023-10-30T13:59:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 -- 172.21.15.31:0/3616408260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f880c74a0 msgr2=0x7f3f880c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 --2- 172.21.15.31:0/3616408260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f880c74a0 0x7f3f880c78d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3f88072bd0 tx=0x7f3f8001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 -- 172.21.15.31:0/3616408260 shutdown_connections 2023-10-30T13:59:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 --2- 172.21.15.31:0/3616408260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f880c74a0 0x7f3f880c78d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 --2- 172.21.15.31:0/3616408260 >> v2:172.21.15.33:3300/0 conn(0x7f3f88151a20 0x7f3f880cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 -- 172.21.15.31:0/3616408260 >> 172.21.15.31:0/3616408260 conn(0x7f3f8806da40 msgr2=0x7f3f8806de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:01.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 -- 172.21.15.31:0/3616408260 shutdown_connections 2023-10-30T13:59:01.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.120+0000 7f3f904d6640 1 -- 172.21.15.31:0/3616408260 wait complete. 2023-10-30T13:59:01.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 Processor -- start 2023-10-30T13:59:01.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 -- start start 2023-10-30T13:59:01.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f880c1020 0x7f3f881efa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f880c74a0 0x7f3f881527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f88151a20 0x7f3f88154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8814fbf0 con 0x7f3f88151a20 2023-10-30T13:59:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f881eefe0 con 0x7f3f880c74a0 2023-10-30T13:59:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f881f2ff0 con 0x7f3f880c1020 2023-10-30T13:59:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f88151a20 0x7f3f88154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f88151a20 0x7f3f88154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38954/0 (socket says 172.21.15.31:38954) 2023-10-30T13:59:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 -- 172.21.15.31:0/2618348498 learned_addr learned my addr 172.21.15.31:0/2618348498 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8e272640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f880c1020 0x7f3f881efa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8da71640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f880c74a0 0x7f3f881527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 -- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f880c1020 msgr2=0x7f3f881efa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f880c1020 0x7f3f881efa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 -- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f880c74a0 msgr2=0x7f3f881527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f880c74a0 0x7f3f881527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 -- 172.21.15.31:0/2618348498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f80009b30 con 0x7f3f88151a20 2023-10-30T13:59:01.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f8ea73640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f88151a20 0x7f3f88154660 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f3f800b2040 tx=0x7f3f8001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f7effd640 1 -- 172.21.15.31:0/2618348498 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f800afd60 con 0x7f3f88151a20 2023-10-30T13:59:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.124+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f881ef280 con 0x7f3f88151a20 2023-10-30T13:59:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f7effd640 1 -- 172.21.15.31:0/2618348498 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f80009510 con 0x7f3f88151a20 2023-10-30T13:59:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f7effd640 1 -- 172.21.15.31:0/2618348498 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f800c6660 con 0x7f3f88151a20 2023-10-30T13:59:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f880c9250 con 0x7f3f88151a20 2023-10-30T13:59:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8804e170 con 0x7f3f88151a20 2023-10-30T13:59:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f7effd640 1 -- 172.21.15.31:0/2618348498 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f800b6040 con 0x7f3f88151a20 2023-10-30T13:59:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f7effd640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f6c0664a0 0x7f3f6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f7effd640 1 -- 172.21.15.31:0/2618348498 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f3f80122480 con 0x7f3f88151a20 2023-10-30T13:59:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.128+0000 7f3f8e272640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f6c0664a0 0x7f3f6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.132+0000 7f3f8e272640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f6c0664a0 0x7f3f6c068960 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400b9d0 tx=0x7f3f84008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.132+0000 7f3f7effd640 1 -- 172.21.15.31:0/2618348498 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f800ef4c0 con 0x7f3f88151a20 2023-10-30T13:59:01.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.288+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f881ef730 con 0x7f3f88151a20 2023-10-30T13:59:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.288+0000 7f3f7effd640 1 -- 172.21.15.31:0/2618348498 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3f800f2ae0 con 0x7f3f88151a20 2023-10-30T13:59:01.291 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-10-30T13:59:01.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f6c0664a0 msgr2=0x7f3f6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f6c0664a0 0x7f3f6c068960 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400b9d0 tx=0x7f3f84008040 comp rx=0 tx=0).stop 2023-10-30T13:59:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f88151a20 msgr2=0x7f3f88154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f88151a20 0x7f3f88154660 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f3f800b2040 tx=0x7f3f8001bf60 comp rx=0 tx=0).stop 2023-10-30T13:59:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 shutdown_connections 2023-10-30T13:59:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f3f6c0664a0 0x7f3f6c068960 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f88151a20 0x7f3f88154660 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f880c74a0 0x7f3f881527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 --2- 172.21.15.31:0/2618348498 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f3f880c1020 0x7f3f881efa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 >> 172.21.15.31:0/2618348498 conn(0x7f3f8806da40 msgr2=0x7f3f881569f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 shutdown_connections 2023-10-30T13:59:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.292+0000 7f3f904d6640 1 -- 172.21.15.31:0/2618348498 wait complete. 2023-10-30T13:59:01.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:59:01.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.460+0000 7f2dd1832640 1 Processor -- start 2023-10-30T13:59:01.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.460+0000 7f2dd1832640 1 -- start start 2023-10-30T13:59:01.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dd1832640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2dcc153ba0 0x7f2dcc0cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dd1832640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc0c4810 0x7f2dcc15a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dd1832640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc153f80 con 0x7f2dcc0c4410 2023-10-30T13:59:01.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dd1832640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dcc159df0 con 0x7f2dcc153ba0 2023-10-30T13:59:01.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dd1832640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc15a7c0 con 0x7f2dcc0c4810 2023-10-30T13:59:01.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dcaffd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f2dcc153ba0 0x7f2dcc0cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dca7fc640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f2dcc0c4410 0x7f2dcc15cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41658/0 (socket says 172.21.15.31:41658) 2023-10-30T13:59:01.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dca7fc640 1 -- 172.21.15.31:0/915070099 learned_addr learned my addr 172.21.15.31:0/915070099 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:01.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dcb7fe640 1 --2- 172.21.15.31:0/915070099 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc0c4810 0x7f2dcc15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dcaffd640 1 -- 172.21.15.31:0/915070099 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc0c4810 msgr2=0x7f2dcc15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dcaffd640 1 --2- 172.21.15.31:0/915070099 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc0c4810 0x7f2dcc15a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dcaffd640 1 -- 172.21.15.31:0/915070099 >> v1:172.21.15.31:6789/0 conn(0x7f2dcc0c4410 legacy=0x7f2dcc15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dcaffd640 1 -- 172.21.15.31:0/915070099 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc1597c0 con 0x7f2dcc153ba0 2023-10-30T13:59:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.464+0000 7f2dc9ffb640 1 -- 172.21.15.31:0/915070099 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175334935 0 0) 0x7f2dcc153f80 con 0x7f2dcc0c4410 2023-10-30T13:59:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dcaffd640 1 --2- 172.21.15.31:0/915070099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc153ba0 0x7f2dcc0cb4e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0097c0 tx=0x7f2dbc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7840b26fe1de27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dc9ffb640 1 -- 172.21.15.31:0/915070099 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0ac620 con 0x7f2dcc153ba0 2023-10-30T13:59:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dc9ffb640 1 -- 172.21.15.31:0/915070099 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc0ac780 con 0x7f2dcc153ba0 2023-10-30T13:59:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dc9ffb640 1 -- 172.21.15.31:0/915070099 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0acb10 con 0x7f2dcc153ba0 2023-10-30T13:59:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 -- 172.21.15.31:0/915070099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc153ba0 msgr2=0x7f2dcc0cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 --2- 172.21.15.31:0/915070099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc153ba0 0x7f2dcc0cb4e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0097c0 tx=0x7f2dbc01c3c0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 -- 172.21.15.31:0/915070099 shutdown_connections 2023-10-30T13:59:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 --2- 172.21.15.31:0/915070099 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc0c4810 0x7f2dcc15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 --2- 172.21.15.31:0/915070099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc153ba0 0x7f2dcc0cb4e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 -- 172.21.15.31:0/915070099 >> 172.21.15.31:0/915070099 conn(0x7f2dcc071790 msgr2=0x7f2dcc071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:01.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 -- 172.21.15.31:0/915070099 shutdown_connections 2023-10-30T13:59:01.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 -- 172.21.15.31:0/915070099 wait complete. 2023-10-30T13:59:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 Processor -- start 2023-10-30T13:59:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.468+0000 7f2dd1832640 1 -- start start 2023-10-30T13:59:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc0c4410 0x7f2dcc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dcc0c4810 0x7f2dcc150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc153ba0 0x7f2dcc152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc0c8580 con 0x7f2dcc0c4410 2023-10-30T13:59:01.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc0c8860 con 0x7f2dcc0c4810 2023-10-30T13:59:01.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc0c8b40 con 0x7f2dcc153ba0 2023-10-30T13:59:01.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc0c4410 0x7f2dcc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc0c4410 0x7f2dcc0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38970/0 (socket says 172.21.15.31:38970) 2023-10-30T13:59:01.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 -- 172.21.15.31:0/971315499 learned_addr learned my addr 172.21.15.31:0/971315499 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:01.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcb7fe640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc153ba0 0x7f2dcc152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dca7fc640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dcc0c4810 0x7f2dcc150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 -- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc153ba0 msgr2=0x7f2dcc152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc153ba0 0x7f2dcc152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 -- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dcc0c4810 msgr2=0x7f2dcc150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dcc0c4810 0x7f2dcc150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 -- 172.21.15.31:0/971315499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc009420 con 0x7f2dcc0c4410 2023-10-30T13:59:01.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcb7fe640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc153ba0 0x7f2dcc152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:01.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dcaffd640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc0c4410 0x7f2dcc0b3e20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0af040 tx=0x7f2dbc01ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2db3fff640 1 -- 172.21.15.31:0/971315499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0b8450 con 0x7f2dcc0c4410 2023-10-30T13:59:01.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2db3fff640 1 -- 172.21.15.31:0/971315499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc0b85b0 con 0x7f2dcc0c4410 2023-10-30T13:59:01.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc0c8e20 con 0x7f2dcc0c4410 2023-10-30T13:59:01.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2db3fff640 1 -- 172.21.15.31:0/971315499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0b8960 con 0x7f2dcc0c4410 2023-10-30T13:59:01.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc0c93e0 con 0x7f2dcc0c4410 2023-10-30T13:59:01.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc0c8c80 con 0x7f2dcc0c4410 2023-10-30T13:59:01.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.472+0000 7f2db3fff640 1 -- 172.21.15.31:0/971315499 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2dbc0aa070 con 0x7f2dcc0c4410 2023-10-30T13:59:01.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.476+0000 7f2db3fff640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2db80664a0 0x7f2db8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.476+0000 7f2db3fff640 1 -- 172.21.15.31:0/971315499 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f2dbc11e9e0 con 0x7f2dcc0c4410 2023-10-30T13:59:01.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.476+0000 7f2dca7fc640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2db80664a0 0x7f2db8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.476+0000 7f2dca7fc640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2db80664a0 0x7f2db8068960 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc15c8b0 tx=0x7f2dc0007be0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.480+0000 7f2db3fff640 1 -- 172.21.15.31:0/971315499 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2dcc0c8c80 con 0x7f2dcc0c4410 2023-10-30T13:59:01.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.636+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2dcc152140 con 0x7f2dcc0c4410 2023-10-30T13:59:01.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.636+0000 7f2db3fff640 1 -- 172.21.15.31:0/971315499 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2dcc152140 con 0x7f2dcc0c4410 2023-10-30T13:59:01.640 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-10-30T13:59:01.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2db80664a0 msgr2=0x7f2db8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2db80664a0 0x7f2db8068960 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc15c8b0 tx=0x7f2dc0007be0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc0c4410 msgr2=0x7f2dcc0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc0c4410 0x7f2dcc0b3e20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0af040 tx=0x7f2dbc01ac00 comp rx=0 tx=0).stop 2023-10-30T13:59:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 shutdown_connections 2023-10-30T13:59:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f2db80664a0 0x7f2db8068960 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2dcc153ba0 0x7f2dcc152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dcc0c4810 0x7f2dcc150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 --2- 172.21.15.31:0/971315499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dcc0c4410 0x7f2dcc0b3e20 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 >> 172.21.15.31:0/971315499 conn(0x7f2dcc071790 msgr2=0x7f2dcc1580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 shutdown_connections 2023-10-30T13:59:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.640+0000 7f2dd1832640 1 -- 172.21.15.31:0/971315499 wait complete. 2023-10-30T13:59:01.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-30T13:59:01.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc959de4640 1 Processor -- start 2023-10-30T13:59:01.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc959de4640 1 -- start start 2023-10-30T13:59:01.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc959de4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 0x7fc9541596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc959de4640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fc9540ca4b0 0x7fc9540c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc959de4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9540c4b10 con 0x7fc9540c5560 2023-10-30T13:59:01.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc959de4640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9540b27b0 con 0x7fc9540ca4b0 2023-10-30T13:59:01.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc959de4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540c50d0 con 0x7fc9540ca0d0 2023-10-30T13:59:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9595e3640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc9540c5560 0x7fc954153520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41660/0 (socket says 172.21.15.31:41660) 2023-10-30T13:59:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9595e3640 1 -- 172.21.15.31:0/2060623167 learned_addr learned my addr 172.21.15.31:0/2060623167 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc953fff640 1 --2- 172.21.15.31:0/2060623167 >> v2:172.21.15.33:3300/0 conn(0x7fc9540ca4b0 0x7fc9540c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc958de2640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 0x7fc9541596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020902562 0 0) 0x7fc9540c4b10 con 0x7fc9540c5560 2023-10-30T13:59:01.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc944003530 con 0x7fc9540c5560 2023-10-30T13:59:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2594605939 0 0) 0x7fc944003530 con 0x7fc9540c5560 2023-10-30T13:59:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 msgr2=0x7fc9541596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 0x7fc9541596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 >> v2:172.21.15.33:3300/0 conn(0x7fc9540ca4b0 msgr2=0x7fc9540c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 --2- 172.21.15.31:0/2060623167 >> v2:172.21.15.33:3300/0 conn(0x7fc9540ca4b0 0x7fc9540c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954159f80 con 0x7fc9540c5560 2023-10-30T13:59:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7fc94c001a00 con 0x7fc9540c5560 2023-10-30T13:59:01.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 >> v1:172.21.15.31:6789/0 conn(0x7fc9540c5560 legacy=0x7fc954153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 0x7fc944003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc944003f00 0x7fc9440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc944004d10 0x7fc944005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540b27b0 con 0x7fc944004d10 2023-10-30T13:59:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540c50d0 con 0x7fc944003f00 2023-10-30T13:59:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.804+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94400b990 con 0x7fc9540ca0d0 2023-10-30T13:59:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9595e3640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc944004d10 0x7fc944005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc958de2640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 0x7fc944003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc953fff640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc944003f00 0x7fc9440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9595e3640 1 -- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 msgr2=0x7fc944003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9595e3640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc9540ca0d0 0x7fc944003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9595e3640 1 -- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc944003f00 msgr2=0x7fc9440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9595e3640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc944003f00 0x7fc9440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9595e3640 1 -- 172.21.15.31:0/2060623167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc944003530 con 0x7fc944004d10 2023-10-30T13:59:01.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9595e3640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc944004d10 0x7fc944005190 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00a830 tx=0x7fc94c01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458fb32c243a9cab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0ad630 con 0x7fc944004d10 2023-10-30T13:59:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c0a8070 con 0x7fc944004d10 2023-10-30T13:59:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.808+0000 7fc9537fe640 1 -- 172.21.15.31:0/2060623167 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0ada30 con 0x7fc944004d10 2023-10-30T13:59:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- 172.21.15.31:0/2060623167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc944004d10 msgr2=0x7fc944005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 --2- 172.21.15.31:0/2060623167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc944004d10 0x7fc944005190 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00a830 tx=0x7fc94c01d300 comp rx=0 tx=0).stop 2023-10-30T13:59:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 -- 172.21.15.31:0/2060623167 reap_dead start 2023-10-30T13:59:01.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- 172.21.15.31:0/2060623167 shutdown_connections 2023-10-30T13:59:01.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- 172.21.15.31:0/2060623167 >> 172.21.15.31:0/2060623167 conn(0x7fc95406bf20 msgr2=0x7fc95406c330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:01.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- 172.21.15.31:0/2060623167 shutdown_connections 2023-10-30T13:59:01.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- 172.21.15.31:0/2060623167 wait complete. 2023-10-30T13:59:01.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 Processor -- start 2023-10-30T13:59:01.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- start start 2023-10-30T13:59:01.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc944004d10 0x7fc9541579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc954154460 0x7fc9541548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc954158960 0x7fc954158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540b4c00 con 0x7fc954158960 2023-10-30T13:59:01.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540b4060 con 0x7fc954154460 2023-10-30T13:59:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc959de4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540b42d0 con 0x7fc944004d10 2023-10-30T13:59:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc954158960 0x7fc954158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc954158960 0x7fc954158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38986/0 (socket says 172.21.15.31:38986) 2023-10-30T13:59:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 -- 172.21.15.31:0/1789931065 learned_addr learned my addr 172.21.15.31:0/1789931065 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc953fff640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc954154460 0x7fc9541548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc958de2640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc944004d10 0x7fc9541579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 -- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc944004d10 msgr2=0x7fc9541579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc944004d10 0x7fc9541579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 -- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc954154460 msgr2=0x7fc9541548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc954154460 0x7fc9541548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 -- 172.21.15.31:0/1789931065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9540b45b0 con 0x7fc954158960 2023-10-30T13:59:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.812+0000 7fc9595e3640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc954158960 0x7fc954158de0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00a830 tx=0x7fc944005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc9517fa640 1 -- 172.21.15.31:0/1789931065 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0017c0 con 0x7fc954158960 2023-10-30T13:59:01.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc9517fa640 1 -- 172.21.15.31:0/1789931065 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c001920 con 0x7fc954158960 2023-10-30T13:59:01.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc9517fa640 1 -- 172.21.15.31:0/1789931065 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c004d70 con 0x7fc954158960 2023-10-30T13:59:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9540b4890 con 0x7fc954158960 2023-10-30T13:59:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc954157ef0 con 0x7fc954158960 2023-10-30T13:59:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc954067540 con 0x7fc954158960 2023-10-30T13:59:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc9517fa640 1 -- 172.21.15.31:0/1789931065 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc94c0aa030 con 0x7fc954158960 2023-10-30T13:59:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc9517fa640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc9380664a0 0x7fc938068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc9517fa640 1 -- 172.21.15.31:0/1789931065 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fc94c11db30 con 0x7fc954158960 2023-10-30T13:59:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.816+0000 7fc958de2640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc9380664a0 0x7fc938068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.820+0000 7fc958de2640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc9380664a0 0x7fc938068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc948009b20 tx=0x7fc948008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.820+0000 7fc9517fa640 1 -- 172.21.15.31:0/1789931065 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc94c0eaa60 con 0x7fc954158960 2023-10-30T13:59:01.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc954150810 con 0x7fc954158960 2023-10-30T13:59:01.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc9517fa640 1 -- 172.21.15.31:0/1789931065 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 22 entries v56) v1 ==== 81+0+1332 (secure 0 0 0) 0x7fc94c0bb080 con 0x7fc954158960 2023-10-30T13:59:01.994 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6836/2267972067 2023-10-31T13:56:02.130126+0000 2023-10-30T13:59:01.995 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6836/3928996203 2023-10-31T13:53:47.073133+0000 2023-10-30T13:59:01.995 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6834/3928996203 2023-10-31T13:53:47.073133+0000 2023-10-30T13:59:01.995 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6837/327248983 2023-10-31T13:51:22.020336+0000 2023-10-30T13:59:01.995 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6836/327248983 2023-10-31T13:51:22.020336+0000 2023-10-30T13:59:01.996 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6834/2281022644 2023-10-31T13:57:42.156901+0000 2023-10-30T13:59:01.996 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6837/2267972067 2023-10-31T13:56:02.130126+0000 2023-10-30T13:59:01.996 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6836/2545654386 2023-10-31T13:36:26.753730+0000 2023-10-30T13:59:01.996 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6836/1956809670 2023-10-31T13:39:06.794112+0000 2023-10-30T13:59:01.997 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6836/15857146 2023-10-31T13:49:36.991529+0000 2023-10-30T13:59:01.997 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6837/1956809670 2023-10-31T13:39:06.794112+0000 2023-10-30T13:59:01.997 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6835/3402459191 2023-10-31T13:40:56.840815+0000 2023-10-30T13:59:01.998 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6836/2281022644 2023-10-31T13:57:42.156901+0000 2023-10-30T13:59:01.998 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6836/210349556 2023-10-31T13:44:26.910290+0000 2023-10-30T13:59:01.998 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6836/349618961 2023-10-31T13:47:11.952274+0000 2023-10-30T13:59:01.998 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6838/2419808617 2023-10-31T13:43:31.895625+0000 2023-10-30T13:59:01.999 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6834/2545654386 2023-10-31T13:36:26.753730+0000 2023-10-30T13:59:01.999 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6834/15857146 2023-10-31T13:49:36.991529+0000 2023-10-30T13:59:01.999 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6839/2419808617 2023-10-31T13:43:31.895625+0000 2023-10-30T13:59:01.999 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6837/3402459191 2023-10-31T13:40:56.840815+0000 2023-10-30T13:59:02.000 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.33:6834/210349556 2023-10-31T13:44:26.910290+0000 2023-10-30T13:59:02.000 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6837/349618961 2023-10-31T13:47:11.952274+0000 2023-10-30T13:59:02.001 INFO:teuthology.orchestra.run.smithi031.stderr:listed 22 entries 2023-10-30T13:59:02.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc9380664a0 msgr2=0x7fc938068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc9380664a0 0x7fc938068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc948009b20 tx=0x7fc948008040 comp rx=0 tx=0).stop 2023-10-30T13:59:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc954158960 msgr2=0x7fc954158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc954158960 0x7fc954158de0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00a830 tx=0x7fc944005800 comp rx=0 tx=0).stop 2023-10-30T13:59:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 shutdown_connections 2023-10-30T13:59:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc9380664a0 0x7fc938068960 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc954158960 0x7fc954158de0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc954154460 0x7fc9541548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 --2- 172.21.15.31:0/1789931065 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc944004d10 0x7fc9541579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:02.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 >> 172.21.15.31:0/1789931065 conn(0x7fc95406bf20 msgr2=0x7fc9541519b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:02.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 shutdown_connections 2023-10-30T13:59:02.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:01.992+0000 7fc959de4640 1 -- 172.21.15.31:0/1789931065 wait complete. 2023-10-30T13:59:02.015 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-30T13:59:02.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:59:02.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-30T13:59:03.870 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T13:59:03.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:59:03.871 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-30T13:59:03.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:59:03.881 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2023-10-30T13:59:03.934 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-30T13:59:03.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:59:03.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T13:59:03.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:59:03.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2023-10-30T13:59:04.199 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T13:59:04.215 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T13:59:04.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:59:04.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3389ac9640 1 Processor -- start 2023-10-30T13:59:04.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3389ac9640 1 -- start start 2023-10-30T13:59:04.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3389ac9640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3384153c80 0x7f338415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3389ac9640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f33840c42a0 0x7f33840ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3389ac9640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3384159f20 con 0x7f33840c4a10 2023-10-30T13:59:04.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3389ac9640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338415dec0 con 0x7f3384153c80 2023-10-30T13:59:04.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3389ac9640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33841599a0 con 0x7f33840c42a0 2023-10-30T13:59:04.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f33837fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f3384153c80 0x7f338415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3382ffd640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f33840c4a10 0x7f338415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:41664/0 (socket says 172.21.15.31:41664) 2023-10-30T13:59:04.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3382ffd640 1 -- 172.21.15.31:0/2210562672 learned_addr learned my addr 172.21.15.31:0/2210562672 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:04.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.360+0000 7f3383fff640 1 --2- 172.21.15.31:0/2210562672 >> v2:172.21.15.33:3300/0 conn(0x7f33840c42a0 0x7f33840ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33837fe640 1 -- 172.21.15.31:0/2210562672 >> v2:172.21.15.33:3300/0 conn(0x7f33840c42a0 msgr2=0x7f33840ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:04.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33837fe640 1 --2- 172.21.15.31:0/2210562672 >> v2:172.21.15.33:3300/0 conn(0x7f33840c42a0 0x7f33840ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33837fe640 1 -- 172.21.15.31:0/2210562672 >> v1:172.21.15.31:6789/0 conn(0x7f33840c4a10 legacy=0x7f338415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33837fe640 1 -- 172.21.15.31:0/2210562672 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338415d620 con 0x7f3384153c80 2023-10-30T13:59:04.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33837fe640 1 --2- 172.21.15.31:0/2210562672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3384153c80 0x7f338415cf90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f33840b1d40 tx=0x7f337401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260aea0394e819d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:04.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33827fc640 1 -- 172.21.15.31:0/2210562672 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33740a8950 con 0x7f3384153c80 2023-10-30T13:59:04.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33827fc640 1 -- 172.21.15.31:0/2210562672 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33740a8ab0 con 0x7f3384153c80 2023-10-30T13:59:04.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f33827fc640 1 -- 172.21.15.31:0/2210562672 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33740a8e40 con 0x7f3384153c80 2023-10-30T13:59:04.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 -- 172.21.15.31:0/2210562672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3384153c80 msgr2=0x7f338415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 --2- 172.21.15.31:0/2210562672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3384153c80 0x7f338415cf90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f33840b1d40 tx=0x7f337401ec50 comp rx=0 tx=0).stop 2023-10-30T13:59:04.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 -- 172.21.15.31:0/2210562672 shutdown_connections 2023-10-30T13:59:04.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 --2- 172.21.15.31:0/2210562672 >> v2:172.21.15.33:3300/0 conn(0x7f33840c42a0 0x7f33840ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 --2- 172.21.15.31:0/2210562672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3384153c80 0x7f338415cf90 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 -- 172.21.15.31:0/2210562672 >> 172.21.15.31:0/2210562672 conn(0x7f33840715c0 msgr2=0x7f33840719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:04.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 -- 172.21.15.31:0/2210562672 shutdown_connections 2023-10-30T13:59:04.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.364+0000 7f3389ac9640 1 -- 172.21.15.31:0/2210562672 wait complete. 2023-10-30T13:59:04.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 Processor -- start 2023-10-30T13:59:04.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 -- start start 2023-10-30T13:59:04.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33840c42a0 0x7f3384152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33840c4a10 0x7f338415b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3384153c80 0x7f338415b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338415afe0 con 0x7f33840c42a0 2023-10-30T13:59:04.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3384152c00 con 0x7f3384153c80 2023-10-30T13:59:04.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33840b38b0 con 0x7f33840c4a10 2023-10-30T13:59:04.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33840c42a0 0x7f3384152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33840c42a0 0x7f3384152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39010/0 (socket says 172.21.15.31:39010) 2023-10-30T13:59:04.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 -- 172.21.15.31:0/3071758572 learned_addr learned my addr 172.21.15.31:0/3071758572 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:04.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3383fff640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3384153c80 0x7f338415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3382ffd640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33840c4a10 0x7f338415b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 -- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33840c4a10 msgr2=0x7f338415b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33840c4a10 0x7f338415b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 -- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3384153c80 msgr2=0x7f338415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3384153c80 0x7f338415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 -- 172.21.15.31:0/3071758572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3374009420 con 0x7f33840c42a0 2023-10-30T13:59:04.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f33837fe640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33840c42a0 0x7f3384152130 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f33740b3040 tx=0x7f337401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:04.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f336bfff640 1 -- 172.21.15.31:0/3071758572 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33740b0ea0 con 0x7f33840c42a0 2023-10-30T13:59:04.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f336bfff640 1 -- 172.21.15.31:0/3071758572 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33740b9410 con 0x7f33840c42a0 2023-10-30T13:59:04.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f336bfff640 1 -- 172.21.15.31:0/3071758572 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33740b97c0 con 0x7f33840c42a0 2023-10-30T13:59:04.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.368+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33840b3b90 con 0x7f33840c42a0 2023-10-30T13:59:04.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.372+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338415caf0 con 0x7f33840c42a0 2023-10-30T13:59:04.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.372+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338404e170 con 0x7f33840c42a0 2023-10-30T13:59:04.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.372+0000 7f336bfff640 1 -- 172.21.15.31:0/3071758572 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33740b7040 con 0x7f33840c42a0 2023-10-30T13:59:04.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.372+0000 7f336bfff640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33700664a0 0x7f3370068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.372+0000 7f336bfff640 1 -- 172.21.15.31:0/3071758572 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f3374121f50 con 0x7f33840c42a0 2023-10-30T13:59:04.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.372+0000 7f3382ffd640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33700664a0 0x7f3370068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.376+0000 7f3382ffd640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33700664a0 0x7f3370068960 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f33840c5310 tx=0x7f3378009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:04.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.376+0000 7f336bfff640 1 -- 172.21.15.31:0/3071758572 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33740eef10 con 0x7f33840c42a0 2023-10-30T13:59:04.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.532+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f338415dbe0 con 0x7f33700664a0 2023-10-30T13:59:04.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.556+0000 7f336bfff640 1 -- 172.21.15.31:0/3071758572 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292756 (secure 0 0 0) 0x7f3378001c80 con 0x7f33700664a0 2023-10-30T13:59:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:04.562 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:59:04.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33700664a0 msgr2=0x7f3370068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33700664a0 0x7f3370068960 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f33840c5310 tx=0x7f3378009040 comp rx=0 tx=0).stop 2023-10-30T13:59:04.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33840c42a0 msgr2=0x7f3384152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33840c42a0 0x7f3384152130 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f33740b3040 tx=0x7f337401b9f0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 shutdown_connections 2023-10-30T13:59:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f33700664a0 0x7f3370068960 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3384153c80 0x7f338415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f33840c4a10 0x7f338415b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 --2- 172.21.15.31:0/3071758572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33840c42a0 0x7f3384152130 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 >> 172.21.15.31:0/3071758572 conn(0x7f33840715c0 msgr2=0x7f3384158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:04.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 shutdown_connections 2023-10-30T13:59:04.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.560+0000 7f3389ac9640 1 -- 172.21.15.31:0/3071758572 wait complete. 2023-10-30T13:59:04.580 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":774,"stamp":"2023-10-30T13:59:03.714457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1214691686,"num_objects":9684,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9684,"num_whiteouts":0,"num_read":8937,"num_read_kb":1843152,"num_write":57224,"num_write_kb":2316674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60045,"ondisk_log_size":60045,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3138908,"kb_used_data":2417628,"kb_used_omap":12,"kb_used_meta":721203,"kb_avail":751835812,"statfs":{"total":773094113280,"available":769879871488,"internally_reserved":0,"allocated":2475651072,"data_stored":2423363610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":738512440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28176770,"num_objects":827,"num_object_clones":0,"num_object_copies":1654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1154,"num_write_kb":28302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004989"},"pg_stats":[{"pgid":"3.3f","version":"56'647","reported_seq":2866,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.625135+0000","last_change":"2023-10-30T13:45:28.601107+0000","last_active":"2023-10-30T13:58:47.625135+0000","last_peered":"2023-10-30T13:58:47.625135+0000","last_clean":"2023-10-30T13:58:47.625135+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:58:47.625135+0000","last_undegraded":"2023-10-30T13:58:47.625135+0000","last_fullsized":"2023-10-30T13:58:47.625135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:58.462635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098613959999999997,"stat_sum":{"num_bytes":30381033,"num_objects":125,"num_object_clones":0,"num_object_copies":250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":124,"num_read_kb":8207,"num_write":615,"num_write_kb":38998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"56'626","reported_seq":2825,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.650883+0000","last_change":"2023-10-30T13:45:27.935552+0000","last_active":"2023-10-30T13:58:47.650883+0000","last_peered":"2023-10-30T13:58:47.650883+0000","last_clean":"2023-10-30T13:58:47.650883+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:58:47.650883+0000","last_undegraded":"2023-10-30T13:58:47.650883+0000","last_fullsized":"2023-10-30T13:58:47.650883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:50.606021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001955118,"stat_sum":{"num_bytes":9798818,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":104,"num_read_kb":11,"num_write":587,"num_write_kb":23396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"56'679","reported_seq":2881,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.382336+0000","last_change":"2023-10-30T13:45:27.939453+0000","last_active":"2023-10-30T13:59:02.382336+0000","last_peered":"2023-10-30T13:59:02.382336+0000","last_clean":"2023-10-30T13:59:02.382336+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:59:02.382336+0000","last_undegraded":"2023-10-30T13:59:02.382336+0000","last_fullsized":"2023-10-30T13:59:02.382336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:59.423451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021120940000000001,"stat_sum":{"num_bytes":23390266,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":110,"num_read_kb":20497,"num_write":638,"num_write_kb":47001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"56'663","reported_seq":2888,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.599578+0000","last_change":"2023-10-30T13:45:28.487484+0000","last_active":"2023-10-30T13:58:47.599578+0000","last_peered":"2023-10-30T13:58:47.599578+0000","last_clean":"2023-10-30T13:58:47.599578+0000","last_became_active":"2023-10-30T13:35:48.779909+0000","last_became_peered":"2023-10-30T13:35:48.779909+0000","last_unstale":"2023-10-30T13:58:47.599578+0000","last_undegraded":"2023-10-30T13:58:47.599578+0000","last_fullsized":"2023-10-30T13:58:47.599578+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:51.343491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001317775,"stat_sum":{"num_bytes":9891362,"num_objects":139,"num_object_clones":0,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":133,"num_read_kb":8205,"num_write":636,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"56'666","reported_seq":2869,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.626617+0000","last_change":"2023-10-30T13:45:28.597729+0000","last_active":"2023-10-30T13:58:47.626617+0000","last_peered":"2023-10-30T13:58:47.626617+0000","last_clean":"2023-10-30T13:58:47.626617+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:58:47.626617+0000","last_undegraded":"2023-10-30T13:58:47.626617+0000","last_fullsized":"2023-10-30T13:58:47.626617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:29.242932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007614844,"stat_sum":{"num_bytes":19656703,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":110,"num_read_kb":8208,"num_write":638,"num_write_kb":39882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"56'661","reported_seq":2858,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.395628+0000","last_change":"2023-10-30T13:45:27.941070+0000","last_active":"2023-10-30T13:59:02.395628+0000","last_peered":"2023-10-30T13:59:02.395628+0000","last_clean":"2023-10-30T13:59:02.395628+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:59:02.395628+0000","last_undegraded":"2023-10-30T13:59:02.395628+0000","last_fullsized":"2023-10-30T13:59:02.395628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:40.443395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030849649999999998,"stat_sum":{"num_bytes":19227960,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":104,"num_read_kb":7536,"num_write":633,"num_write_kb":20026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"56'662","reported_seq":2846,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.643842+0000","last_change":"2023-10-30T13:45:28.597665+0000","last_active":"2023-10-30T13:58:47.643842+0000","last_peered":"2023-10-30T13:58:47.643842+0000","last_clean":"2023-10-30T13:58:47.643842+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:58:47.643842+0000","last_undegraded":"2023-10-30T13:58:47.643842+0000","last_fullsized":"2023-10-30T13:58:47.643842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:44.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082986029999999999,"stat_sum":{"num_bytes":9880254,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":91,"num_read_kb":11,"num_write":635,"num_write_kb":29079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"56'645","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.642291+0000","last_change":"2023-10-30T13:45:28.780454+0000","last_active":"2023-10-30T13:58:47.642291+0000","last_peered":"2023-10-30T13:58:47.642291+0000","last_clean":"2023-10-30T13:58:47.642291+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:58:47.642291+0000","last_undegraded":"2023-10-30T13:58:47.642291+0000","last_fullsized":"2023-10-30T13:58:47.642291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:57.938730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021728070000000001,"stat_sum":{"num_bytes":10096886,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":132,"num_read_kb":1147,"num_write":620,"num_write_kb":24646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"56'631","reported_seq":2847,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.628018+0000","last_change":"2023-10-30T13:45:28.601219+0000","last_active":"2023-10-30T13:58:47.628018+0000","last_peered":"2023-10-30T13:58:47.628018+0000","last_clean":"2023-10-30T13:58:47.628018+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:58:47.628018+0000","last_undegraded":"2023-10-30T13:58:47.628018+0000","last_fullsized":"2023-10-30T13:58:47.628018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:53.268188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010045174,"stat_sum":{"num_bytes":10190840,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":124,"num_read_kb":4115,"num_write":586,"num_write_kb":23657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"56'612","reported_seq":2818,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.344375+0000","last_change":"2023-10-30T13:45:28.050787+0000","last_active":"2023-10-30T13:59:02.344375+0000","last_peered":"2023-10-30T13:59:02.344375+0000","last_clean":"2023-10-30T13:59:02.344375+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:59:02.344375+0000","last_undegraded":"2023-10-30T13:59:02.344375+0000","last_fullsized":"2023-10-30T13:59:02.344375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:02.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047277760000000004,"stat_sum":{"num_bytes":20230336,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":113,"num_read_kb":28,"num_write":585,"num_write_kb":25967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"56'610","reported_seq":2800,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.302667+0000","last_change":"2023-10-30T13:45:28.050006+0000","last_active":"2023-10-30T13:59:02.302667+0000","last_peered":"2023-10-30T13:59:02.302667+0000","last_clean":"2023-10-30T13:59:02.302667+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:59:02.302667+0000","last_undegraded":"2023-10-30T13:59:02.302667+0000","last_fullsized":"2023-10-30T13:59:02.302667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:30.724902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00429868,"stat_sum":{"num_bytes":2384023,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":97,"num_read_kb":17,"num_write":586,"num_write_kb":9055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"56'626","reported_seq":2835,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.294793+0000","last_change":"2023-10-30T13:45:27.946083+0000","last_active":"2023-10-30T13:59:02.294793+0000","last_peered":"2023-10-30T13:59:02.294793+0000","last_clean":"2023-10-30T13:59:02.294793+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:59:02.294793+0000","last_undegraded":"2023-10-30T13:59:02.294793+0000","last_fullsized":"2023-10-30T13:59:02.294793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:19.253547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066533010000000004,"stat_sum":{"num_bytes":18341132,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":8208,"num_write":596,"num_write_kb":32418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"56'610","reported_seq":2816,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.362815+0000","last_change":"2023-10-30T13:45:28.780475+0000","last_active":"2023-10-30T13:58:51.362815+0000","last_peered":"2023-10-30T13:58:51.362815+0000","last_clean":"2023-10-30T13:58:51.362815+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:58:51.362815+0000","last_undegraded":"2023-10-30T13:58:51.362815+0000","last_fullsized":"2023-10-30T13:58:51.362815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:10:02.225437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002338237,"stat_sum":{"num_bytes":13957458,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":114,"num_read_kb":12311,"num_write":572,"num_write_kb":36850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"56'731","reported_seq":2944,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.434802+0000","last_change":"2023-10-30T13:45:27.940425+0000","last_active":"2023-10-30T13:59:02.434802+0000","last_peered":"2023-10-30T13:59:02.434802+0000","last_clean":"2023-10-30T13:59:02.434802+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:59:02.434802+0000","last_undegraded":"2023-10-30T13:59:02.434802+0000","last_fullsized":"2023-10-30T13:59:02.434802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:57.204443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029248669999999998,"stat_sum":{"num_bytes":22804734,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":119,"num_read_kb":24041,"num_write":691,"num_write_kb":57342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"56'690","reported_seq":2896,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.631995+0000","last_change":"2023-10-30T13:45:27.934641+0000","last_active":"2023-10-30T13:58:47.631995+0000","last_peered":"2023-10-30T13:58:47.631995+0000","last_clean":"2023-10-30T13:58:47.631995+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:58:47.631995+0000","last_undegraded":"2023-10-30T13:58:47.631995+0000","last_fullsized":"2023-10-30T13:58:47.631995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:06:31.910235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021050410000000002,"stat_sum":{"num_bytes":18067659,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":112,"num_read_kb":7334,"num_write":654,"num_write_kb":39561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"56'726","reported_seq":2943,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.419953+0000","last_change":"2023-10-30T13:45:27.943054+0000","last_active":"2023-10-30T13:59:02.419953+0000","last_peered":"2023-10-30T13:59:02.419953+0000","last_clean":"2023-10-30T13:59:02.419953+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:59:02.419953+0000","last_undegraded":"2023-10-30T13:59:02.419953+0000","last_fullsized":"2023-10-30T13:59:02.419953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:29.930187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052579080000000004,"stat_sum":{"num_bytes":35259912,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":123,"num_read_kb":12306,"num_write":693,"num_write_kb":43349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"56'699","reported_seq":2895,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.379018+0000","last_change":"2023-10-30T13:45:27.950182+0000","last_active":"2023-10-30T13:59:02.379018+0000","last_peered":"2023-10-30T13:59:02.379018+0000","last_clean":"2023-10-30T13:59:02.379018+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:59:02.379018+0000","last_undegraded":"2023-10-30T13:59:02.379018+0000","last_fullsized":"2023-10-30T13:59:02.379018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:09.257269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052923609999999998,"stat_sum":{"num_bytes":14461920,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":103,"num_read_kb":4113,"num_write":659,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"56'628","reported_seq":2840,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.619440+0000","last_change":"2023-10-30T13:45:28.487735+0000","last_active":"2023-10-30T13:58:47.619440+0000","last_peered":"2023-10-30T13:58:47.619440+0000","last_clean":"2023-10-30T13:58:47.619440+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:58:47.619440+0000","last_undegraded":"2023-10-30T13:58:47.619440+0000","last_fullsized":"2023-10-30T13:58:47.619440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:05.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001369274,"stat_sum":{"num_bytes":15189051,"num_objects":119,"num_object_clones":0,"num_object_copies":238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":118,"num_read_kb":4123,"num_write":600,"num_write_kb":31137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"56'685","reported_seq":2912,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.624637+0000","last_change":"2023-10-30T13:45:28.600018+0000","last_active":"2023-10-30T13:58:47.624637+0000","last_peered":"2023-10-30T13:58:47.624637+0000","last_clean":"2023-10-30T13:58:47.624637+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:58:47.624637+0000","last_undegraded":"2023-10-30T13:58:47.624637+0000","last_fullsized":"2023-10-30T13:58:47.624637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:15.777688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092525119999999992,"stat_sum":{"num_bytes":8625831,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":134,"num_read_kb":8209,"num_write":656,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"56'660","reported_seq":2853,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.287934+0000","last_change":"2023-10-30T13:45:28.051528+0000","last_active":"2023-10-30T13:59:02.287934+0000","last_peered":"2023-10-30T13:59:02.287934+0000","last_clean":"2023-10-30T13:59:02.287934+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:59:02.287934+0000","last_undegraded":"2023-10-30T13:59:02.287934+0000","last_fullsized":"2023-10-30T13:59:02.287934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:24.699943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00502631,"stat_sum":{"num_bytes":6211385,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":100,"num_read_kb":8209,"num_write":620,"num_write_kb":19966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"56'684","reported_seq":2889,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.481517+0000","last_change":"2023-10-30T13:45:27.940012+0000","last_active":"2023-10-30T13:59:02.481517+0000","last_peered":"2023-10-30T13:59:02.481517+0000","last_clean":"2023-10-30T13:59:02.481517+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:59:02.481517+0000","last_undegraded":"2023-10-30T13:59:02.481517+0000","last_fullsized":"2023-10-30T13:59:02.481517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:13.888934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026024469999999999,"stat_sum":{"num_bytes":18097476,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":111,"num_read_kb":21,"num_write":651,"num_write_kb":32402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"56'700","reported_seq":2899,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.370220+0000","last_change":"2023-10-30T13:45:27.947213+0000","last_active":"2023-10-30T13:59:02.370220+0000","last_peered":"2023-10-30T13:59:02.370220+0000","last_clean":"2023-10-30T13:59:02.370220+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:59:02.370220+0000","last_undegraded":"2023-10-30T13:59:02.370220+0000","last_fullsized":"2023-10-30T13:59:02.370220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:24.312446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054353270000000002,"stat_sum":{"num_bytes":10272429,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":105,"num_read_kb":4117,"num_write":665,"num_write_kb":16082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"56'675","reported_seq":2882,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.598533+0000","last_change":"2023-10-30T13:45:28.781012+0000","last_active":"2023-10-30T13:58:47.598533+0000","last_peered":"2023-10-30T13:58:47.598533+0000","last_clean":"2023-10-30T13:58:47.598533+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:58:47.598533+0000","last_undegraded":"2023-10-30T13:58:47.598533+0000","last_fullsized":"2023-10-30T13:58:47.598533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:40:44.651276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025617359999999998,"stat_sum":{"num_bytes":13960173,"num_objects":128,"num_object_clones":0,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":114,"num_read_kb":8213,"num_write":647,"num_write_kb":36096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"56'705","reported_seq":2926,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.388279+0000","last_change":"2023-10-30T13:45:27.942487+0000","last_active":"2023-10-30T13:59:02.388279+0000","last_peered":"2023-10-30T13:59:02.388279+0000","last_clean":"2023-10-30T13:59:02.388279+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:59:02.388279+0000","last_undegraded":"2023-10-30T13:59:02.388279+0000","last_fullsized":"2023-10-30T13:59:02.388279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:39.369789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004034279,"stat_sum":{"num_bytes":1595822,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":126,"num_read_kb":16,"num_write":676,"num_write_kb":12551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"56'645","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.596722+0000","last_change":"2023-10-30T13:45:28.782023+0000","last_active":"2023-10-30T13:58:47.596722+0000","last_peered":"2023-10-30T13:58:47.596722+0000","last_clean":"2023-10-30T13:58:47.596722+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:58:47.596722+0000","last_undegraded":"2023-10-30T13:58:47.596722+0000","last_fullsized":"2023-10-30T13:58:47.596722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:09:53.916658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030593479999999999,"stat_sum":{"num_bytes":7588015,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":117,"num_read_kb":4108,"num_write":613,"num_write_kb":17885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"56'712","reported_seq":2918,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.378475+0000","last_change":"2023-10-30T13:45:28.051170+0000","last_active":"2023-10-30T13:59:02.378475+0000","last_peered":"2023-10-30T13:59:02.378475+0000","last_clean":"2023-10-30T13:59:02.378475+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:59:02.378475+0000","last_undegraded":"2023-10-30T13:59:02.378475+0000","last_fullsized":"2023-10-30T13:59:02.378475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:05.790369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049754609999999996,"stat_sum":{"num_bytes":9799460,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":111,"num_read_kb":24,"num_write":679,"num_write_kb":16350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"56'641","reported_seq":2846,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.609884+0000","last_change":"2023-10-30T13:45:27.935505+0000","last_active":"2023-10-30T13:58:47.609884+0000","last_peered":"2023-10-30T13:58:47.609884+0000","last_clean":"2023-10-30T13:58:47.609884+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:58:47.609884+0000","last_undegraded":"2023-10-30T13:58:47.609884+0000","last_fullsized":"2023-10-30T13:58:47.609884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:28.046138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028468439999999999,"stat_sum":{"num_bytes":33204376,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":110,"num_read_kb":16400,"num_write":604,"num_write_kb":35545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"56'610","reported_seq":2816,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.349462+0000","last_change":"2023-10-30T13:45:28.781733+0000","last_active":"2023-10-30T13:58:51.349462+0000","last_peered":"2023-10-30T13:58:51.349462+0000","last_clean":"2023-10-30T13:58:51.349462+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:58:51.349462+0000","last_undegraded":"2023-10-30T13:58:51.349462+0000","last_fullsized":"2023-10-30T13:58:51.349462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:10.458130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002839121,"stat_sum":{"num_bytes":9452766,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":112,"num_read_kb":10701,"num_write":586,"num_write_kb":14944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"56'679","reported_seq":2883,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.489266+0000","last_change":"2023-10-30T13:45:27.941228+0000","last_active":"2023-10-30T13:59:02.489266+0000","last_peered":"2023-10-30T13:59:02.489266+0000","last_clean":"2023-10-30T13:59:02.489266+0000","last_became_active":"2023-10-30T13:35:48.847825+0000","last_became_peered":"2023-10-30T13:35:48.847825+0000","last_unstale":"2023-10-30T13:59:02.489266+0000","last_undegraded":"2023-10-30T13:59:02.489266+0000","last_fullsized":"2023-10-30T13:59:02.489266+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:46.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028647770000000002,"stat_sum":{"num_bytes":7231756,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":102,"num_read_kb":4848,"num_write":653,"num_write_kb":26441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"56'688","reported_seq":2903,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.382620+0000","last_change":"2023-10-30T13:45:27.948917+0000","last_active":"2023-10-30T13:59:02.382620+0000","last_peered":"2023-10-30T13:59:02.382620+0000","last_clean":"2023-10-30T13:59:02.382620+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:59:02.382620+0000","last_undegraded":"2023-10-30T13:59:02.382620+0000","last_fullsized":"2023-10-30T13:59:02.382620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:51.840613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060200970000000003,"stat_sum":{"num_bytes":22333767,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":120,"num_read_kb":4116,"num_write":656,"num_write_kb":43887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"56'753","reported_seq":2987,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.633520+0000","last_change":"2023-10-30T13:45:28.487621+0000","last_active":"2023-10-30T13:58:47.633520+0000","last_peered":"2023-10-30T13:58:47.633520+0000","last_clean":"2023-10-30T13:58:47.633520+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:58:47.633520+0000","last_undegraded":"2023-10-30T13:58:47.633520+0000","last_fullsized":"2023-10-30T13:58:47.633520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:58:52.921718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001263171,"stat_sum":{"num_bytes":10399401,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":139,"num_read_kb":4118,"num_write":726,"num_write_kb":34561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"56'704","reported_seq":2909,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.381095+0000","last_change":"2023-10-30T13:45:28.048779+0000","last_active":"2023-10-30T13:59:02.381095+0000","last_peered":"2023-10-30T13:59:02.381095+0000","last_clean":"2023-10-30T13:59:02.381095+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:59:02.381095+0000","last_undegraded":"2023-10-30T13:59:02.381095+0000","last_fullsized":"2023-10-30T13:59:02.381095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:18.375625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036881090000000002,"stat_sum":{"num_bytes":14815628,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":110,"num_read_kb":20,"num_write":677,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"56'677","reported_seq":2878,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.248436+0000","last_change":"2023-10-30T13:45:27.944400+0000","last_active":"2023-10-30T13:59:02.248436+0000","last_peered":"2023-10-30T13:59:02.248436+0000","last_clean":"2023-10-30T13:59:02.248436+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:59:02.248436+0000","last_undegraded":"2023-10-30T13:59:02.248436+0000","last_fullsized":"2023-10-30T13:59:02.248436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:12.964999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065387930000000002,"stat_sum":{"num_bytes":14536361,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":108,"num_read_kb":8117,"num_write":641,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"56'695","reported_seq":764,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382522+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:58:51.382522+0000","last_peered":"2023-10-30T13:58:51.382522+0000","last_clean":"2023-10-30T13:58:51.382522+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:58:51.382522+0000","last_undegraded":"2023-10-30T13:58:51.382522+0000","last_fullsized":"2023-10-30T13:58:51.382522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45057,"num_write":652,"num_write_kb":16749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"56'665","reported_seq":2885,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.488063+0000","last_change":"2023-10-30T13:45:27.941002+0000","last_active":"2023-10-30T13:59:02.488063+0000","last_peered":"2023-10-30T13:59:02.488063+0000","last_clean":"2023-10-30T13:59:02.488063+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:59:02.488063+0000","last_undegraded":"2023-10-30T13:59:02.488063+0000","last_fullsized":"2023-10-30T13:59:02.488063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:33.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031306289999999998,"stat_sum":{"num_bytes":22886210,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":127,"num_read_kb":6230,"num_write":644,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"56'743","reported_seq":874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.377388+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:58:51.377388+0000","last_peered":"2023-10-30T13:58:51.377388+0000","last_clean":"2023-10-30T13:58:51.377388+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:58:51.377388+0000","last_undegraded":"2023-10-30T13:58:51.377388+0000","last_fullsized":"2023-10-30T13:58:51.377388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21047876,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":114393,"num_write":693,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"56'141","reported_seq":249,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.571367+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:58:52.571367+0000","last_peered":"2023-10-30T13:58:52.571367+0000","last_clean":"2023-10-30T13:58:52.571367+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:58:52.571367+0000","last_undegraded":"2023-10-30T13:58:52.571367+0000","last_fullsized":"2023-10-30T13:58:52.571367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297526,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":20619,"num_write":102,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"56'752","reported_seq":2988,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.668049+0000","last_change":"2023-10-30T13:45:28.781311+0000","last_active":"2023-10-30T13:58:47.668049+0000","last_peered":"2023-10-30T13:58:47.668049+0000","last_clean":"2023-10-30T13:58:47.668049+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:58:47.668049+0000","last_undegraded":"2023-10-30T13:58:47.668049+0000","last_fullsized":"2023-10-30T13:58:47.668049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:42.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027758349999999999,"stat_sum":{"num_bytes":1481853,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":144,"num_read_kb":4120,"num_write":721,"num_write_kb":40298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"56'594","reported_seq":676,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:31.559442+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:58:31.559442+0000","last_peered":"2023-10-30T13:58:31.559442+0000","last_clean":"2023-10-30T13:58:31.559442+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:58:31.559442+0000","last_undegraded":"2023-10-30T13:58:31.559442+0000","last_fullsized":"2023-10-30T13:58:31.559442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":96432,"num_write":540,"num_write_kb":20769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"56'616","reported_seq":2823,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.628582+0000","last_change":"2023-10-30T13:45:27.934297+0000","last_active":"2023-10-30T13:58:47.628582+0000","last_peered":"2023-10-30T13:58:47.628582+0000","last_clean":"2023-10-30T13:58:47.628582+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:58:47.628582+0000","last_undegraded":"2023-10-30T13:58:47.628582+0000","last_fullsized":"2023-10-30T13:58:47.628582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:48.732244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017556029999999999,"stat_sum":{"num_bytes":5986884,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":115,"num_read_kb":8211,"num_write":580,"num_write_kb":32401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"56'127","reported_seq":186,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382513+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:58:51.382513+0000","last_peered":"2023-10-30T13:58:51.382513+0000","last_clean":"2023-10-30T13:58:51.382513+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:58:51.382513+0000","last_undegraded":"2023-10-30T13:58:51.382513+0000","last_fullsized":"2023-10-30T13:58:51.382513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":91,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"56'666","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.619983+0000","last_change":"2023-10-30T13:45:28.487297+0000","last_active":"2023-10-30T13:58:47.619983+0000","last_peered":"2023-10-30T13:58:47.619983+0000","last_clean":"2023-10-30T13:58:47.619983+0000","last_became_active":"2023-10-30T13:35:49.007835+0000","last_became_peered":"2023-10-30T13:35:49.007835+0000","last_unstale":"2023-10-30T13:58:47.619983+0000","last_undegraded":"2023-10-30T13:58:47.619983+0000","last_fullsized":"2023-10-30T13:58:47.619983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:17:24.830286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011632000000000001,"stat_sum":{"num_bytes":15463549,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":101,"num_read_kb":5707,"num_write":628,"num_write_kb":26855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"56'384","reported_seq":457,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.335510+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:59:02.335510+0000","last_peered":"2023-10-30T13:59:02.335510+0000","last_clean":"2023-10-30T13:59:02.335510+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:59:02.335510+0000","last_undegraded":"2023-10-30T13:59:02.335510+0000","last_fullsized":"2023-10-30T13:59:02.335510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20383142,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":61443,"num_write":344,"num_write_kb":20149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"56'654","reported_seq":2875,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.418220+0000","last_change":"2023-10-30T13:45:27.941812+0000","last_active":"2023-10-30T13:59:02.418220+0000","last_peered":"2023-10-30T13:59:02.418220+0000","last_clean":"2023-10-30T13:59:02.418220+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:59:02.418220+0000","last_undegraded":"2023-10-30T13:59:02.418220+0000","last_fullsized":"2023-10-30T13:59:02.418220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:17.717541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028607060000000002,"stat_sum":{"num_bytes":18966683,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":128,"num_read_kb":11963,"num_write":623,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"56'713","reported_seq":798,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.576749+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:58:52.576749+0000","last_peered":"2023-10-30T13:58:52.576749+0000","last_clean":"2023-10-30T13:58:52.576749+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:58:52.576749+0000","last_undegraded":"2023-10-30T13:58:52.576749+0000","last_fullsized":"2023-10-30T13:58:52.576749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":49185,"num_write":677,"num_write_kb":16860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"56'662","reported_seq":2855,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.355115+0000","last_change":"2023-10-30T13:45:27.947116+0000","last_active":"2023-10-30T13:59:02.355115+0000","last_peered":"2023-10-30T13:59:02.355115+0000","last_clean":"2023-10-30T13:59:02.355115+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:59:02.355115+0000","last_undegraded":"2023-10-30T13:59:02.355115+0000","last_fullsized":"2023-10-30T13:59:02.355115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:44.472162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070741379999999998,"stat_sum":{"num_bytes":9996128,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":99,"num_read_kb":8209,"num_write":632,"num_write_kb":23925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"56'1127","reported_seq":1231,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.351736+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:58:56.351736+0000","last_peered":"2023-10-30T13:58:56.351736+0000","last_clean":"2023-10-30T13:58:56.351736+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:58:56.351736+0000","last_undegraded":"2023-10-30T13:58:56.351736+0000","last_fullsized":"2023-10-30T13:58:56.351736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1127,"log_dups_size":0,"ondisk_log_size":1127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":65,"num_read_kb":86059,"num_write":1076,"num_write_kb":17029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"56'645","reported_seq":2842,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.611995+0000","last_change":"2023-10-30T13:45:28.596691+0000","last_active":"2023-10-30T13:58:47.611995+0000","last_peered":"2023-10-30T13:58:47.611995+0000","last_clean":"2023-10-30T13:58:47.611995+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:58:47.611995+0000","last_undegraded":"2023-10-30T13:58:47.611995+0000","last_fullsized":"2023-10-30T13:58:47.611995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:49:41.351825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070810969999999997,"stat_sum":{"num_bytes":18060472,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":104,"num_read_kb":12299,"num_write":619,"num_write_kb":31647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"56'430","reported_seq":527,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:46.388525+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:58:46.388525+0000","last_peered":"2023-10-30T13:58:46.388525+0000","last_clean":"2023-10-30T13:58:46.388525+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:58:46.388525+0000","last_undegraded":"2023-10-30T13:58:46.388525+0000","last_fullsized":"2023-10-30T13:58:46.388525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":71,"num_read_kb":77906,"num_write":383,"num_write_kb":12555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"56'673","reported_seq":2900,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.616338+0000","last_change":"2023-10-30T13:45:28.487276+0000","last_active":"2023-10-30T13:58:47.616338+0000","last_peered":"2023-10-30T13:58:47.616338+0000","last_clean":"2023-10-30T13:58:47.616338+0000","last_became_active":"2023-10-30T13:35:49.010460+0000","last_became_peered":"2023-10-30T13:35:49.010460+0000","last_unstale":"2023-10-30T13:58:47.616338+0000","last_undegraded":"2023-10-30T13:58:47.616338+0000","last_fullsized":"2023-10-30T13:58:47.616338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:47:08.712606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001187553,"stat_sum":{"num_bytes":13551160,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":138,"num_read_kb":8210,"num_write":640,"num_write_kb":55768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"56'177","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.380435+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:58:51.380435+0000","last_peered":"2023-10-30T13:58:51.380435+0000","last_clean":"2023-10-30T13:58:51.380435+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:58:51.380435+0000","last_undegraded":"2023-10-30T13:58:51.380435+0000","last_fullsized":"2023-10-30T13:58:51.380435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":632,"num_write":147,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"56'672","reported_seq":2891,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.617238+0000","last_change":"2023-10-30T13:45:28.599507+0000","last_active":"2023-10-30T13:58:47.617238+0000","last_peered":"2023-10-30T13:58:47.617238+0000","last_clean":"2023-10-30T13:58:47.617238+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:58:47.617238+0000","last_undegraded":"2023-10-30T13:58:47.617238+0000","last_fullsized":"2023-10-30T13:58:47.617238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:11.218437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092364379999999996,"stat_sum":{"num_bytes":15694014,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":126,"num_read_kb":8225,"num_write":638,"num_write_kb":34297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"56'171","reported_seq":234,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.363936+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:58:51.363936+0000","last_peered":"2023-10-30T13:58:51.363936+0000","last_clean":"2023-10-30T13:58:51.363936+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:58:51.363936+0000","last_undegraded":"2023-10-30T13:58:51.363936+0000","last_fullsized":"2023-10-30T13:58:51.363936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4103,"num_write":119,"num_write_kb":4220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"56'655","reported_seq":2848,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.372721+0000","last_change":"2023-10-30T13:45:28.049648+0000","last_active":"2023-10-30T13:59:02.372721+0000","last_peered":"2023-10-30T13:59:02.372721+0000","last_clean":"2023-10-30T13:59:02.372721+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:59:02.372721+0000","last_undegraded":"2023-10-30T13:59:02.372721+0000","last_fullsized":"2023-10-30T13:59:02.372721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:03.640232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045350950000000003,"stat_sum":{"num_bytes":14031523,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":99,"num_read_kb":8208,"num_write":613,"num_write_kb":14857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"56'613","reported_seq":722,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.572185+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:58:51.572185+0000","last_peered":"2023-10-30T13:58:51.572185+0000","last_clean":"2023-10-30T13:58:51.572185+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:58:51.572185+0000","last_undegraded":"2023-10-30T13:58:51.572185+0000","last_fullsized":"2023-10-30T13:58:51.572185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":130,"num_read_kb":41105,"num_write":605,"num_write_kb":12685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"56'710","reported_seq":2920,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.607871+0000","last_change":"2023-10-30T13:45:28.598560+0000","last_active":"2023-10-30T13:58:47.607871+0000","last_peered":"2023-10-30T13:58:47.607871+0000","last_clean":"2023-10-30T13:58:47.607871+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:58:47.607871+0000","last_undegraded":"2023-10-30T13:58:47.607871+0000","last_fullsized":"2023-10-30T13:58:47.607871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:25.254829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089036940000000002,"stat_sum":{"num_bytes":5954339,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":119,"num_read_kb":8210,"num_write":672,"num_write_kb":34961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'351","reported_seq":425,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.537629+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:58:32.537629+0000","last_peered":"2023-10-30T13:58:32.537629+0000","last_clean":"2023-10-30T13:58:32.537629+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:58:32.537629+0000","last_undegraded":"2023-10-30T13:58:32.537629+0000","last_fullsized":"2023-10-30T13:58:32.537629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":28724,"num_write":357,"num_write_kb":8477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"56'621","reported_seq":2813,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.600642+0000","last_change":"2023-10-30T13:45:28.600359+0000","last_active":"2023-10-30T13:58:47.600642+0000","last_peered":"2023-10-30T13:58:47.600642+0000","last_clean":"2023-10-30T13:58:47.600642+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:58:47.600642+0000","last_undegraded":"2023-10-30T13:58:47.600642+0000","last_fullsized":"2023-10-30T13:58:47.600642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:40.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094781629999999995,"stat_sum":{"num_bytes":5933515,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":99,"num_read_kb":8205,"num_write":588,"num_write_kb":15719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"56'638","reported_seq":715,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:49.590466+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:58:49.590466+0000","last_peered":"2023-10-30T13:58:49.590466+0000","last_clean":"2023-10-30T13:58:49.590466+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:58:49.590466+0000","last_undegraded":"2023-10-30T13:58:49.590466+0000","last_fullsized":"2023-10-30T13:58:49.590466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33457,"num_write":581,"num_write_kb":12711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"56'639","reported_seq":2843,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.631413+0000","last_change":"2023-10-30T13:45:28.601986+0000","last_active":"2023-10-30T13:58:47.631413+0000","last_peered":"2023-10-30T13:58:47.631413+0000","last_clean":"2023-10-30T13:58:47.631413+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:58:47.631413+0000","last_undegraded":"2023-10-30T13:58:47.631413+0000","last_fullsized":"2023-10-30T13:58:47.631413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:59.052456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010320058,"stat_sum":{"num_bytes":13685218,"num_objects":132,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":111,"num_read_kb":8210,"num_write":603,"num_write_kb":35843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"56'400","reported_seq":474,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.554499+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:58:32.554499+0000","last_peered":"2023-10-30T13:58:32.554499+0000","last_clean":"2023-10-30T13:58:32.554499+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:58:32.554499+0000","last_undegraded":"2023-10-30T13:58:32.554499+0000","last_fullsized":"2023-10-30T13:58:32.554499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582982,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":36872,"num_write":369,"num_write_kb":12516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"56'744","reported_seq":2973,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.369177+0000","last_change":"2023-10-30T13:45:27.948762+0000","last_active":"2023-10-30T13:59:02.369177+0000","last_peered":"2023-10-30T13:59:02.369177+0000","last_clean":"2023-10-30T13:59:02.369177+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:59:02.369177+0000","last_undegraded":"2023-10-30T13:59:02.369177+0000","last_fullsized":"2023-10-30T13:59:02.369177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.722813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066640720000000001,"stat_sum":{"num_bytes":18395347,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":135,"num_read_kb":4114,"num_write":711,"num_write_kb":24326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"56'222","reported_seq":290,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.553986+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:58:32.553986+0000","last_peered":"2023-10-30T13:58:32.553986+0000","last_clean":"2023-10-30T13:58:32.553986+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:58:32.553986+0000","last_undegraded":"2023-10-30T13:58:32.553986+0000","last_fullsized":"2023-10-30T13:58:32.553986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":20494,"num_write":198,"num_write_kb":4363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"56'645","reported_seq":2844,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.612268+0000","last_change":"2023-10-30T13:45:28.598725+0000","last_active":"2023-10-30T13:58:47.612268+0000","last_peered":"2023-10-30T13:58:47.612268+0000","last_clean":"2023-10-30T13:58:47.612268+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:58:47.612268+0000","last_undegraded":"2023-10-30T13:58:47.612268+0000","last_fullsized":"2023-10-30T13:58:47.612268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:29.217863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084474579999999997,"stat_sum":{"num_bytes":17958459,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":106,"num_read_kb":8207,"num_write":607,"num_write_kb":40298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:57:42.172646+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:57:42.172646+0000","last_peered":"2023-10-30T13:57:42.172646+0000","last_clean":"2023-10-30T13:57:42.172646+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:57:42.172646+0000","last_undegraded":"2023-10-30T13:57:42.172646+0000","last_fullsized":"2023-10-30T13:57:42.172646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'346","reported_seq":413,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.460048+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:58:56.460048+0000","last_peered":"2023-10-30T13:58:56.460048+0000","last_clean":"2023-10-30T13:58:56.460048+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:58:56.460048+0000","last_undegraded":"2023-10-30T13:58:56.460048+0000","last_fullsized":"2023-10-30T13:58:56.460048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388674,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":20491,"num_write":340,"num_write_kb":8443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'643","reported_seq":2845,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.646867+0000","last_change":"2023-10-30T13:45:28.780750+0000","last_active":"2023-10-30T13:58:47.646867+0000","last_peered":"2023-10-30T13:58:47.646867+0000","last_clean":"2023-10-30T13:58:47.646867+0000","last_became_active":"2023-10-30T13:35:48.847905+0000","last_became_peered":"2023-10-30T13:35:48.847905+0000","last_unstale":"2023-10-30T13:58:47.646867+0000","last_undegraded":"2023-10-30T13:58:47.646867+0000","last_fullsized":"2023-10-30T13:58:47.646867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:37.243152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002389721,"stat_sum":{"num_bytes":10609565,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":100,"num_read_kb":11,"num_write":618,"num_write_kb":20098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"56'352","reported_seq":439,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.575618+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:58:51.575618+0000","last_peered":"2023-10-30T13:58:51.575618+0000","last_clean":"2023-10-30T13:58:51.575618+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:58:51.575618+0000","last_undegraded":"2023-10-30T13:58:51.575618+0000","last_fullsized":"2023-10-30T13:58:51.575618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":102412,"num_write":306,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"56'696","reported_seq":2908,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.336775+0000","last_change":"2023-10-30T13:45:27.939160+0000","last_active":"2023-10-30T13:59:02.336775+0000","last_peered":"2023-10-30T13:59:02.336775+0000","last_clean":"2023-10-30T13:59:02.336775+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:59:02.336775+0000","last_undegraded":"2023-10-30T13:59:02.336775+0000","last_fullsized":"2023-10-30T13:59:02.336775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:16.251173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018610090000000001,"stat_sum":{"num_bytes":24271212,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":117,"num_read_kb":10,"num_write":671,"num_write_kb":33782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"56'1498","reported_seq":2022,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.567569+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:58:52.567569+0000","last_peered":"2023-10-30T13:58:52.567569+0000","last_clean":"2023-10-30T13:58:52.567569+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:58:52.567569+0000","last_undegraded":"2023-10-30T13:58:52.567569+0000","last_fullsized":"2023-10-30T13:58:52.567569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959716,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":475,"num_read_kb":12479,"num_write":1460,"num_write_kb":17319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'629","reported_seq":2840,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.636121+0000","last_change":"2023-10-30T13:45:27.934841+0000","last_active":"2023-10-30T13:58:47.636121+0000","last_peered":"2023-10-30T13:58:47.636121+0000","last_clean":"2023-10-30T13:58:47.636121+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:58:47.636121+0000","last_undegraded":"2023-10-30T13:58:47.636121+0000","last_fullsized":"2023-10-30T13:58:47.636121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:35.639553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022228600000000001,"stat_sum":{"num_bytes":9645659,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":116,"num_read_kb":4117,"num_write":587,"num_write_kb":24506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'628","reported_seq":701,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.549494+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:58:32.549494+0000","last_peered":"2023-10-30T13:58:32.549494+0000","last_clean":"2023-10-30T13:58:32.549494+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:58:32.549494+0000","last_undegraded":"2023-10-30T13:58:32.549494+0000","last_fullsized":"2023-10-30T13:58:32.549494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":61440,"num_write":565,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'712","reported_seq":2908,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.318656+0000","last_change":"2023-10-30T13:45:27.945262+0000","last_active":"2023-10-30T13:59:02.318656+0000","last_peered":"2023-10-30T13:59:02.318656+0000","last_clean":"2023-10-30T13:59:02.318656+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:59:02.318656+0000","last_undegraded":"2023-10-30T13:59:02.318656+0000","last_fullsized":"2023-10-30T13:59:02.318656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:31.086225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066434720000000001,"stat_sum":{"num_bytes":26554596,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":101,"num_read_kb":3528,"num_write":689,"num_write_kb":37408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"56'726","reported_seq":791,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.543895+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:58:32.543895+0000","last_peered":"2023-10-30T13:58:32.543895+0000","last_clean":"2023-10-30T13:58:32.543895+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:58:32.543895+0000","last_undegraded":"2023-10-30T13:58:32.543895+0000","last_fullsized":"2023-10-30T13:58:32.543895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":18914,"num_write":694,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"56'695","reported_seq":2898,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.390308+0000","last_change":"2023-10-30T13:45:27.949717+0000","last_active":"2023-10-30T13:59:02.390308+0000","last_peered":"2023-10-30T13:59:02.390308+0000","last_clean":"2023-10-30T13:59:02.390308+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:59:02.390308+0000","last_undegraded":"2023-10-30T13:59:02.390308+0000","last_fullsized":"2023-10-30T13:59:02.390308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:55.918379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064085849999999996,"stat_sum":{"num_bytes":10071990,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":4109,"num_write":659,"num_write_kb":40765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"56'756","reported_seq":826,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.368827+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:58:51.368827+0000","last_peered":"2023-10-30T13:58:51.368827+0000","last_clean":"2023-10-30T13:58:51.368827+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:58:51.368827+0000","last_undegraded":"2023-10-30T13:58:51.368827+0000","last_fullsized":"2023-10-30T13:58:51.368827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":707,"num_write_kb":16664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"56'647","reported_seq":2850,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.626256+0000","last_change":"2023-10-30T13:45:27.934883+0000","last_active":"2023-10-30T13:58:47.626256+0000","last_peered":"2023-10-30T13:58:47.626256+0000","last_clean":"2023-10-30T13:58:47.626256+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:58:47.626256+0000","last_undegraded":"2023-10-30T13:58:47.626256+0000","last_fullsized":"2023-10-30T13:58:47.626256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:16.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001735594,"stat_sum":{"num_bytes":6219710,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":108,"num_read_kb":4114,"num_write":624,"num_write_kb":12697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"56'277","reported_seq":355,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.425915+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:58:56.425915+0000","last_peered":"2023-10-30T13:58:56.425915+0000","last_clean":"2023-10-30T13:58:56.425915+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:58:56.425915+0000","last_undegraded":"2023-10-30T13:58:56.425915+0000","last_fullsized":"2023-10-30T13:58:56.425915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":698,"num_write":513,"num_write_kb":2808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"56'586","reported_seq":2787,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:48.029365+0000","last_change":"2023-10-30T13:45:28.781685+0000","last_active":"2023-10-30T13:58:48.029365+0000","last_peered":"2023-10-30T13:58:48.029365+0000","last_clean":"2023-10-30T13:58:48.029365+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:58:48.029365+0000","last_undegraded":"2023-10-30T13:58:48.029365+0000","last_fullsized":"2023-10-30T13:58:48.029365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:24.835692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002858771,"stat_sum":{"num_bytes":14189536,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":106,"num_read_kb":4110,"num_write":557,"num_write_kb":24121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"56'119","reported_seq":196,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.385695+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:58:51.385695+0000","last_peered":"2023-10-30T13:58:51.385695+0000","last_clean":"2023-10-30T13:58:51.385695+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:58:51.385695+0000","last_undegraded":"2023-10-30T13:58:51.385695+0000","last_fullsized":"2023-10-30T13:58:51.385695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":20522,"num_write":112,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"56'681","reported_seq":2888,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.392109+0000","last_change":"2023-10-30T13:45:27.941313+0000","last_active":"2023-10-30T13:59:02.392109+0000","last_peered":"2023-10-30T13:59:02.392109+0000","last_clean":"2023-10-30T13:59:02.392109+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:59:02.392109+0000","last_undegraded":"2023-10-30T13:59:02.392109+0000","last_fullsized":"2023-10-30T13:59:02.392109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:13.966713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027315170000000001,"stat_sum":{"num_bytes":18657914,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":112,"num_read_kb":8214,"num_write":646,"num_write_kb":32163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"56'144","reported_seq":213,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:41.727964+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:58:41.727964+0000","last_peered":"2023-10-30T13:58:41.727964+0000","last_clean":"2023-10-30T13:58:41.727964+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:58:41.727964+0000","last_undegraded":"2023-10-30T13:58:41.727964+0000","last_fullsized":"2023-10-30T13:58:41.727964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":30625,"num_write":105,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"56'677","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.279925+0000","last_change":"2023-10-30T13:45:27.946293+0000","last_active":"2023-10-30T13:59:02.279925+0000","last_peered":"2023-10-30T13:59:02.279925+0000","last_clean":"2023-10-30T13:59:02.279925+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:59:02.279925+0000","last_undegraded":"2023-10-30T13:59:02.279925+0000","last_fullsized":"2023-10-30T13:59:02.279925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:04.199638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075826330000000001,"stat_sum":{"num_bytes":5755271,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":98,"num_read_kb":7,"num_write":639,"num_write_kb":16186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'941","reported_seq":1007,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382666+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:58:51.382666+0000","last_peered":"2023-10-30T13:58:51.382666+0000","last_clean":"2023-10-30T13:58:51.382666+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:58:51.382666+0000","last_undegraded":"2023-10-30T13:58:51.382666+0000","last_fullsized":"2023-10-30T13:58:51.382666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":20487,"num_write":909,"num_write_kb":8631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"56'696","reported_seq":2903,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.359470+0000","last_change":"2023-10-30T13:45:28.782068+0000","last_active":"2023-10-30T13:58:51.359470+0000","last_peered":"2023-10-30T13:58:51.359470+0000","last_clean":"2023-10-30T13:58:51.359470+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:58:51.359470+0000","last_undegraded":"2023-10-30T13:58:51.359470+0000","last_fullsized":"2023-10-30T13:58:51.359470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:04:56.579303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029726169999999999,"stat_sum":{"num_bytes":19759817,"num_objects":119,"num_object_clones":0,"num_object_copies":238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":112,"num_read_kb":12306,"num_write":660,"num_write_kb":31083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"56'477","reported_seq":549,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:31.537236+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:58:31.537236+0000","last_peered":"2023-10-30T13:58:31.537236+0000","last_clean":"2023-10-30T13:58:31.537236+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:58:31.537236+0000","last_undegraded":"2023-10-30T13:58:31.537236+0000","last_fullsized":"2023-10-30T13:58:31.537236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":53262,"num_write":506,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"56'660","reported_seq":2865,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.640353+0000","last_change":"2023-10-30T13:45:28.781107+0000","last_active":"2023-10-30T13:58:47.640353+0000","last_peered":"2023-10-30T13:58:47.640353+0000","last_clean":"2023-10-30T13:58:47.640353+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:58:47.640353+0000","last_undegraded":"2023-10-30T13:58:47.640353+0000","last_fullsized":"2023-10-30T13:58:47.640353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:36.715896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002600925,"stat_sum":{"num_bytes":9507297,"num_objects":132,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":111,"num_read_kb":4112,"num_write":621,"num_write_kb":24574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"56'1285","reported_seq":1372,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.561985+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:58:32.561985+0000","last_peered":"2023-10-30T13:58:32.561985+0000","last_clean":"2023-10-30T13:58:32.561985+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:58:32.561985+0000","last_undegraded":"2023-10-30T13:58:32.561985+0000","last_fullsized":"2023-10-30T13:58:32.561985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":81960,"num_write":1252,"num_write_kb":21130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"56'708","reported_seq":2911,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.279723+0000","last_change":"2023-10-30T13:45:28.050416+0000","last_active":"2023-10-30T13:59:02.279723+0000","last_peered":"2023-10-30T13:59:02.279723+0000","last_clean":"2023-10-30T13:59:02.279723+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:59:02.279723+0000","last_undegraded":"2023-10-30T13:59:02.279723+0000","last_fullsized":"2023-10-30T13:59:02.279723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:34.848723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051869250000000002,"stat_sum":{"num_bytes":18626842,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":5015,"num_write":667,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"56'654","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.352749+0000","last_change":"2023-10-30T13:45:27.949465+0000","last_active":"2023-10-30T13:59:02.352749+0000","last_peered":"2023-10-30T13:59:02.352749+0000","last_clean":"2023-10-30T13:59:02.352749+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:59:02.352749+0000","last_undegraded":"2023-10-30T13:59:02.352749+0000","last_fullsized":"2023-10-30T13:59:02.352749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:57.261270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061609869999999997,"stat_sum":{"num_bytes":14245271,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":123,"num_read_kb":12313,"num_write":620,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"56'230","reported_seq":302,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.389069+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:58:51.389069+0000","last_peered":"2023-10-30T13:58:51.389069+0000","last_clean":"2023-10-30T13:58:51.389069+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:58:51.389069+0000","last_undegraded":"2023-10-30T13:58:51.389069+0000","last_fullsized":"2023-10-30T13:58:51.389069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":24606,"num_write":205,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"56'700","reported_seq":2905,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.336024+0000","last_change":"2023-10-30T13:45:27.948077+0000","last_active":"2023-10-30T13:59:02.336024+0000","last_peered":"2023-10-30T13:59:02.336024+0000","last_clean":"2023-10-30T13:59:02.336024+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:59:02.336024+0000","last_undegraded":"2023-10-30T13:59:02.336024+0000","last_fullsized":"2023-10-30T13:59:02.336024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:48.941885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061457359999999997,"stat_sum":{"num_bytes":11313833,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":112,"num_read_kb":16,"num_write":670,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"56'950","reported_seq":1044,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.465087+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:58:56.465087+0000","last_peered":"2023-10-30T13:58:56.465087+0000","last_clean":"2023-10-30T13:58:56.465087+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:58:56.465087+0000","last_undegraded":"2023-10-30T13:58:56.465087+0000","last_fullsized":"2023-10-30T13:58:56.465087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":103644,"num_write":908,"num_write_kb":16901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"56'670","reported_seq":2886,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.415474+0000","last_change":"2023-10-30T13:45:27.941557+0000","last_active":"2023-10-30T13:59:02.415474+0000","last_peered":"2023-10-30T13:59:02.415474+0000","last_clean":"2023-10-30T13:59:02.415474+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:59:02.415474+0000","last_undegraded":"2023-10-30T13:59:02.415474+0000","last_fullsized":"2023-10-30T13:59:02.415474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:51.049883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029730300000000002,"stat_sum":{"num_bytes":19886092,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":121,"num_read_kb":4113,"num_write":647,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"56'585","reported_seq":664,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:41.720146+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:58:41.720146+0000","last_peered":"2023-10-30T13:58:41.720146+0000","last_clean":"2023-10-30T13:58:41.720146+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:58:41.720146+0000","last_undegraded":"2023-10-30T13:58:41.720146+0000","last_fullsized":"2023-10-30T13:58:41.720146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":72355,"num_write":540,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"56'629","reported_seq":2822,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.591316+0000","last_change":"2023-10-30T13:45:28.781420+0000","last_active":"2023-10-30T13:58:47.591316+0000","last_peered":"2023-10-30T13:58:47.591316+0000","last_clean":"2023-10-30T13:58:47.591316+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:58:47.591316+0000","last_undegraded":"2023-10-30T13:58:47.591316+0000","last_fullsized":"2023-10-30T13:58:47.591316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:15.218245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026777319999999999,"stat_sum":{"num_bytes":7610379,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":98,"num_read_kb":4107,"num_write":602,"num_write_kb":17238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"56'356","reported_seq":451,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382987+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:58:51.382987+0000","last_peered":"2023-10-30T13:58:51.382987+0000","last_clean":"2023-10-30T13:58:51.382987+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:58:51.382987+0000","last_undegraded":"2023-10-30T13:58:51.382987+0000","last_fullsized":"2023-10-30T13:58:51.382987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4418509,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":57427,"num_write":339,"num_write_kb":8706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":916305331,"num_objects":9467,"num_object_clones":0,"num_object_copies":18934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9467,"num_whiteouts":0,"num_read":7268,"num_read_kb":400185,"num_write":40670,"num_write_kb":1941038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880854528,"data_stored":1831121052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42732,"ondisk_log_size":42732,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":297795987,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1553,"num_read_kb":1442867,"num_write":16395,"num_write_kb":373220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595755008,"data_stored":595591974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17196,"ondisk_log_size":17196,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509734,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447344,"kb_used_data":339816,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93924496,"statfs":{"total":96636764160,"available":96178683904,"internally_reserved":0,"allocated":347971584,"data_stored":341021827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":110098891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85199999999999998,"15min":0.81200000000000006},"min":{"1min":0.433,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.63},{"interface":"front","average":{"1min":1.1830000000000001,"5min":0.96299999999999997,"15min":0.90600000000000003},"min":{"1min":0.53500000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.3499999999999996,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.0900000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.86199999999999999,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.0289999999999999,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.02,"5min":0.95499999999999996,"15min":0.92300000000000004},"min":{"1min":0.57399999999999995,"5min":0.45200000000000001,"15min":0.33400000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.625},"last":0.74099999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0389999999999999,"15min":0.97299999999999998},"min":{"1min":0.626,"5min":0.51300000000000001,"15min":0.32000000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.877},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.91700000000000004,"15min":0.94499999999999995},"min":{"1min":0.63300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":4.1120000000000001},"last":0.998}]},{"osd":3,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.75600000000000001,"5min":0.55300000000000005,"15min":0.45900000000000002},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":4.2069999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":4.4189999999999996,"5min":1.645,"15min":1.204},"min":{"1min":0.83299999999999996,"5min":0.47199999999999998,"15min":0.44400000000000001},"max":{"1min":59.844999999999999,"5min":59.844999999999999,"15min":59.844999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1080000000000001,"15min":1.089},"min":{"1min":0.85899999999999999,"5min":0.623,"15min":0.38100000000000001},"max":{"1min":1.853,"5min":1.853,"15min":3.1059999999999999},"last":1.329},{"interface":"front","average":{"1min":1.335,"5min":1.3220000000000001,"15min":1.2769999999999999},"min":{"1min":1.137,"5min":0.754,"15min":0.42399999999999999},"max":{"1min":1.907,"5min":1.907,"15min":3.198},"last":1.401}]},{"osd":5,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0049999999999999,"15min":1.093},"min":{"1min":0.65600000000000003,"5min":0.53100000000000003,"15min":0.40300000000000002},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":4.2759999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.228,"5min":1.258,"15min":1.238},"min":{"1min":0.80000000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5329999999999999,"5min":1.8380000000000001,"15min":3.4380000000000002},"last":1.252}]},{"osd":6,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92200000000000004,"15min":1.091},"min":{"1min":0.55400000000000005,"5min":0.48999999999999999,"15min":0.48399999999999999},"max":{"1min":1.3109999999999999,"5min":1.429,"15min":3.9630000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.458,"5min":1.131,"15min":1.216},"min":{"1min":0.84399999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":7.7060000000000004,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.95599999999999996}]}]},{"osd":3,"up_from":15,"seq":64424509733,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410640,"kb_used_data":311432,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93961200,"statfs":{"total":96636764160,"available":96216268800,"internally_reserved":0,"allocated":318906368,"data_stored":311949814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":101579208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":9.3320000000000007},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0369999999999999,"15min":0.72399999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.26100000000000001},"max":{"1min":0.71599999999999997,"5min":38.082999999999998,"15min":38.082999999999998},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.70199999999999996,"15min":0.60699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":11.599,"15min":11.599},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.70899999999999996,"15min":0.66100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":15.715999999999999,"15min":15.715999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.72699999999999998,"15min":0.66700000000000004},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.28999999999999998},"max":{"1min":0.83899999999999997,"5min":11.646000000000001,"15min":12.351000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.755,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.83399999999999996,"5min":15.641999999999999,"15min":15.641999999999999},"last":0.502}]},{"osd":4,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.016,"15min":0.92000000000000004},"min":{"1min":0.59999999999999998,"5min":0.55400000000000005,"15min":0.51600000000000001},"max":{"1min":0.98899999999999999,"5min":15.759,"15min":15.759},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.298,"15min":1.0640000000000001},"min":{"1min":0.58499999999999996,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":38.137999999999998,"15min":38.137999999999998},"last":0.88400000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.4139999999999999,"15min":1.097},"min":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.52600000000000002},"max":{"1min":1.036,"5min":40.808,"15min":40.808},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.0309999999999999,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.55800000000000005,"15min":0.49299999999999999},"max":{"1min":1.0660000000000001,"5min":16.498000000000001,"15min":19.835999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.96499999999999997,"15min":0.91800000000000004},"min":{"1min":0.621,"5min":0.56799999999999995,"15min":0.53500000000000003},"max":{"1min":1.2370000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.998,"15min":0.96099999999999997},"min":{"1min":0.68500000000000005,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":1.091,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1020000000000001,"15min":0.95799999999999996},"min":{"1min":0.72499999999999998,"5min":0.60599999999999998,"15min":0.59999999999999998},"max":{"1min":1.2549999999999999,"5min":16.082999999999998,"15min":16.082999999999998},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.88,"5min":1.5609999999999999,"15min":1.111},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.58199999999999996},"max":{"1min":1.228,"5min":52.902999999999999,"15min":52.902999999999999},"last":0.98299999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232516,"kb_used_data":187452,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94139324,"statfs":{"total":96636764160,"available":96398667776,"internally_reserved":0,"allocated":191950848,"data_stored":187647968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.45700000000000002},"min":{"1min":0.32100000000000001,"5min":0.251,"15min":0.246},"max":{"1min":0.83199999999999996,"5min":3.407,"15min":3.8969999999999998},"last":0.495},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.371,"5min":0.35299999999999998,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":5.6980000000000004,"15min":5.6980000000000004},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51000000000000001,"15min":0.60099999999999998},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68100000000000005,"5min":5.5750000000000002,"15min":36.109000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58899999999999997,"5min":5.4569999999999999,"15min":8.1890000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.63500000000000001,"15min":0.68999999999999995},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.29199999999999998},"max":{"1min":0.73999999999999999,"5min":5.532,"15min":35.890999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66800000000000004,"15min":0.69799999999999995},"min":{"1min":0.48199999999999998,"5min":0.42899999999999999,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":5.2830000000000004,"15min":22.846},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84099999999999997},"min":{"1min":0.67000000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0549999999999999,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.496,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":5.6059999999999999,"15min":7.8220000000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.93999999999999995,"5min":5.4260000000000002,"15min":9.9109999999999996},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.86499999999999999},"min":{"1min":0.70799999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":5.4870000000000001,"15min":5.4870000000000001},"last":0.874}]},{"osd":6,"last update":"Mon Oct 30 13:58:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":1.2030000000000001,"15min":0.95999999999999996},"min":{"1min":0.64100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0960000000000001,"15min":0.97599999999999998},"min":{"1min":0.68300000000000005,"5min":0.60099999999999998,"15min":0.56599999999999995},"max":{"1min":15.975,"5min":15.975,"15min":15.975},"last":0.86299999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.75700000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.169,"5min":5.3899999999999997,"15min":5.3899999999999997},"last":1.036},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.91900000000000004,"15min":0.91600000000000004},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53700000000000003},"max":{"1min":1.0800000000000001,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":0.84199999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509732,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365392,"kb_used_data":257928,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94006448,"statfs":{"total":96636764160,"available":96262602752,"internally_reserved":0,"allocated":264118272,"data_stored":259563065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.437},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.245},"max":{"1min":0.67200000000000004,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55100000000000005,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":0.66700000000000004,"5min":2.5920000000000001,"15min":22.149000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.315},"max":{"1min":0.749,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.443}]},{"osd":3,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.75900000000000001},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":2.8130000000000002,"15min":54.530000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.67800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34100000000000003},"max":{"1min":0.83799999999999997,"5min":2.879,"15min":25.457000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78300000000000003,"15min":0.81299999999999994},"min":{"1min":0.53900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94699999999999995,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.79500000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":1.054,"5min":2.101,"15min":2.101},"last":0.70199999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83699999999999997,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.505},"max":{"1min":1.012,"5min":2.7810000000000001,"15min":11.164999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83599999999999997},"min":{"1min":0.624,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":0.94399999999999995,"5min":2.8460000000000001,"15min":2.8820000000000001},"last":0.79000000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.83699999999999997},"min":{"1min":0.57299999999999995,"5min":0.55600000000000005,"15min":0.497},"max":{"1min":0.95399999999999996,"5min":2.903,"15min":5.9180000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.85299999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":1.9930000000000001,"15min":2.2629999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:58:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.002,"15min":0.96899999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":12.221,"5min":12.221,"15min":12.221},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.373,"5min":1.0209999999999999,"15min":0.94499999999999995},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":12.153,"5min":12.153,"15min":12.153},"last":0.77800000000000002}]}]},{"osd":6,"up_from":14,"seq":60129542439,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333848,"kb_used_data":240272,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94037992,"statfs":{"total":96636764160,"available":96294903808,"internally_reserved":0,"allocated":246038528,"data_stored":240883050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":95812039},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.1240000000000001,"15min":1.042},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.41999999999999998},"max":{"1min":1.0389999999999999,"5min":19.741,"15min":19.741},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.131,"15min":0.99099999999999999},"min":{"1min":0.69599999999999995,"5min":0.55800000000000005,"15min":0.34899999999999998},"max":{"1min":1.3600000000000001,"5min":13.863,"15min":13.863},"last":0.98199999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93700000000000006,"15min":0.86799999999999999},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.46100000000000002},"max":{"1min":1.0669999999999999,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.1819999999999999,"15min":1.0289999999999999},"min":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.46800000000000003},"max":{"1min":1.2929999999999999,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96199999999999997,"15min":0.97699999999999998},"min":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.335,"5min":3.5329999999999999,"15min":3.7709999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.095,"15min":1.083},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.4059999999999999,"5min":3.407,"15min":4.6840000000000002},"last":1.1519999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.97599999999999998,"15min":1.014},"min":{"1min":0.67100000000000004,"5min":0.52800000000000002,"15min":0.46700000000000003},"max":{"1min":1.359,"5min":3.7999999999999998,"15min":4.7629999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.171,"15min":1.0840000000000001},"min":{"1min":0.92900000000000005,"5min":0.59299999999999997,"15min":0.54600000000000004},"max":{"1min":1.4179999999999999,"5min":3.968,"15min":3.968},"last":1.0589999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2290000000000001,"15min":1.208},"min":{"1min":0.79600000000000004,"5min":0.55800000000000005,"15min":0.51000000000000001},"max":{"1min":1.4710000000000001,"5min":3.8900000000000001,"15min":16.204000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1519999999999999,"15min":1.2649999999999999},"min":{"1min":0.63700000000000001,"5min":0.622,"15min":0.42699999999999999},"max":{"1min":1.399,"5min":19.398,"15min":43.198},"last":1.246}]},{"osd":5,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.98499999999999999,"15min":1.0489999999999999},"min":{"1min":0.70899999999999996,"5min":0.47699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98399999999999999,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3129999999999999,"15min":1.3009999999999999},"min":{"1min":1.1879999999999999,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":1.522,"5min":3.7269999999999999,"15min":3.802},"last":1.196}]},{"osd":7,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.833,"15min":1.5880000000000001},"min":{"1min":0.83499999999999996,"5min":0.58899999999999997,"15min":0.55400000000000005},"max":{"1min":1.2709999999999999,"5min":70.335999999999999,"15min":73.228999999999999},"last":1.026},{"interface":"front","average":{"1min":1.079,"5min":1.675,"15min":1.637},"min":{"1min":0.82699999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.381,"5min":42.164999999999999,"15min":73.289000000000001},"last":0.76300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542437,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446168,"kb_used_data":352528,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93925672,"statfs":{"total":96636764160,"available":96179888128,"internally_reserved":0,"allocated":360988672,"data_stored":352973919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.89900000000000002,"15min":0.879},"min":{"1min":0.54600000000000004,"5min":0.46999999999999997,"15min":0.40400000000000003},"max":{"1min":3.9159999999999999,"5min":3.9159999999999999,"15min":5.0599999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.83099999999999996,"15min":0.872},"min":{"1min":0.52900000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0569999999999999,"5min":1.5620000000000001,"15min":3.4849999999999999},"last":1.724}]},{"osd":1,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.92600000000000005,"15min":0.871},"min":{"1min":0.55300000000000005,"5min":0.40600000000000003,"15min":0.33600000000000002},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.85099999999999998,"15min":0.90000000000000002},"min":{"1min":0.50700000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95899999999999996,"15min":0.95599999999999996},"min":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.51100000000000001},"max":{"1min":1.462,"5min":1.897,"15min":5.3710000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1279999999999999,"15min":1.056},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.53200000000000003},"max":{"1min":1.369,"5min":2.081,"15min":2.887},"last":0.70099999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.095,"15min":1.0620000000000001},"min":{"1min":0.82999999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":5.476},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.97599999999999998,"15min":1.0349999999999999},"min":{"1min":0.67200000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1779999999999999,"5min":1.52,"15min":3.8500000000000001},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95599999999999996,"15min":0.98699999999999999},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.38400000000000001},"max":{"1min":1.3360000000000001,"5min":1.847,"15min":4.1710000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.66900000000000004,"5min":0.49199999999999999,"15min":0.41799999999999998},"max":{"1min":1.3009999999999999,"5min":1.7270000000000001,"15min":4.2519999999999998},"last":0.752}]},{"osd":6,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1899999999999999,"15min":1.1739999999999999},"min":{"1min":0.747,"5min":0.65000000000000002,"15min":0.621},"max":{"1min":1.546,"5min":2.177,"15min":5.6029999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1459999999999999,"15min":1.1739999999999999},"min":{"1min":0.69499999999999995,"5min":0.56100000000000005,"15min":0.41899999999999998},"max":{"1min":1.403,"5min":1.839,"15min":4.077},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.147,"15min":1.23},"min":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.45100000000000001},"max":{"1min":1.4650000000000001,"5min":1.7949999999999999,"15min":9.9390000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.2430000000000001,"15min":1.2889999999999999},"min":{"1min":0.97399999999999998,"5min":0.78000000000000003,"15min":0.48499999999999999},"max":{"1min":1.415,"5min":1.907,"15min":4.3399999999999999},"last":0.5}]}]},{"osd":4,"up_from":14,"seq":60129542437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433140,"kb_used_data":351980,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93938700,"statfs":{"total":96636764160,"available":96193228800,"internally_reserved":0,"allocated":360427520,"data_stored":352125363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.91100000000000003,"15min":0.873},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.167,"5min":8.7680000000000007,"15min":8.7680000000000007},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.89000000000000001,"15min":0.86099999999999999},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.125,"5min":6.5990000000000002,"15min":6.5990000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82899999999999996,"15min":0.89700000000000002},"min":{"1min":0.51600000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.236,"5min":1.375,"15min":5.8010000000000002},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.85899999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.262,"5min":1.6970000000000001,"15min":3.0499999999999998},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88500000000000001,"15min":0.91600000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.212,"5min":1.331,"15min":3.323},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.98599999999999999,"15min":0.97099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.252,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.1100000000000001,"15min":1.0209999999999999},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.151,"5min":8.0570000000000004,"15min":8.0570000000000004},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.056,"15min":1.0009999999999999},"min":{"1min":0.65900000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.381,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.018,"15min":1.0600000000000001},"min":{"1min":0.82699999999999996,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.325,"5min":4.0030000000000001,"15min":8.4550000000000001},"last":1.149},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.175,"15min":1.171},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.57699999999999996},"max":{"1min":1.3959999999999999,"5min":4.0609999999999999,"15min":8.5399999999999991},"last":0.92300000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1859999999999999,"15min":1.083},"min":{"1min":0.76300000000000001,"5min":0.503,"15min":0.45100000000000001},"max":{"1min":1.4870000000000001,"5min":16.989999999999998,"15min":16.989999999999998},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.173,"15min":1.149},"min":{"1min":0.86699999999999999,"5min":0.41299999999999998,"15min":0.36699999999999999},"max":{"1min":1.599,"5min":4.4530000000000003,"15min":4.4530000000000003},"last":1.1819999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:42 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1719999999999999,"15min":1.163},"min":{"1min":0.84199999999999997,"5min":0.45100000000000001,"15min":0.42199999999999999},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":3.4510000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.298,"15min":1.3049999999999999},"min":{"1min":1.0269999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.5470000000000002},"last":1.2450000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469860,"kb_used_data":376220,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93901980,"statfs":{"total":96636764160,"available":96155627520,"internally_reserved":0,"allocated":385249280,"data_stored":377198604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.98899999999999999,"15min":0.63900000000000001},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":0.63900000000000001,"5min":45.311,"15min":45.311},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.59599999999999997},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.27200000000000002},"max":{"1min":0.72599999999999998,"5min":2.2410000000000001,"15min":27.616},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.29099999999999998},"max":{"1min":0.82999999999999996,"5min":0.88100000000000001,"15min":2.5449999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.44,"5min":0.40899999999999997,"15min":0.35199999999999998},"max":{"1min":0.752,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.57599999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.67500000000000004,"15min":0.56100000000000005},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":0.79300000000000004,"5min":12.368,"15min":12.368},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.75800000000000001,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.40899999999999997},"max":{"1min":0.83299999999999996,"5min":12.442,"15min":12.442},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84799999999999998,"15min":0.85999999999999999},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.41599999999999998},"max":{"1min":1,"5min":2.6949999999999998,"15min":9.4979999999999993},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.871,"15min":0.86499999999999999},"min":{"1min":0.629,"5min":0.60199999999999998,"15min":0.46700000000000003},"max":{"1min":1.1220000000000001,"5min":2.6440000000000001,"15min":9.5429999999999993},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.88,"15min":0.878},"min":{"1min":0.72199999999999998,"5min":0.47099999999999997,"15min":0.317},"max":{"1min":1.087,"5min":1.163,"15min":11.026},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.375,"15min":1.0309999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.46999999999999997},"max":{"1min":1.1759999999999999,"5min":44.225999999999999,"15min":44.225999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88,"15min":0.873},"min":{"1min":0.73899999999999999,"5min":0.63,"15min":0.48799999999999999},"max":{"1min":1.0649999999999999,"5min":2.5169999999999999,"15min":2.9300000000000002},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.435},"max":{"1min":1.133,"5min":1.236,"15min":11.087999999999999},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.91400000000000003,"15min":0.879},"min":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.50900000000000001},"max":{"1min":1.0740000000000001,"5min":5.6849999999999996,"15min":5.6849999999999996},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.93200000000000005,"15min":0.93300000000000005},"min":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.57199999999999995},"max":{"1min":1.0840000000000001,"5min":2.6030000000000002,"15min":9.5890000000000004},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58961920,"data_stored":58944539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50540544,"data_stored":50514908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75808768,"data_stored":75783932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":88170496,"data_stored":88156866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":87515136,"data_stored":87493416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":70914048,"data_stored":70895130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79933440,"data_stored":79916159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325066752,"data_stored":317352305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179568640,"data_stored":175353072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140771328,"data_stored":136810091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242458624,"data_stored":235842913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":273088512,"data_stored":265087328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":275906560,"data_stored":268118820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":177188864,"data_stored":172364192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266805248,"data_stored":260192331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:59:04.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:59:04.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8dad2ea640 1 Processor -- start 2023-10-30T13:59:04.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8dad2ea640 1 -- start start 2023-10-30T13:59:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8dad2ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da80701a0 0x7f8da815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8dad2ea640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f8da806b2f0 0x7f8da806b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8dad2ea640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da80c2aa0 con 0x7f8da81537f0 2023-10-30T13:59:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8dad2ea640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da80c3ab0 con 0x7f8da806b2f0 2023-10-30T13:59:04.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8dad2ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da815a820 con 0x7f8da80701a0 2023-10-30T13:59:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8da6ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f8da81537f0 0x7f8da80ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.31:40768/0 (socket says 172.21.15.31:40768) 2023-10-30T13:59:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8da6ffd640 1 -- 172.21.15.31:0/3210797490 learned_addr learned my addr 172.21.15.31:0/3210797490 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8da77fe640 1 --2- 172.21.15.31:0/3210797490 >> v2:172.21.15.33:3300/0 conn(0x7f8da806b2f0 0x7f8da806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.764+0000 7f8d9ffff640 1 --2- 172.21.15.31:0/3210797490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da80701a0 0x7f8da815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8da67fc640 1 -- 172.21.15.31:0/3210797490 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377994508 0 0) 0x7f8da80c2aa0 con 0x7f8da81537f0 2023-10-30T13:59:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8da67fc640 1 -- 172.21.15.31:0/3210797490 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d94003530 con 0x7f8da81537f0 2023-10-30T13:59:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8d9ffff640 1 -- 172.21.15.31:0/3210797490 >> v2:172.21.15.33:3300/0 conn(0x7f8da806b2f0 msgr2=0x7f8da806b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8d9ffff640 1 --2- 172.21.15.31:0/3210797490 >> v2:172.21.15.33:3300/0 conn(0x7f8da806b2f0 0x7f8da806b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8d9ffff640 1 -- 172.21.15.31:0/3210797490 >> v1:172.21.15.33:6789/0 conn(0x7f8da81537f0 legacy=0x7f8da80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8d9ffff640 1 -- 172.21.15.31:0/3210797490 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da806bca0 con 0x7f8da80701a0 2023-10-30T13:59:04.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8d9ffff640 1 --2- 172.21.15.31:0/3210797490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da80701a0 0x7f8da815a2e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d900097c0 tx=0x7f8d9001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cbd605eb70fde30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:04.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8da67fc640 1 -- 172.21.15.31:0/3210797490 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900a8970 con 0x7f8da80701a0 2023-10-30T13:59:04.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8da67fc640 1 -- 172.21.15.31:0/3210797490 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d900a8ad0 con 0x7f8da80701a0 2023-10-30T13:59:04.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.768+0000 7f8da67fc640 1 -- 172.21.15.31:0/3210797490 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900a8e40 con 0x7f8da80701a0 2023-10-30T13:59:04.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- 172.21.15.31:0/3210797490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da80701a0 msgr2=0x7f8da815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/3210797490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da80701a0 0x7f8da815a2e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d900097c0 tx=0x7f8d9001ec70 comp rx=0 tx=0).stop 2023-10-30T13:59:04.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- 172.21.15.31:0/3210797490 shutdown_connections 2023-10-30T13:59:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/3210797490 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da80701a0 0x7f8da815a2e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/3210797490 >> v2:172.21.15.33:3300/0 conn(0x7f8da806b2f0 0x7f8da806b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- 172.21.15.31:0/3210797490 >> 172.21.15.31:0/3210797490 conn(0x7f8da8071fb0 msgr2=0x7f8da80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- 172.21.15.31:0/3210797490 shutdown_connections 2023-10-30T13:59:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- 172.21.15.31:0/3210797490 wait complete. 2023-10-30T13:59:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 Processor -- start 2023-10-30T13:59:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- start start 2023-10-30T13:59:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da806b2f0 0x7f8da80c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8da80701a0 0x7f8da81567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da81537f0 0x7f8da8156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8155100 con 0x7f8da806b2f0 2023-10-30T13:59:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da815b610 con 0x7f8da80701a0 2023-10-30T13:59:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8dad2ea640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da815b350 con 0x7f8da81537f0 2023-10-30T13:59:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da806b2f0 0x7f8da80c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da806b2f0 0x7f8da80c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35620/0 (socket says 172.21.15.31:35620) 2023-10-30T13:59:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 -- 172.21.15.31:0/1910425938 learned_addr learned my addr 172.21.15.31:0/1910425938 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8d9ffff640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8da80701a0 0x7f8da81567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da77fe640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da81537f0 0x7f8da8156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 -- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da81537f0 msgr2=0x7f8da8156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da81537f0 0x7f8da8156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 -- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8da80701a0 msgr2=0x7f8da81567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8da80701a0 0x7f8da81567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8da6ffd640 1 -- 172.21.15.31:0/1910425938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d90009420 con 0x7f8da806b2f0 2023-10-30T13:59:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.772+0000 7f8d9ffff640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8da80701a0 0x7f8da81567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8da6ffd640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da806b2f0 0x7f8da80c99b0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c450 tx=0x7f8d9800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8d9f7fe640 1 -- 172.21.15.31:0/1910425938 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98005cf0 con 0x7f8da806b2f0 2023-10-30T13:59:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8d9f7fe640 1 -- 172.21.15.31:0/1910425938 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d98005e50 con 0x7f8da806b2f0 2023-10-30T13:59:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8d9f7fe640 1 -- 172.21.15.31:0/1910425938 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98007290 con 0x7f8da806b2f0 2023-10-30T13:59:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da81eea40 con 0x7f8da806b2f0 2023-10-30T13:59:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da81eed00 con 0x7f8da806b2f0 2023-10-30T13:59:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da804e170 con 0x7f8da806b2f0 2023-10-30T13:59:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8d9f7fe640 1 -- 172.21.15.31:0/1910425938 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d980073f0 con 0x7f8da806b2f0 2023-10-30T13:59:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8d9f7fe640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d880664d0 0x7f8d88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8d9f7fe640 1 -- 172.21.15.31:0/1910425938 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f8d98083780 con 0x7f8da806b2f0 2023-10-30T13:59:04.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.776+0000 7f8d9ffff640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d880664d0 0x7f8d88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:04.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.780+0000 7f8d9ffff640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d880664d0 0x7f8d88068990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b2000 tx=0x7f8d900aa070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:04.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.780+0000 7f8d9f7fe640 1 -- 172.21.15.31:0/1910425938 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d98050880 con 0x7f8da806b2f0 2023-10-30T13:59:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.936+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8da80c1eb0 con 0x7f8d880664d0 2023-10-30T13:59:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.956+0000 7f8d9f7fe640 1 -- 172.21.15.31:0/1910425938 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292756 (secure 0 0 0) 0x7f8d900bf090 con 0x7f8d880664d0 2023-10-30T13:59:04.963 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:59:04.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d880664d0 msgr2=0x7f8d88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d880664d0 0x7f8d88068990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b2000 tx=0x7f8d900aa070 comp rx=0 tx=0).stop 2023-10-30T13:59:04.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da806b2f0 msgr2=0x7f8da80c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:04.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da806b2f0 0x7f8da80c99b0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c450 tx=0x7f8d9800c910 comp rx=0 tx=0).stop 2023-10-30T13:59:04.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 shutdown_connections 2023-10-30T13:59:04.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d880664d0 0x7f8d88068990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8da81537f0 0x7f8da8156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8da80701a0 0x7f8da81567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 --2- 172.21.15.31:0/1910425938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da806b2f0 0x7f8da80c99b0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 >> 172.21.15.31:0/1910425938 conn(0x7f8da8071fb0 msgr2=0x7f8da806c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 shutdown_connections 2023-10-30T13:59:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:04.960+0000 7f8dad2ea640 1 -- 172.21.15.31:0/1910425938 wait complete. 2023-10-30T13:59:04.983 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":774,"stamp":"2023-10-30T13:59:03.714457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1214691686,"num_objects":9684,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9684,"num_whiteouts":0,"num_read":8937,"num_read_kb":1843152,"num_write":57224,"num_write_kb":2316674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60045,"ondisk_log_size":60045,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3138908,"kb_used_data":2417628,"kb_used_omap":12,"kb_used_meta":721203,"kb_avail":751835812,"statfs":{"total":773094113280,"available":769879871488,"internally_reserved":0,"allocated":2475651072,"data_stored":2423363610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":738512440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28176770,"num_objects":827,"num_object_clones":0,"num_object_copies":1654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1154,"num_write_kb":28302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004989"},"pg_stats":[{"pgid":"3.3f","version":"56'647","reported_seq":2866,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.625135+0000","last_change":"2023-10-30T13:45:28.601107+0000","last_active":"2023-10-30T13:58:47.625135+0000","last_peered":"2023-10-30T13:58:47.625135+0000","last_clean":"2023-10-30T13:58:47.625135+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:58:47.625135+0000","last_undegraded":"2023-10-30T13:58:47.625135+0000","last_fullsized":"2023-10-30T13:58:47.625135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:58.462635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098613959999999997,"stat_sum":{"num_bytes":30381033,"num_objects":125,"num_object_clones":0,"num_object_copies":250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":124,"num_read_kb":8207,"num_write":615,"num_write_kb":38998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"56'626","reported_seq":2825,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.650883+0000","last_change":"2023-10-30T13:45:27.935552+0000","last_active":"2023-10-30T13:58:47.650883+0000","last_peered":"2023-10-30T13:58:47.650883+0000","last_clean":"2023-10-30T13:58:47.650883+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:58:47.650883+0000","last_undegraded":"2023-10-30T13:58:47.650883+0000","last_fullsized":"2023-10-30T13:58:47.650883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:50.606021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001955118,"stat_sum":{"num_bytes":9798818,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":104,"num_read_kb":11,"num_write":587,"num_write_kb":23396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"56'679","reported_seq":2881,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.382336+0000","last_change":"2023-10-30T13:45:27.939453+0000","last_active":"2023-10-30T13:59:02.382336+0000","last_peered":"2023-10-30T13:59:02.382336+0000","last_clean":"2023-10-30T13:59:02.382336+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:59:02.382336+0000","last_undegraded":"2023-10-30T13:59:02.382336+0000","last_fullsized":"2023-10-30T13:59:02.382336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:59.423451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021120940000000001,"stat_sum":{"num_bytes":23390266,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":110,"num_read_kb":20497,"num_write":638,"num_write_kb":47001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"56'663","reported_seq":2888,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.599578+0000","last_change":"2023-10-30T13:45:28.487484+0000","last_active":"2023-10-30T13:58:47.599578+0000","last_peered":"2023-10-30T13:58:47.599578+0000","last_clean":"2023-10-30T13:58:47.599578+0000","last_became_active":"2023-10-30T13:35:48.779909+0000","last_became_peered":"2023-10-30T13:35:48.779909+0000","last_unstale":"2023-10-30T13:58:47.599578+0000","last_undegraded":"2023-10-30T13:58:47.599578+0000","last_fullsized":"2023-10-30T13:58:47.599578+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:51.343491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001317775,"stat_sum":{"num_bytes":9891362,"num_objects":139,"num_object_clones":0,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":133,"num_read_kb":8205,"num_write":636,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"56'666","reported_seq":2869,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.626617+0000","last_change":"2023-10-30T13:45:28.597729+0000","last_active":"2023-10-30T13:58:47.626617+0000","last_peered":"2023-10-30T13:58:47.626617+0000","last_clean":"2023-10-30T13:58:47.626617+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:58:47.626617+0000","last_undegraded":"2023-10-30T13:58:47.626617+0000","last_fullsized":"2023-10-30T13:58:47.626617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:29.242932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007614844,"stat_sum":{"num_bytes":19656703,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":110,"num_read_kb":8208,"num_write":638,"num_write_kb":39882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"56'661","reported_seq":2858,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.395628+0000","last_change":"2023-10-30T13:45:27.941070+0000","last_active":"2023-10-30T13:59:02.395628+0000","last_peered":"2023-10-30T13:59:02.395628+0000","last_clean":"2023-10-30T13:59:02.395628+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:59:02.395628+0000","last_undegraded":"2023-10-30T13:59:02.395628+0000","last_fullsized":"2023-10-30T13:59:02.395628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:40.443395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030849649999999998,"stat_sum":{"num_bytes":19227960,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":104,"num_read_kb":7536,"num_write":633,"num_write_kb":20026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"56'662","reported_seq":2846,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.643842+0000","last_change":"2023-10-30T13:45:28.597665+0000","last_active":"2023-10-30T13:58:47.643842+0000","last_peered":"2023-10-30T13:58:47.643842+0000","last_clean":"2023-10-30T13:58:47.643842+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:58:47.643842+0000","last_undegraded":"2023-10-30T13:58:47.643842+0000","last_fullsized":"2023-10-30T13:58:47.643842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:44.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082986029999999999,"stat_sum":{"num_bytes":9880254,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":91,"num_read_kb":11,"num_write":635,"num_write_kb":29079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"56'645","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.642291+0000","last_change":"2023-10-30T13:45:28.780454+0000","last_active":"2023-10-30T13:58:47.642291+0000","last_peered":"2023-10-30T13:58:47.642291+0000","last_clean":"2023-10-30T13:58:47.642291+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:58:47.642291+0000","last_undegraded":"2023-10-30T13:58:47.642291+0000","last_fullsized":"2023-10-30T13:58:47.642291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:57.938730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021728070000000001,"stat_sum":{"num_bytes":10096886,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":132,"num_read_kb":1147,"num_write":620,"num_write_kb":24646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"56'631","reported_seq":2847,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.628018+0000","last_change":"2023-10-30T13:45:28.601219+0000","last_active":"2023-10-30T13:58:47.628018+0000","last_peered":"2023-10-30T13:58:47.628018+0000","last_clean":"2023-10-30T13:58:47.628018+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:58:47.628018+0000","last_undegraded":"2023-10-30T13:58:47.628018+0000","last_fullsized":"2023-10-30T13:58:47.628018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:53.268188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010045174,"stat_sum":{"num_bytes":10190840,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":124,"num_read_kb":4115,"num_write":586,"num_write_kb":23657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"56'612","reported_seq":2818,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.344375+0000","last_change":"2023-10-30T13:45:28.050787+0000","last_active":"2023-10-30T13:59:02.344375+0000","last_peered":"2023-10-30T13:59:02.344375+0000","last_clean":"2023-10-30T13:59:02.344375+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:59:02.344375+0000","last_undegraded":"2023-10-30T13:59:02.344375+0000","last_fullsized":"2023-10-30T13:59:02.344375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:02.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047277760000000004,"stat_sum":{"num_bytes":20230336,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":113,"num_read_kb":28,"num_write":585,"num_write_kb":25967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"56'610","reported_seq":2800,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.302667+0000","last_change":"2023-10-30T13:45:28.050006+0000","last_active":"2023-10-30T13:59:02.302667+0000","last_peered":"2023-10-30T13:59:02.302667+0000","last_clean":"2023-10-30T13:59:02.302667+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:59:02.302667+0000","last_undegraded":"2023-10-30T13:59:02.302667+0000","last_fullsized":"2023-10-30T13:59:02.302667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:30.724902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00429868,"stat_sum":{"num_bytes":2384023,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":97,"num_read_kb":17,"num_write":586,"num_write_kb":9055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"56'626","reported_seq":2835,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.294793+0000","last_change":"2023-10-30T13:45:27.946083+0000","last_active":"2023-10-30T13:59:02.294793+0000","last_peered":"2023-10-30T13:59:02.294793+0000","last_clean":"2023-10-30T13:59:02.294793+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:59:02.294793+0000","last_undegraded":"2023-10-30T13:59:02.294793+0000","last_fullsized":"2023-10-30T13:59:02.294793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:19.253547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066533010000000004,"stat_sum":{"num_bytes":18341132,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":8208,"num_write":596,"num_write_kb":32418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"56'610","reported_seq":2816,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.362815+0000","last_change":"2023-10-30T13:45:28.780475+0000","last_active":"2023-10-30T13:58:51.362815+0000","last_peered":"2023-10-30T13:58:51.362815+0000","last_clean":"2023-10-30T13:58:51.362815+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:58:51.362815+0000","last_undegraded":"2023-10-30T13:58:51.362815+0000","last_fullsized":"2023-10-30T13:58:51.362815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:10:02.225437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002338237,"stat_sum":{"num_bytes":13957458,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":114,"num_read_kb":12311,"num_write":572,"num_write_kb":36850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"56'731","reported_seq":2944,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.434802+0000","last_change":"2023-10-30T13:45:27.940425+0000","last_active":"2023-10-30T13:59:02.434802+0000","last_peered":"2023-10-30T13:59:02.434802+0000","last_clean":"2023-10-30T13:59:02.434802+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:59:02.434802+0000","last_undegraded":"2023-10-30T13:59:02.434802+0000","last_fullsized":"2023-10-30T13:59:02.434802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:57.204443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029248669999999998,"stat_sum":{"num_bytes":22804734,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":119,"num_read_kb":24041,"num_write":691,"num_write_kb":57342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"56'690","reported_seq":2896,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.631995+0000","last_change":"2023-10-30T13:45:27.934641+0000","last_active":"2023-10-30T13:58:47.631995+0000","last_peered":"2023-10-30T13:58:47.631995+0000","last_clean":"2023-10-30T13:58:47.631995+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:58:47.631995+0000","last_undegraded":"2023-10-30T13:58:47.631995+0000","last_fullsized":"2023-10-30T13:58:47.631995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:06:31.910235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021050410000000002,"stat_sum":{"num_bytes":18067659,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":112,"num_read_kb":7334,"num_write":654,"num_write_kb":39561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"56'726","reported_seq":2943,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.419953+0000","last_change":"2023-10-30T13:45:27.943054+0000","last_active":"2023-10-30T13:59:02.419953+0000","last_peered":"2023-10-30T13:59:02.419953+0000","last_clean":"2023-10-30T13:59:02.419953+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:59:02.419953+0000","last_undegraded":"2023-10-30T13:59:02.419953+0000","last_fullsized":"2023-10-30T13:59:02.419953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:29.930187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052579080000000004,"stat_sum":{"num_bytes":35259912,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":123,"num_read_kb":12306,"num_write":693,"num_write_kb":43349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"56'699","reported_seq":2895,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.379018+0000","last_change":"2023-10-30T13:45:27.950182+0000","last_active":"2023-10-30T13:59:02.379018+0000","last_peered":"2023-10-30T13:59:02.379018+0000","last_clean":"2023-10-30T13:59:02.379018+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:59:02.379018+0000","last_undegraded":"2023-10-30T13:59:02.379018+0000","last_fullsized":"2023-10-30T13:59:02.379018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:09.257269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052923609999999998,"stat_sum":{"num_bytes":14461920,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":103,"num_read_kb":4113,"num_write":659,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"56'628","reported_seq":2840,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.619440+0000","last_change":"2023-10-30T13:45:28.487735+0000","last_active":"2023-10-30T13:58:47.619440+0000","last_peered":"2023-10-30T13:58:47.619440+0000","last_clean":"2023-10-30T13:58:47.619440+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:58:47.619440+0000","last_undegraded":"2023-10-30T13:58:47.619440+0000","last_fullsized":"2023-10-30T13:58:47.619440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:05.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001369274,"stat_sum":{"num_bytes":15189051,"num_objects":119,"num_object_clones":0,"num_object_copies":238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":118,"num_read_kb":4123,"num_write":600,"num_write_kb":31137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"56'685","reported_seq":2912,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.624637+0000","last_change":"2023-10-30T13:45:28.600018+0000","last_active":"2023-10-30T13:58:47.624637+0000","last_peered":"2023-10-30T13:58:47.624637+0000","last_clean":"2023-10-30T13:58:47.624637+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:58:47.624637+0000","last_undegraded":"2023-10-30T13:58:47.624637+0000","last_fullsized":"2023-10-30T13:58:47.624637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:15.777688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092525119999999992,"stat_sum":{"num_bytes":8625831,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":134,"num_read_kb":8209,"num_write":656,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"56'660","reported_seq":2853,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.287934+0000","last_change":"2023-10-30T13:45:28.051528+0000","last_active":"2023-10-30T13:59:02.287934+0000","last_peered":"2023-10-30T13:59:02.287934+0000","last_clean":"2023-10-30T13:59:02.287934+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:59:02.287934+0000","last_undegraded":"2023-10-30T13:59:02.287934+0000","last_fullsized":"2023-10-30T13:59:02.287934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:24.699943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00502631,"stat_sum":{"num_bytes":6211385,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":100,"num_read_kb":8209,"num_write":620,"num_write_kb":19966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"56'684","reported_seq":2889,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.481517+0000","last_change":"2023-10-30T13:45:27.940012+0000","last_active":"2023-10-30T13:59:02.481517+0000","last_peered":"2023-10-30T13:59:02.481517+0000","last_clean":"2023-10-30T13:59:02.481517+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:59:02.481517+0000","last_undegraded":"2023-10-30T13:59:02.481517+0000","last_fullsized":"2023-10-30T13:59:02.481517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:13.888934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026024469999999999,"stat_sum":{"num_bytes":18097476,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":111,"num_read_kb":21,"num_write":651,"num_write_kb":32402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"56'700","reported_seq":2899,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.370220+0000","last_change":"2023-10-30T13:45:27.947213+0000","last_active":"2023-10-30T13:59:02.370220+0000","last_peered":"2023-10-30T13:59:02.370220+0000","last_clean":"2023-10-30T13:59:02.370220+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:59:02.370220+0000","last_undegraded":"2023-10-30T13:59:02.370220+0000","last_fullsized":"2023-10-30T13:59:02.370220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:24.312446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054353270000000002,"stat_sum":{"num_bytes":10272429,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":105,"num_read_kb":4117,"num_write":665,"num_write_kb":16082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"56'675","reported_seq":2882,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.598533+0000","last_change":"2023-10-30T13:45:28.781012+0000","last_active":"2023-10-30T13:58:47.598533+0000","last_peered":"2023-10-30T13:58:47.598533+0000","last_clean":"2023-10-30T13:58:47.598533+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:58:47.598533+0000","last_undegraded":"2023-10-30T13:58:47.598533+0000","last_fullsized":"2023-10-30T13:58:47.598533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:40:44.651276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025617359999999998,"stat_sum":{"num_bytes":13960173,"num_objects":128,"num_object_clones":0,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":114,"num_read_kb":8213,"num_write":647,"num_write_kb":36096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"56'705","reported_seq":2926,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.388279+0000","last_change":"2023-10-30T13:45:27.942487+0000","last_active":"2023-10-30T13:59:02.388279+0000","last_peered":"2023-10-30T13:59:02.388279+0000","last_clean":"2023-10-30T13:59:02.388279+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:59:02.388279+0000","last_undegraded":"2023-10-30T13:59:02.388279+0000","last_fullsized":"2023-10-30T13:59:02.388279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:39.369789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004034279,"stat_sum":{"num_bytes":1595822,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":126,"num_read_kb":16,"num_write":676,"num_write_kb":12551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"56'645","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.596722+0000","last_change":"2023-10-30T13:45:28.782023+0000","last_active":"2023-10-30T13:58:47.596722+0000","last_peered":"2023-10-30T13:58:47.596722+0000","last_clean":"2023-10-30T13:58:47.596722+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:58:47.596722+0000","last_undegraded":"2023-10-30T13:58:47.596722+0000","last_fullsized":"2023-10-30T13:58:47.596722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:09:53.916658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030593479999999999,"stat_sum":{"num_bytes":7588015,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":117,"num_read_kb":4108,"num_write":613,"num_write_kb":17885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"56'712","reported_seq":2918,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.378475+0000","last_change":"2023-10-30T13:45:28.051170+0000","last_active":"2023-10-30T13:59:02.378475+0000","last_peered":"2023-10-30T13:59:02.378475+0000","last_clean":"2023-10-30T13:59:02.378475+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:59:02.378475+0000","last_undegraded":"2023-10-30T13:59:02.378475+0000","last_fullsized":"2023-10-30T13:59:02.378475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:05.790369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049754609999999996,"stat_sum":{"num_bytes":9799460,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":111,"num_read_kb":24,"num_write":679,"num_write_kb":16350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"56'641","reported_seq":2846,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.609884+0000","last_change":"2023-10-30T13:45:27.935505+0000","last_active":"2023-10-30T13:58:47.609884+0000","last_peered":"2023-10-30T13:58:47.609884+0000","last_clean":"2023-10-30T13:58:47.609884+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:58:47.609884+0000","last_undegraded":"2023-10-30T13:58:47.609884+0000","last_fullsized":"2023-10-30T13:58:47.609884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:28.046138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028468439999999999,"stat_sum":{"num_bytes":33204376,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":110,"num_read_kb":16400,"num_write":604,"num_write_kb":35545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"56'610","reported_seq":2816,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.349462+0000","last_change":"2023-10-30T13:45:28.781733+0000","last_active":"2023-10-30T13:58:51.349462+0000","last_peered":"2023-10-30T13:58:51.349462+0000","last_clean":"2023-10-30T13:58:51.349462+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:58:51.349462+0000","last_undegraded":"2023-10-30T13:58:51.349462+0000","last_fullsized":"2023-10-30T13:58:51.349462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:10.458130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002839121,"stat_sum":{"num_bytes":9452766,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":112,"num_read_kb":10701,"num_write":586,"num_write_kb":14944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"56'679","reported_seq":2883,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.489266+0000","last_change":"2023-10-30T13:45:27.941228+0000","last_active":"2023-10-30T13:59:02.489266+0000","last_peered":"2023-10-30T13:59:02.489266+0000","last_clean":"2023-10-30T13:59:02.489266+0000","last_became_active":"2023-10-30T13:35:48.847825+0000","last_became_peered":"2023-10-30T13:35:48.847825+0000","last_unstale":"2023-10-30T13:59:02.489266+0000","last_undegraded":"2023-10-30T13:59:02.489266+0000","last_fullsized":"2023-10-30T13:59:02.489266+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:46.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028647770000000002,"stat_sum":{"num_bytes":7231756,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":102,"num_read_kb":4848,"num_write":653,"num_write_kb":26441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"56'688","reported_seq":2903,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.382620+0000","last_change":"2023-10-30T13:45:27.948917+0000","last_active":"2023-10-30T13:59:02.382620+0000","last_peered":"2023-10-30T13:59:02.382620+0000","last_clean":"2023-10-30T13:59:02.382620+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:59:02.382620+0000","last_undegraded":"2023-10-30T13:59:02.382620+0000","last_fullsized":"2023-10-30T13:59:02.382620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:51.840613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060200970000000003,"stat_sum":{"num_bytes":22333767,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":120,"num_read_kb":4116,"num_write":656,"num_write_kb":43887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"56'753","reported_seq":2987,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.633520+0000","last_change":"2023-10-30T13:45:28.487621+0000","last_active":"2023-10-30T13:58:47.633520+0000","last_peered":"2023-10-30T13:58:47.633520+0000","last_clean":"2023-10-30T13:58:47.633520+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:58:47.633520+0000","last_undegraded":"2023-10-30T13:58:47.633520+0000","last_fullsized":"2023-10-30T13:58:47.633520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:58:52.921718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001263171,"stat_sum":{"num_bytes":10399401,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":139,"num_read_kb":4118,"num_write":726,"num_write_kb":34561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"56'704","reported_seq":2909,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.381095+0000","last_change":"2023-10-30T13:45:28.048779+0000","last_active":"2023-10-30T13:59:02.381095+0000","last_peered":"2023-10-30T13:59:02.381095+0000","last_clean":"2023-10-30T13:59:02.381095+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:59:02.381095+0000","last_undegraded":"2023-10-30T13:59:02.381095+0000","last_fullsized":"2023-10-30T13:59:02.381095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:18.375625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036881090000000002,"stat_sum":{"num_bytes":14815628,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":110,"num_read_kb":20,"num_write":677,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"56'677","reported_seq":2878,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.248436+0000","last_change":"2023-10-30T13:45:27.944400+0000","last_active":"2023-10-30T13:59:02.248436+0000","last_peered":"2023-10-30T13:59:02.248436+0000","last_clean":"2023-10-30T13:59:02.248436+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:59:02.248436+0000","last_undegraded":"2023-10-30T13:59:02.248436+0000","last_fullsized":"2023-10-30T13:59:02.248436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:12.964999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065387930000000002,"stat_sum":{"num_bytes":14536361,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":108,"num_read_kb":8117,"num_write":641,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"56'695","reported_seq":764,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382522+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:58:51.382522+0000","last_peered":"2023-10-30T13:58:51.382522+0000","last_clean":"2023-10-30T13:58:51.382522+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:58:51.382522+0000","last_undegraded":"2023-10-30T13:58:51.382522+0000","last_fullsized":"2023-10-30T13:58:51.382522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45057,"num_write":652,"num_write_kb":16749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"56'665","reported_seq":2885,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.488063+0000","last_change":"2023-10-30T13:45:27.941002+0000","last_active":"2023-10-30T13:59:02.488063+0000","last_peered":"2023-10-30T13:59:02.488063+0000","last_clean":"2023-10-30T13:59:02.488063+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:59:02.488063+0000","last_undegraded":"2023-10-30T13:59:02.488063+0000","last_fullsized":"2023-10-30T13:59:02.488063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:33.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031306289999999998,"stat_sum":{"num_bytes":22886210,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":127,"num_read_kb":6230,"num_write":644,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"56'743","reported_seq":874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.377388+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:58:51.377388+0000","last_peered":"2023-10-30T13:58:51.377388+0000","last_clean":"2023-10-30T13:58:51.377388+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:58:51.377388+0000","last_undegraded":"2023-10-30T13:58:51.377388+0000","last_fullsized":"2023-10-30T13:58:51.377388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21047876,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":114393,"num_write":693,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"56'141","reported_seq":249,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.571367+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:58:52.571367+0000","last_peered":"2023-10-30T13:58:52.571367+0000","last_clean":"2023-10-30T13:58:52.571367+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:58:52.571367+0000","last_undegraded":"2023-10-30T13:58:52.571367+0000","last_fullsized":"2023-10-30T13:58:52.571367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297526,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":20619,"num_write":102,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"56'752","reported_seq":2988,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.668049+0000","last_change":"2023-10-30T13:45:28.781311+0000","last_active":"2023-10-30T13:58:47.668049+0000","last_peered":"2023-10-30T13:58:47.668049+0000","last_clean":"2023-10-30T13:58:47.668049+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:58:47.668049+0000","last_undegraded":"2023-10-30T13:58:47.668049+0000","last_fullsized":"2023-10-30T13:58:47.668049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:42.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027758349999999999,"stat_sum":{"num_bytes":1481853,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":144,"num_read_kb":4120,"num_write":721,"num_write_kb":40298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"56'594","reported_seq":676,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:31.559442+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:58:31.559442+0000","last_peered":"2023-10-30T13:58:31.559442+0000","last_clean":"2023-10-30T13:58:31.559442+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:58:31.559442+0000","last_undegraded":"2023-10-30T13:58:31.559442+0000","last_fullsized":"2023-10-30T13:58:31.559442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":96432,"num_write":540,"num_write_kb":20769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"56'616","reported_seq":2823,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.628582+0000","last_change":"2023-10-30T13:45:27.934297+0000","last_active":"2023-10-30T13:58:47.628582+0000","last_peered":"2023-10-30T13:58:47.628582+0000","last_clean":"2023-10-30T13:58:47.628582+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:58:47.628582+0000","last_undegraded":"2023-10-30T13:58:47.628582+0000","last_fullsized":"2023-10-30T13:58:47.628582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:48.732244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017556029999999999,"stat_sum":{"num_bytes":5986884,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":115,"num_read_kb":8211,"num_write":580,"num_write_kb":32401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"56'127","reported_seq":186,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382513+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:58:51.382513+0000","last_peered":"2023-10-30T13:58:51.382513+0000","last_clean":"2023-10-30T13:58:51.382513+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:58:51.382513+0000","last_undegraded":"2023-10-30T13:58:51.382513+0000","last_fullsized":"2023-10-30T13:58:51.382513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":91,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"56'666","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.619983+0000","last_change":"2023-10-30T13:45:28.487297+0000","last_active":"2023-10-30T13:58:47.619983+0000","last_peered":"2023-10-30T13:58:47.619983+0000","last_clean":"2023-10-30T13:58:47.619983+0000","last_became_active":"2023-10-30T13:35:49.007835+0000","last_became_peered":"2023-10-30T13:35:49.007835+0000","last_unstale":"2023-10-30T13:58:47.619983+0000","last_undegraded":"2023-10-30T13:58:47.619983+0000","last_fullsized":"2023-10-30T13:58:47.619983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:17:24.830286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011632000000000001,"stat_sum":{"num_bytes":15463549,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":101,"num_read_kb":5707,"num_write":628,"num_write_kb":26855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"56'384","reported_seq":457,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.335510+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:59:02.335510+0000","last_peered":"2023-10-30T13:59:02.335510+0000","last_clean":"2023-10-30T13:59:02.335510+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:59:02.335510+0000","last_undegraded":"2023-10-30T13:59:02.335510+0000","last_fullsized":"2023-10-30T13:59:02.335510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20383142,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":61443,"num_write":344,"num_write_kb":20149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"56'654","reported_seq":2875,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.418220+0000","last_change":"2023-10-30T13:45:27.941812+0000","last_active":"2023-10-30T13:59:02.418220+0000","last_peered":"2023-10-30T13:59:02.418220+0000","last_clean":"2023-10-30T13:59:02.418220+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:59:02.418220+0000","last_undegraded":"2023-10-30T13:59:02.418220+0000","last_fullsized":"2023-10-30T13:59:02.418220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:17.717541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028607060000000002,"stat_sum":{"num_bytes":18966683,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":128,"num_read_kb":11963,"num_write":623,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"56'713","reported_seq":798,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.576749+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:58:52.576749+0000","last_peered":"2023-10-30T13:58:52.576749+0000","last_clean":"2023-10-30T13:58:52.576749+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:58:52.576749+0000","last_undegraded":"2023-10-30T13:58:52.576749+0000","last_fullsized":"2023-10-30T13:58:52.576749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":49185,"num_write":677,"num_write_kb":16860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"56'662","reported_seq":2855,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.355115+0000","last_change":"2023-10-30T13:45:27.947116+0000","last_active":"2023-10-30T13:59:02.355115+0000","last_peered":"2023-10-30T13:59:02.355115+0000","last_clean":"2023-10-30T13:59:02.355115+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:59:02.355115+0000","last_undegraded":"2023-10-30T13:59:02.355115+0000","last_fullsized":"2023-10-30T13:59:02.355115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:44.472162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070741379999999998,"stat_sum":{"num_bytes":9996128,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":99,"num_read_kb":8209,"num_write":632,"num_write_kb":23925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"56'1127","reported_seq":1231,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.351736+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:58:56.351736+0000","last_peered":"2023-10-30T13:58:56.351736+0000","last_clean":"2023-10-30T13:58:56.351736+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:58:56.351736+0000","last_undegraded":"2023-10-30T13:58:56.351736+0000","last_fullsized":"2023-10-30T13:58:56.351736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1127,"log_dups_size":0,"ondisk_log_size":1127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":65,"num_read_kb":86059,"num_write":1076,"num_write_kb":17029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"56'645","reported_seq":2842,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.611995+0000","last_change":"2023-10-30T13:45:28.596691+0000","last_active":"2023-10-30T13:58:47.611995+0000","last_peered":"2023-10-30T13:58:47.611995+0000","last_clean":"2023-10-30T13:58:47.611995+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:58:47.611995+0000","last_undegraded":"2023-10-30T13:58:47.611995+0000","last_fullsized":"2023-10-30T13:58:47.611995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:49:41.351825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070810969999999997,"stat_sum":{"num_bytes":18060472,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":104,"num_read_kb":12299,"num_write":619,"num_write_kb":31647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"56'430","reported_seq":527,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:46.388525+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:58:46.388525+0000","last_peered":"2023-10-30T13:58:46.388525+0000","last_clean":"2023-10-30T13:58:46.388525+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:58:46.388525+0000","last_undegraded":"2023-10-30T13:58:46.388525+0000","last_fullsized":"2023-10-30T13:58:46.388525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":71,"num_read_kb":77906,"num_write":383,"num_write_kb":12555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"56'673","reported_seq":2900,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.616338+0000","last_change":"2023-10-30T13:45:28.487276+0000","last_active":"2023-10-30T13:58:47.616338+0000","last_peered":"2023-10-30T13:58:47.616338+0000","last_clean":"2023-10-30T13:58:47.616338+0000","last_became_active":"2023-10-30T13:35:49.010460+0000","last_became_peered":"2023-10-30T13:35:49.010460+0000","last_unstale":"2023-10-30T13:58:47.616338+0000","last_undegraded":"2023-10-30T13:58:47.616338+0000","last_fullsized":"2023-10-30T13:58:47.616338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:47:08.712606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001187553,"stat_sum":{"num_bytes":13551160,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":138,"num_read_kb":8210,"num_write":640,"num_write_kb":55768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"56'177","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.380435+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:58:51.380435+0000","last_peered":"2023-10-30T13:58:51.380435+0000","last_clean":"2023-10-30T13:58:51.380435+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:58:51.380435+0000","last_undegraded":"2023-10-30T13:58:51.380435+0000","last_fullsized":"2023-10-30T13:58:51.380435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":632,"num_write":147,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"56'672","reported_seq":2891,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.617238+0000","last_change":"2023-10-30T13:45:28.599507+0000","last_active":"2023-10-30T13:58:47.617238+0000","last_peered":"2023-10-30T13:58:47.617238+0000","last_clean":"2023-10-30T13:58:47.617238+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:58:47.617238+0000","last_undegraded":"2023-10-30T13:58:47.617238+0000","last_fullsized":"2023-10-30T13:58:47.617238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:11.218437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092364379999999996,"stat_sum":{"num_bytes":15694014,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":126,"num_read_kb":8225,"num_write":638,"num_write_kb":34297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"56'171","reported_seq":234,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.363936+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:58:51.363936+0000","last_peered":"2023-10-30T13:58:51.363936+0000","last_clean":"2023-10-30T13:58:51.363936+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:58:51.363936+0000","last_undegraded":"2023-10-30T13:58:51.363936+0000","last_fullsized":"2023-10-30T13:58:51.363936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4103,"num_write":119,"num_write_kb":4220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"56'655","reported_seq":2848,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.372721+0000","last_change":"2023-10-30T13:45:28.049648+0000","last_active":"2023-10-30T13:59:02.372721+0000","last_peered":"2023-10-30T13:59:02.372721+0000","last_clean":"2023-10-30T13:59:02.372721+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:59:02.372721+0000","last_undegraded":"2023-10-30T13:59:02.372721+0000","last_fullsized":"2023-10-30T13:59:02.372721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:03.640232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045350950000000003,"stat_sum":{"num_bytes":14031523,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":99,"num_read_kb":8208,"num_write":613,"num_write_kb":14857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"56'613","reported_seq":722,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.572185+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:58:51.572185+0000","last_peered":"2023-10-30T13:58:51.572185+0000","last_clean":"2023-10-30T13:58:51.572185+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:58:51.572185+0000","last_undegraded":"2023-10-30T13:58:51.572185+0000","last_fullsized":"2023-10-30T13:58:51.572185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":130,"num_read_kb":41105,"num_write":605,"num_write_kb":12685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"56'710","reported_seq":2920,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.607871+0000","last_change":"2023-10-30T13:45:28.598560+0000","last_active":"2023-10-30T13:58:47.607871+0000","last_peered":"2023-10-30T13:58:47.607871+0000","last_clean":"2023-10-30T13:58:47.607871+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:58:47.607871+0000","last_undegraded":"2023-10-30T13:58:47.607871+0000","last_fullsized":"2023-10-30T13:58:47.607871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:25.254829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089036940000000002,"stat_sum":{"num_bytes":5954339,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":119,"num_read_kb":8210,"num_write":672,"num_write_kb":34961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'351","reported_seq":425,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.537629+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:58:32.537629+0000","last_peered":"2023-10-30T13:58:32.537629+0000","last_clean":"2023-10-30T13:58:32.537629+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:58:32.537629+0000","last_undegraded":"2023-10-30T13:58:32.537629+0000","last_fullsized":"2023-10-30T13:58:32.537629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":28724,"num_write":357,"num_write_kb":8477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"56'621","reported_seq":2813,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.600642+0000","last_change":"2023-10-30T13:45:28.600359+0000","last_active":"2023-10-30T13:58:47.600642+0000","last_peered":"2023-10-30T13:58:47.600642+0000","last_clean":"2023-10-30T13:58:47.600642+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:58:47.600642+0000","last_undegraded":"2023-10-30T13:58:47.600642+0000","last_fullsized":"2023-10-30T13:58:47.600642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:40.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094781629999999995,"stat_sum":{"num_bytes":5933515,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":99,"num_read_kb":8205,"num_write":588,"num_write_kb":15719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"56'638","reported_seq":715,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:49.590466+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:58:49.590466+0000","last_peered":"2023-10-30T13:58:49.590466+0000","last_clean":"2023-10-30T13:58:49.590466+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:58:49.590466+0000","last_undegraded":"2023-10-30T13:58:49.590466+0000","last_fullsized":"2023-10-30T13:58:49.590466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33457,"num_write":581,"num_write_kb":12711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"56'639","reported_seq":2843,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.631413+0000","last_change":"2023-10-30T13:45:28.601986+0000","last_active":"2023-10-30T13:58:47.631413+0000","last_peered":"2023-10-30T13:58:47.631413+0000","last_clean":"2023-10-30T13:58:47.631413+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:58:47.631413+0000","last_undegraded":"2023-10-30T13:58:47.631413+0000","last_fullsized":"2023-10-30T13:58:47.631413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:59.052456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010320058,"stat_sum":{"num_bytes":13685218,"num_objects":132,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":111,"num_read_kb":8210,"num_write":603,"num_write_kb":35843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"56'400","reported_seq":474,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.554499+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:58:32.554499+0000","last_peered":"2023-10-30T13:58:32.554499+0000","last_clean":"2023-10-30T13:58:32.554499+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:58:32.554499+0000","last_undegraded":"2023-10-30T13:58:32.554499+0000","last_fullsized":"2023-10-30T13:58:32.554499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582982,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":36872,"num_write":369,"num_write_kb":12516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"56'744","reported_seq":2973,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.369177+0000","last_change":"2023-10-30T13:45:27.948762+0000","last_active":"2023-10-30T13:59:02.369177+0000","last_peered":"2023-10-30T13:59:02.369177+0000","last_clean":"2023-10-30T13:59:02.369177+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:59:02.369177+0000","last_undegraded":"2023-10-30T13:59:02.369177+0000","last_fullsized":"2023-10-30T13:59:02.369177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.722813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066640720000000001,"stat_sum":{"num_bytes":18395347,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":135,"num_read_kb":4114,"num_write":711,"num_write_kb":24326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"56'222","reported_seq":290,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.553986+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:58:32.553986+0000","last_peered":"2023-10-30T13:58:32.553986+0000","last_clean":"2023-10-30T13:58:32.553986+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:58:32.553986+0000","last_undegraded":"2023-10-30T13:58:32.553986+0000","last_fullsized":"2023-10-30T13:58:32.553986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":20494,"num_write":198,"num_write_kb":4363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"56'645","reported_seq":2844,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.612268+0000","last_change":"2023-10-30T13:45:28.598725+0000","last_active":"2023-10-30T13:58:47.612268+0000","last_peered":"2023-10-30T13:58:47.612268+0000","last_clean":"2023-10-30T13:58:47.612268+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:58:47.612268+0000","last_undegraded":"2023-10-30T13:58:47.612268+0000","last_fullsized":"2023-10-30T13:58:47.612268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:29.217863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084474579999999997,"stat_sum":{"num_bytes":17958459,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":106,"num_read_kb":8207,"num_write":607,"num_write_kb":40298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:57:42.172646+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:57:42.172646+0000","last_peered":"2023-10-30T13:57:42.172646+0000","last_clean":"2023-10-30T13:57:42.172646+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:57:42.172646+0000","last_undegraded":"2023-10-30T13:57:42.172646+0000","last_fullsized":"2023-10-30T13:57:42.172646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'346","reported_seq":413,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.460048+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:58:56.460048+0000","last_peered":"2023-10-30T13:58:56.460048+0000","last_clean":"2023-10-30T13:58:56.460048+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:58:56.460048+0000","last_undegraded":"2023-10-30T13:58:56.460048+0000","last_fullsized":"2023-10-30T13:58:56.460048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388674,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":20491,"num_write":340,"num_write_kb":8443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'643","reported_seq":2845,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.646867+0000","last_change":"2023-10-30T13:45:28.780750+0000","last_active":"2023-10-30T13:58:47.646867+0000","last_peered":"2023-10-30T13:58:47.646867+0000","last_clean":"2023-10-30T13:58:47.646867+0000","last_became_active":"2023-10-30T13:35:48.847905+0000","last_became_peered":"2023-10-30T13:35:48.847905+0000","last_unstale":"2023-10-30T13:58:47.646867+0000","last_undegraded":"2023-10-30T13:58:47.646867+0000","last_fullsized":"2023-10-30T13:58:47.646867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:37.243152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002389721,"stat_sum":{"num_bytes":10609565,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":100,"num_read_kb":11,"num_write":618,"num_write_kb":20098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"56'352","reported_seq":439,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.575618+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:58:51.575618+0000","last_peered":"2023-10-30T13:58:51.575618+0000","last_clean":"2023-10-30T13:58:51.575618+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:58:51.575618+0000","last_undegraded":"2023-10-30T13:58:51.575618+0000","last_fullsized":"2023-10-30T13:58:51.575618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":102412,"num_write":306,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"56'696","reported_seq":2908,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.336775+0000","last_change":"2023-10-30T13:45:27.939160+0000","last_active":"2023-10-30T13:59:02.336775+0000","last_peered":"2023-10-30T13:59:02.336775+0000","last_clean":"2023-10-30T13:59:02.336775+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:59:02.336775+0000","last_undegraded":"2023-10-30T13:59:02.336775+0000","last_fullsized":"2023-10-30T13:59:02.336775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:16.251173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018610090000000001,"stat_sum":{"num_bytes":24271212,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":117,"num_read_kb":10,"num_write":671,"num_write_kb":33782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"56'1498","reported_seq":2022,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.567569+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:58:52.567569+0000","last_peered":"2023-10-30T13:58:52.567569+0000","last_clean":"2023-10-30T13:58:52.567569+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:58:52.567569+0000","last_undegraded":"2023-10-30T13:58:52.567569+0000","last_fullsized":"2023-10-30T13:58:52.567569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959716,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":475,"num_read_kb":12479,"num_write":1460,"num_write_kb":17319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'629","reported_seq":2840,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.636121+0000","last_change":"2023-10-30T13:45:27.934841+0000","last_active":"2023-10-30T13:58:47.636121+0000","last_peered":"2023-10-30T13:58:47.636121+0000","last_clean":"2023-10-30T13:58:47.636121+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:58:47.636121+0000","last_undegraded":"2023-10-30T13:58:47.636121+0000","last_fullsized":"2023-10-30T13:58:47.636121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:35.639553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022228600000000001,"stat_sum":{"num_bytes":9645659,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":116,"num_read_kb":4117,"num_write":587,"num_write_kb":24506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'628","reported_seq":701,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.549494+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:58:32.549494+0000","last_peered":"2023-10-30T13:58:32.549494+0000","last_clean":"2023-10-30T13:58:32.549494+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:58:32.549494+0000","last_undegraded":"2023-10-30T13:58:32.549494+0000","last_fullsized":"2023-10-30T13:58:32.549494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":61440,"num_write":565,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'712","reported_seq":2908,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.318656+0000","last_change":"2023-10-30T13:45:27.945262+0000","last_active":"2023-10-30T13:59:02.318656+0000","last_peered":"2023-10-30T13:59:02.318656+0000","last_clean":"2023-10-30T13:59:02.318656+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:59:02.318656+0000","last_undegraded":"2023-10-30T13:59:02.318656+0000","last_fullsized":"2023-10-30T13:59:02.318656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:31.086225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066434720000000001,"stat_sum":{"num_bytes":26554596,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":101,"num_read_kb":3528,"num_write":689,"num_write_kb":37408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"56'726","reported_seq":791,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.543895+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:58:32.543895+0000","last_peered":"2023-10-30T13:58:32.543895+0000","last_clean":"2023-10-30T13:58:32.543895+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:58:32.543895+0000","last_undegraded":"2023-10-30T13:58:32.543895+0000","last_fullsized":"2023-10-30T13:58:32.543895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":18914,"num_write":694,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"56'695","reported_seq":2898,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.390308+0000","last_change":"2023-10-30T13:45:27.949717+0000","last_active":"2023-10-30T13:59:02.390308+0000","last_peered":"2023-10-30T13:59:02.390308+0000","last_clean":"2023-10-30T13:59:02.390308+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:59:02.390308+0000","last_undegraded":"2023-10-30T13:59:02.390308+0000","last_fullsized":"2023-10-30T13:59:02.390308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:55.918379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064085849999999996,"stat_sum":{"num_bytes":10071990,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":4109,"num_write":659,"num_write_kb":40765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"56'756","reported_seq":826,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.368827+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:58:51.368827+0000","last_peered":"2023-10-30T13:58:51.368827+0000","last_clean":"2023-10-30T13:58:51.368827+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:58:51.368827+0000","last_undegraded":"2023-10-30T13:58:51.368827+0000","last_fullsized":"2023-10-30T13:58:51.368827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":707,"num_write_kb":16664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"56'647","reported_seq":2850,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.626256+0000","last_change":"2023-10-30T13:45:27.934883+0000","last_active":"2023-10-30T13:58:47.626256+0000","last_peered":"2023-10-30T13:58:47.626256+0000","last_clean":"2023-10-30T13:58:47.626256+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:58:47.626256+0000","last_undegraded":"2023-10-30T13:58:47.626256+0000","last_fullsized":"2023-10-30T13:58:47.626256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:16.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001735594,"stat_sum":{"num_bytes":6219710,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":108,"num_read_kb":4114,"num_write":624,"num_write_kb":12697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"56'277","reported_seq":355,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.425915+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:58:56.425915+0000","last_peered":"2023-10-30T13:58:56.425915+0000","last_clean":"2023-10-30T13:58:56.425915+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:58:56.425915+0000","last_undegraded":"2023-10-30T13:58:56.425915+0000","last_fullsized":"2023-10-30T13:58:56.425915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":698,"num_write":513,"num_write_kb":2808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"56'586","reported_seq":2787,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:48.029365+0000","last_change":"2023-10-30T13:45:28.781685+0000","last_active":"2023-10-30T13:58:48.029365+0000","last_peered":"2023-10-30T13:58:48.029365+0000","last_clean":"2023-10-30T13:58:48.029365+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:58:48.029365+0000","last_undegraded":"2023-10-30T13:58:48.029365+0000","last_fullsized":"2023-10-30T13:58:48.029365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:24.835692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002858771,"stat_sum":{"num_bytes":14189536,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":106,"num_read_kb":4110,"num_write":557,"num_write_kb":24121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"56'119","reported_seq":196,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.385695+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:58:51.385695+0000","last_peered":"2023-10-30T13:58:51.385695+0000","last_clean":"2023-10-30T13:58:51.385695+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:58:51.385695+0000","last_undegraded":"2023-10-30T13:58:51.385695+0000","last_fullsized":"2023-10-30T13:58:51.385695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":20522,"num_write":112,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"56'681","reported_seq":2888,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.392109+0000","last_change":"2023-10-30T13:45:27.941313+0000","last_active":"2023-10-30T13:59:02.392109+0000","last_peered":"2023-10-30T13:59:02.392109+0000","last_clean":"2023-10-30T13:59:02.392109+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:59:02.392109+0000","last_undegraded":"2023-10-30T13:59:02.392109+0000","last_fullsized":"2023-10-30T13:59:02.392109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:13.966713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027315170000000001,"stat_sum":{"num_bytes":18657914,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":112,"num_read_kb":8214,"num_write":646,"num_write_kb":32163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"56'144","reported_seq":213,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:41.727964+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:58:41.727964+0000","last_peered":"2023-10-30T13:58:41.727964+0000","last_clean":"2023-10-30T13:58:41.727964+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:58:41.727964+0000","last_undegraded":"2023-10-30T13:58:41.727964+0000","last_fullsized":"2023-10-30T13:58:41.727964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":30625,"num_write":105,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"56'677","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.279925+0000","last_change":"2023-10-30T13:45:27.946293+0000","last_active":"2023-10-30T13:59:02.279925+0000","last_peered":"2023-10-30T13:59:02.279925+0000","last_clean":"2023-10-30T13:59:02.279925+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:59:02.279925+0000","last_undegraded":"2023-10-30T13:59:02.279925+0000","last_fullsized":"2023-10-30T13:59:02.279925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:04.199638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075826330000000001,"stat_sum":{"num_bytes":5755271,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":98,"num_read_kb":7,"num_write":639,"num_write_kb":16186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'941","reported_seq":1007,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382666+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:58:51.382666+0000","last_peered":"2023-10-30T13:58:51.382666+0000","last_clean":"2023-10-30T13:58:51.382666+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:58:51.382666+0000","last_undegraded":"2023-10-30T13:58:51.382666+0000","last_fullsized":"2023-10-30T13:58:51.382666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":20487,"num_write":909,"num_write_kb":8631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"56'696","reported_seq":2903,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.359470+0000","last_change":"2023-10-30T13:45:28.782068+0000","last_active":"2023-10-30T13:58:51.359470+0000","last_peered":"2023-10-30T13:58:51.359470+0000","last_clean":"2023-10-30T13:58:51.359470+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:58:51.359470+0000","last_undegraded":"2023-10-30T13:58:51.359470+0000","last_fullsized":"2023-10-30T13:58:51.359470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:04:56.579303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029726169999999999,"stat_sum":{"num_bytes":19759817,"num_objects":119,"num_object_clones":0,"num_object_copies":238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":112,"num_read_kb":12306,"num_write":660,"num_write_kb":31083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"56'477","reported_seq":549,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:31.537236+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:58:31.537236+0000","last_peered":"2023-10-30T13:58:31.537236+0000","last_clean":"2023-10-30T13:58:31.537236+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:58:31.537236+0000","last_undegraded":"2023-10-30T13:58:31.537236+0000","last_fullsized":"2023-10-30T13:58:31.537236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":53262,"num_write":506,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"56'660","reported_seq":2865,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.640353+0000","last_change":"2023-10-30T13:45:28.781107+0000","last_active":"2023-10-30T13:58:47.640353+0000","last_peered":"2023-10-30T13:58:47.640353+0000","last_clean":"2023-10-30T13:58:47.640353+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:58:47.640353+0000","last_undegraded":"2023-10-30T13:58:47.640353+0000","last_fullsized":"2023-10-30T13:58:47.640353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:36.715896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002600925,"stat_sum":{"num_bytes":9507297,"num_objects":132,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":111,"num_read_kb":4112,"num_write":621,"num_write_kb":24574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"56'1285","reported_seq":1372,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.561985+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:58:32.561985+0000","last_peered":"2023-10-30T13:58:32.561985+0000","last_clean":"2023-10-30T13:58:32.561985+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:58:32.561985+0000","last_undegraded":"2023-10-30T13:58:32.561985+0000","last_fullsized":"2023-10-30T13:58:32.561985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":81960,"num_write":1252,"num_write_kb":21130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"56'708","reported_seq":2911,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.279723+0000","last_change":"2023-10-30T13:45:28.050416+0000","last_active":"2023-10-30T13:59:02.279723+0000","last_peered":"2023-10-30T13:59:02.279723+0000","last_clean":"2023-10-30T13:59:02.279723+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:59:02.279723+0000","last_undegraded":"2023-10-30T13:59:02.279723+0000","last_fullsized":"2023-10-30T13:59:02.279723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:34.848723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051869250000000002,"stat_sum":{"num_bytes":18626842,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":5015,"num_write":667,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"56'654","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.352749+0000","last_change":"2023-10-30T13:45:27.949465+0000","last_active":"2023-10-30T13:59:02.352749+0000","last_peered":"2023-10-30T13:59:02.352749+0000","last_clean":"2023-10-30T13:59:02.352749+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:59:02.352749+0000","last_undegraded":"2023-10-30T13:59:02.352749+0000","last_fullsized":"2023-10-30T13:59:02.352749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:57.261270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061609869999999997,"stat_sum":{"num_bytes":14245271,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":123,"num_read_kb":12313,"num_write":620,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"56'230","reported_seq":302,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.389069+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:58:51.389069+0000","last_peered":"2023-10-30T13:58:51.389069+0000","last_clean":"2023-10-30T13:58:51.389069+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:58:51.389069+0000","last_undegraded":"2023-10-30T13:58:51.389069+0000","last_fullsized":"2023-10-30T13:58:51.389069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":24606,"num_write":205,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"56'700","reported_seq":2905,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.336024+0000","last_change":"2023-10-30T13:45:27.948077+0000","last_active":"2023-10-30T13:59:02.336024+0000","last_peered":"2023-10-30T13:59:02.336024+0000","last_clean":"2023-10-30T13:59:02.336024+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:59:02.336024+0000","last_undegraded":"2023-10-30T13:59:02.336024+0000","last_fullsized":"2023-10-30T13:59:02.336024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:48.941885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061457359999999997,"stat_sum":{"num_bytes":11313833,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":112,"num_read_kb":16,"num_write":670,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"56'950","reported_seq":1044,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.465087+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:58:56.465087+0000","last_peered":"2023-10-30T13:58:56.465087+0000","last_clean":"2023-10-30T13:58:56.465087+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:58:56.465087+0000","last_undegraded":"2023-10-30T13:58:56.465087+0000","last_fullsized":"2023-10-30T13:58:56.465087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":103644,"num_write":908,"num_write_kb":16901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"56'670","reported_seq":2886,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.415474+0000","last_change":"2023-10-30T13:45:27.941557+0000","last_active":"2023-10-30T13:59:02.415474+0000","last_peered":"2023-10-30T13:59:02.415474+0000","last_clean":"2023-10-30T13:59:02.415474+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:59:02.415474+0000","last_undegraded":"2023-10-30T13:59:02.415474+0000","last_fullsized":"2023-10-30T13:59:02.415474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:51.049883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029730300000000002,"stat_sum":{"num_bytes":19886092,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":121,"num_read_kb":4113,"num_write":647,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"56'585","reported_seq":664,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:41.720146+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:58:41.720146+0000","last_peered":"2023-10-30T13:58:41.720146+0000","last_clean":"2023-10-30T13:58:41.720146+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:58:41.720146+0000","last_undegraded":"2023-10-30T13:58:41.720146+0000","last_fullsized":"2023-10-30T13:58:41.720146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":72355,"num_write":540,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"56'629","reported_seq":2822,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.591316+0000","last_change":"2023-10-30T13:45:28.781420+0000","last_active":"2023-10-30T13:58:47.591316+0000","last_peered":"2023-10-30T13:58:47.591316+0000","last_clean":"2023-10-30T13:58:47.591316+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:58:47.591316+0000","last_undegraded":"2023-10-30T13:58:47.591316+0000","last_fullsized":"2023-10-30T13:58:47.591316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:15.218245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026777319999999999,"stat_sum":{"num_bytes":7610379,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":98,"num_read_kb":4107,"num_write":602,"num_write_kb":17238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"56'356","reported_seq":451,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382987+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:58:51.382987+0000","last_peered":"2023-10-30T13:58:51.382987+0000","last_clean":"2023-10-30T13:58:51.382987+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:58:51.382987+0000","last_undegraded":"2023-10-30T13:58:51.382987+0000","last_fullsized":"2023-10-30T13:58:51.382987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4418509,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":57427,"num_write":339,"num_write_kb":8706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":916305331,"num_objects":9467,"num_object_clones":0,"num_object_copies":18934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9467,"num_whiteouts":0,"num_read":7268,"num_read_kb":400185,"num_write":40670,"num_write_kb":1941038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880854528,"data_stored":1831121052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42732,"ondisk_log_size":42732,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":297795987,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1553,"num_read_kb":1442867,"num_write":16395,"num_write_kb":373220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595755008,"data_stored":595591974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17196,"ondisk_log_size":17196,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509734,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447344,"kb_used_data":339816,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93924496,"statfs":{"total":96636764160,"available":96178683904,"internally_reserved":0,"allocated":347971584,"data_stored":341021827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":110098891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85199999999999998,"15min":0.81200000000000006},"min":{"1min":0.433,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.63},{"interface":"front","average":{"1min":1.1830000000000001,"5min":0.96299999999999997,"15min":0.90600000000000003},"min":{"1min":0.53500000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.3499999999999996,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.0900000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.86199999999999999,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.0289999999999999,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.02,"5min":0.95499999999999996,"15min":0.92300000000000004},"min":{"1min":0.57399999999999995,"5min":0.45200000000000001,"15min":0.33400000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.625},"last":0.74099999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0389999999999999,"15min":0.97299999999999998},"min":{"1min":0.626,"5min":0.51300000000000001,"15min":0.32000000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.877},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.91700000000000004,"15min":0.94499999999999995},"min":{"1min":0.63300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":4.1120000000000001},"last":0.998}]},{"osd":3,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.75600000000000001,"5min":0.55300000000000005,"15min":0.45900000000000002},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":4.2069999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":4.4189999999999996,"5min":1.645,"15min":1.204},"min":{"1min":0.83299999999999996,"5min":0.47199999999999998,"15min":0.44400000000000001},"max":{"1min":59.844999999999999,"5min":59.844999999999999,"15min":59.844999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1080000000000001,"15min":1.089},"min":{"1min":0.85899999999999999,"5min":0.623,"15min":0.38100000000000001},"max":{"1min":1.853,"5min":1.853,"15min":3.1059999999999999},"last":1.329},{"interface":"front","average":{"1min":1.335,"5min":1.3220000000000001,"15min":1.2769999999999999},"min":{"1min":1.137,"5min":0.754,"15min":0.42399999999999999},"max":{"1min":1.907,"5min":1.907,"15min":3.198},"last":1.401}]},{"osd":5,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0049999999999999,"15min":1.093},"min":{"1min":0.65600000000000003,"5min":0.53100000000000003,"15min":0.40300000000000002},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":4.2759999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.228,"5min":1.258,"15min":1.238},"min":{"1min":0.80000000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5329999999999999,"5min":1.8380000000000001,"15min":3.4380000000000002},"last":1.252}]},{"osd":6,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92200000000000004,"15min":1.091},"min":{"1min":0.55400000000000005,"5min":0.48999999999999999,"15min":0.48399999999999999},"max":{"1min":1.3109999999999999,"5min":1.429,"15min":3.9630000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.458,"5min":1.131,"15min":1.216},"min":{"1min":0.84399999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":7.7060000000000004,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.95599999999999996}]}]},{"osd":3,"up_from":15,"seq":64424509733,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410640,"kb_used_data":311432,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93961200,"statfs":{"total":96636764160,"available":96216268800,"internally_reserved":0,"allocated":318906368,"data_stored":311949814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":101579208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":9.3320000000000007},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0369999999999999,"15min":0.72399999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.26100000000000001},"max":{"1min":0.71599999999999997,"5min":38.082999999999998,"15min":38.082999999999998},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.70199999999999996,"15min":0.60699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":11.599,"15min":11.599},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.70899999999999996,"15min":0.66100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":15.715999999999999,"15min":15.715999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.72699999999999998,"15min":0.66700000000000004},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.28999999999999998},"max":{"1min":0.83899999999999997,"5min":11.646000000000001,"15min":12.351000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.755,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.83399999999999996,"5min":15.641999999999999,"15min":15.641999999999999},"last":0.502}]},{"osd":4,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.016,"15min":0.92000000000000004},"min":{"1min":0.59999999999999998,"5min":0.55400000000000005,"15min":0.51600000000000001},"max":{"1min":0.98899999999999999,"5min":15.759,"15min":15.759},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.298,"15min":1.0640000000000001},"min":{"1min":0.58499999999999996,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":38.137999999999998,"15min":38.137999999999998},"last":0.88400000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.4139999999999999,"15min":1.097},"min":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.52600000000000002},"max":{"1min":1.036,"5min":40.808,"15min":40.808},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.0309999999999999,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.55800000000000005,"15min":0.49299999999999999},"max":{"1min":1.0660000000000001,"5min":16.498000000000001,"15min":19.835999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.96499999999999997,"15min":0.91800000000000004},"min":{"1min":0.621,"5min":0.56799999999999995,"15min":0.53500000000000003},"max":{"1min":1.2370000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.998,"15min":0.96099999999999997},"min":{"1min":0.68500000000000005,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":1.091,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1020000000000001,"15min":0.95799999999999996},"min":{"1min":0.72499999999999998,"5min":0.60599999999999998,"15min":0.59999999999999998},"max":{"1min":1.2549999999999999,"5min":16.082999999999998,"15min":16.082999999999998},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.88,"5min":1.5609999999999999,"15min":1.111},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.58199999999999996},"max":{"1min":1.228,"5min":52.902999999999999,"15min":52.902999999999999},"last":0.98299999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232516,"kb_used_data":187452,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94139324,"statfs":{"total":96636764160,"available":96398667776,"internally_reserved":0,"allocated":191950848,"data_stored":187647968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.45700000000000002},"min":{"1min":0.32100000000000001,"5min":0.251,"15min":0.246},"max":{"1min":0.83199999999999996,"5min":3.407,"15min":3.8969999999999998},"last":0.495},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.371,"5min":0.35299999999999998,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":5.6980000000000004,"15min":5.6980000000000004},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51000000000000001,"15min":0.60099999999999998},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68100000000000005,"5min":5.5750000000000002,"15min":36.109000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58899999999999997,"5min":5.4569999999999999,"15min":8.1890000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.63500000000000001,"15min":0.68999999999999995},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.29199999999999998},"max":{"1min":0.73999999999999999,"5min":5.532,"15min":35.890999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66800000000000004,"15min":0.69799999999999995},"min":{"1min":0.48199999999999998,"5min":0.42899999999999999,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":5.2830000000000004,"15min":22.846},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84099999999999997},"min":{"1min":0.67000000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0549999999999999,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.496,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":5.6059999999999999,"15min":7.8220000000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.93999999999999995,"5min":5.4260000000000002,"15min":9.9109999999999996},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.86499999999999999},"min":{"1min":0.70799999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":5.4870000000000001,"15min":5.4870000000000001},"last":0.874}]},{"osd":6,"last update":"Mon Oct 30 13:58:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":1.2030000000000001,"15min":0.95999999999999996},"min":{"1min":0.64100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0960000000000001,"15min":0.97599999999999998},"min":{"1min":0.68300000000000005,"5min":0.60099999999999998,"15min":0.56599999999999995},"max":{"1min":15.975,"5min":15.975,"15min":15.975},"last":0.86299999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.75700000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.169,"5min":5.3899999999999997,"15min":5.3899999999999997},"last":1.036},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.91900000000000004,"15min":0.91600000000000004},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53700000000000003},"max":{"1min":1.0800000000000001,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":0.84199999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509732,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365392,"kb_used_data":257928,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94006448,"statfs":{"total":96636764160,"available":96262602752,"internally_reserved":0,"allocated":264118272,"data_stored":259563065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.437},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.245},"max":{"1min":0.67200000000000004,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55100000000000005,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":0.66700000000000004,"5min":2.5920000000000001,"15min":22.149000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.315},"max":{"1min":0.749,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.443}]},{"osd":3,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.75900000000000001},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":2.8130000000000002,"15min":54.530000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.67800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34100000000000003},"max":{"1min":0.83799999999999997,"5min":2.879,"15min":25.457000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78300000000000003,"15min":0.81299999999999994},"min":{"1min":0.53900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94699999999999995,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.79500000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":1.054,"5min":2.101,"15min":2.101},"last":0.70199999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83699999999999997,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.505},"max":{"1min":1.012,"5min":2.7810000000000001,"15min":11.164999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83599999999999997},"min":{"1min":0.624,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":0.94399999999999995,"5min":2.8460000000000001,"15min":2.8820000000000001},"last":0.79000000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.83699999999999997},"min":{"1min":0.57299999999999995,"5min":0.55600000000000005,"15min":0.497},"max":{"1min":0.95399999999999996,"5min":2.903,"15min":5.9180000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.85299999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":1.9930000000000001,"15min":2.2629999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:58:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.002,"15min":0.96899999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":12.221,"5min":12.221,"15min":12.221},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.373,"5min":1.0209999999999999,"15min":0.94499999999999995},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":12.153,"5min":12.153,"15min":12.153},"last":0.77800000000000002}]}]},{"osd":6,"up_from":14,"seq":60129542439,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333848,"kb_used_data":240272,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94037992,"statfs":{"total":96636764160,"available":96294903808,"internally_reserved":0,"allocated":246038528,"data_stored":240883050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":95812039},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.1240000000000001,"15min":1.042},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.41999999999999998},"max":{"1min":1.0389999999999999,"5min":19.741,"15min":19.741},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.131,"15min":0.99099999999999999},"min":{"1min":0.69599999999999995,"5min":0.55800000000000005,"15min":0.34899999999999998},"max":{"1min":1.3600000000000001,"5min":13.863,"15min":13.863},"last":0.98199999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93700000000000006,"15min":0.86799999999999999},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.46100000000000002},"max":{"1min":1.0669999999999999,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.1819999999999999,"15min":1.0289999999999999},"min":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.46800000000000003},"max":{"1min":1.2929999999999999,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96199999999999997,"15min":0.97699999999999998},"min":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.335,"5min":3.5329999999999999,"15min":3.7709999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.095,"15min":1.083},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.4059999999999999,"5min":3.407,"15min":4.6840000000000002},"last":1.1519999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.97599999999999998,"15min":1.014},"min":{"1min":0.67100000000000004,"5min":0.52800000000000002,"15min":0.46700000000000003},"max":{"1min":1.359,"5min":3.7999999999999998,"15min":4.7629999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.171,"15min":1.0840000000000001},"min":{"1min":0.92900000000000005,"5min":0.59299999999999997,"15min":0.54600000000000004},"max":{"1min":1.4179999999999999,"5min":3.968,"15min":3.968},"last":1.0589999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2290000000000001,"15min":1.208},"min":{"1min":0.79600000000000004,"5min":0.55800000000000005,"15min":0.51000000000000001},"max":{"1min":1.4710000000000001,"5min":3.8900000000000001,"15min":16.204000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1519999999999999,"15min":1.2649999999999999},"min":{"1min":0.63700000000000001,"5min":0.622,"15min":0.42699999999999999},"max":{"1min":1.399,"5min":19.398,"15min":43.198},"last":1.246}]},{"osd":5,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.98499999999999999,"15min":1.0489999999999999},"min":{"1min":0.70899999999999996,"5min":0.47699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98399999999999999,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3129999999999999,"15min":1.3009999999999999},"min":{"1min":1.1879999999999999,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":1.522,"5min":3.7269999999999999,"15min":3.802},"last":1.196}]},{"osd":7,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.833,"15min":1.5880000000000001},"min":{"1min":0.83499999999999996,"5min":0.58899999999999997,"15min":0.55400000000000005},"max":{"1min":1.2709999999999999,"5min":70.335999999999999,"15min":73.228999999999999},"last":1.026},{"interface":"front","average":{"1min":1.079,"5min":1.675,"15min":1.637},"min":{"1min":0.82699999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.381,"5min":42.164999999999999,"15min":73.289000000000001},"last":0.76300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542437,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446168,"kb_used_data":352528,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93925672,"statfs":{"total":96636764160,"available":96179888128,"internally_reserved":0,"allocated":360988672,"data_stored":352973919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.89900000000000002,"15min":0.879},"min":{"1min":0.54600000000000004,"5min":0.46999999999999997,"15min":0.40400000000000003},"max":{"1min":3.9159999999999999,"5min":3.9159999999999999,"15min":5.0599999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.83099999999999996,"15min":0.872},"min":{"1min":0.52900000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0569999999999999,"5min":1.5620000000000001,"15min":3.4849999999999999},"last":1.724}]},{"osd":1,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.92600000000000005,"15min":0.871},"min":{"1min":0.55300000000000005,"5min":0.40600000000000003,"15min":0.33600000000000002},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.85099999999999998,"15min":0.90000000000000002},"min":{"1min":0.50700000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95899999999999996,"15min":0.95599999999999996},"min":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.51100000000000001},"max":{"1min":1.462,"5min":1.897,"15min":5.3710000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1279999999999999,"15min":1.056},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.53200000000000003},"max":{"1min":1.369,"5min":2.081,"15min":2.887},"last":0.70099999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.095,"15min":1.0620000000000001},"min":{"1min":0.82999999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":5.476},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.97599999999999998,"15min":1.0349999999999999},"min":{"1min":0.67200000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1779999999999999,"5min":1.52,"15min":3.8500000000000001},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95599999999999996,"15min":0.98699999999999999},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.38400000000000001},"max":{"1min":1.3360000000000001,"5min":1.847,"15min":4.1710000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.66900000000000004,"5min":0.49199999999999999,"15min":0.41799999999999998},"max":{"1min":1.3009999999999999,"5min":1.7270000000000001,"15min":4.2519999999999998},"last":0.752}]},{"osd":6,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1899999999999999,"15min":1.1739999999999999},"min":{"1min":0.747,"5min":0.65000000000000002,"15min":0.621},"max":{"1min":1.546,"5min":2.177,"15min":5.6029999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1459999999999999,"15min":1.1739999999999999},"min":{"1min":0.69499999999999995,"5min":0.56100000000000005,"15min":0.41899999999999998},"max":{"1min":1.403,"5min":1.839,"15min":4.077},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.147,"15min":1.23},"min":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.45100000000000001},"max":{"1min":1.4650000000000001,"5min":1.7949999999999999,"15min":9.9390000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.2430000000000001,"15min":1.2889999999999999},"min":{"1min":0.97399999999999998,"5min":0.78000000000000003,"15min":0.48499999999999999},"max":{"1min":1.415,"5min":1.907,"15min":4.3399999999999999},"last":0.5}]}]},{"osd":4,"up_from":14,"seq":60129542437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433140,"kb_used_data":351980,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93938700,"statfs":{"total":96636764160,"available":96193228800,"internally_reserved":0,"allocated":360427520,"data_stored":352125363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.91100000000000003,"15min":0.873},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.167,"5min":8.7680000000000007,"15min":8.7680000000000007},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.89000000000000001,"15min":0.86099999999999999},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.125,"5min":6.5990000000000002,"15min":6.5990000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82899999999999996,"15min":0.89700000000000002},"min":{"1min":0.51600000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.236,"5min":1.375,"15min":5.8010000000000002},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.85899999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.262,"5min":1.6970000000000001,"15min":3.0499999999999998},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88500000000000001,"15min":0.91600000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.212,"5min":1.331,"15min":3.323},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.98599999999999999,"15min":0.97099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.252,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.1100000000000001,"15min":1.0209999999999999},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.151,"5min":8.0570000000000004,"15min":8.0570000000000004},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.056,"15min":1.0009999999999999},"min":{"1min":0.65900000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.381,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.018,"15min":1.0600000000000001},"min":{"1min":0.82699999999999996,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.325,"5min":4.0030000000000001,"15min":8.4550000000000001},"last":1.149},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.175,"15min":1.171},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.57699999999999996},"max":{"1min":1.3959999999999999,"5min":4.0609999999999999,"15min":8.5399999999999991},"last":0.92300000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1859999999999999,"15min":1.083},"min":{"1min":0.76300000000000001,"5min":0.503,"15min":0.45100000000000001},"max":{"1min":1.4870000000000001,"5min":16.989999999999998,"15min":16.989999999999998},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.173,"15min":1.149},"min":{"1min":0.86699999999999999,"5min":0.41299999999999998,"15min":0.36699999999999999},"max":{"1min":1.599,"5min":4.4530000000000003,"15min":4.4530000000000003},"last":1.1819999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:42 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1719999999999999,"15min":1.163},"min":{"1min":0.84199999999999997,"5min":0.45100000000000001,"15min":0.42199999999999999},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":3.4510000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.298,"15min":1.3049999999999999},"min":{"1min":1.0269999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.5470000000000002},"last":1.2450000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469860,"kb_used_data":376220,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93901980,"statfs":{"total":96636764160,"available":96155627520,"internally_reserved":0,"allocated":385249280,"data_stored":377198604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.98899999999999999,"15min":0.63900000000000001},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":0.63900000000000001,"5min":45.311,"15min":45.311},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.59599999999999997},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.27200000000000002},"max":{"1min":0.72599999999999998,"5min":2.2410000000000001,"15min":27.616},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.29099999999999998},"max":{"1min":0.82999999999999996,"5min":0.88100000000000001,"15min":2.5449999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.44,"5min":0.40899999999999997,"15min":0.35199999999999998},"max":{"1min":0.752,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.57599999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.67500000000000004,"15min":0.56100000000000005},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":0.79300000000000004,"5min":12.368,"15min":12.368},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.75800000000000001,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.40899999999999997},"max":{"1min":0.83299999999999996,"5min":12.442,"15min":12.442},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84799999999999998,"15min":0.85999999999999999},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.41599999999999998},"max":{"1min":1,"5min":2.6949999999999998,"15min":9.4979999999999993},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.871,"15min":0.86499999999999999},"min":{"1min":0.629,"5min":0.60199999999999998,"15min":0.46700000000000003},"max":{"1min":1.1220000000000001,"5min":2.6440000000000001,"15min":9.5429999999999993},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.88,"15min":0.878},"min":{"1min":0.72199999999999998,"5min":0.47099999999999997,"15min":0.317},"max":{"1min":1.087,"5min":1.163,"15min":11.026},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.375,"15min":1.0309999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.46999999999999997},"max":{"1min":1.1759999999999999,"5min":44.225999999999999,"15min":44.225999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88,"15min":0.873},"min":{"1min":0.73899999999999999,"5min":0.63,"15min":0.48799999999999999},"max":{"1min":1.0649999999999999,"5min":2.5169999999999999,"15min":2.9300000000000002},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.435},"max":{"1min":1.133,"5min":1.236,"15min":11.087999999999999},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.91400000000000003,"15min":0.879},"min":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.50900000000000001},"max":{"1min":1.0740000000000001,"5min":5.6849999999999996,"15min":5.6849999999999996},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.93200000000000005,"15min":0.93300000000000005},"min":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.57199999999999995},"max":{"1min":1.0840000000000001,"5min":2.6030000000000002,"15min":9.5890000000000004},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58961920,"data_stored":58944539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50540544,"data_stored":50514908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75808768,"data_stored":75783932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":88170496,"data_stored":88156866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":87515136,"data_stored":87493416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":70914048,"data_stored":70895130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79933440,"data_stored":79916159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325066752,"data_stored":317352305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179568640,"data_stored":175353072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140771328,"data_stored":136810091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242458624,"data_stored":235842913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":273088512,"data_stored":265087328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":275906560,"data_stored":268118820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":177188864,"data_stored":172364192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266805248,"data_stored":260192331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:59:04.989 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T13:59:04.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:59:05.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f536553e640 1 Processor -- start 2023-10-30T13:59:05.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f536553e640 1 -- start start 2023-10-30T13:59:05.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f536553e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5360153c80 0x7f536015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f536553e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f53600c4a10 0x7f53600c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f536553e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5360159f20 con 0x7f536015a3b0 2023-10-30T13:59:05.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f536553e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536015dec0 con 0x7f5360153c80 2023-10-30T13:59:05.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f536553e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53601599a0 con 0x7f53600c4a10 2023-10-30T13:59:05.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f535effd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5360153c80 0x7f536015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f535effd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5360153c80 0x7f536015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35624/0 (socket says 172.21.15.31:35624) 2023-10-30T13:59:05.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f535effd640 1 -- 172.21.15.31:0/3049457685 learned_addr learned my addr 172.21.15.31:0/3049457685 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:05.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f5357fff640 1 --2- 172.21.15.31:0/3049457685 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f53600c4a10 0x7f53600c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f535effd640 1 -- 172.21.15.31:0/3049457685 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f53600c4a10 msgr2=0x7f53600c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:05.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f535effd640 1 --2- 172.21.15.31:0/3049457685 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f53600c4a10 0x7f53600c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f535effd640 1 -- 172.21.15.31:0/3049457685 >> v1:172.21.15.33:6789/0 conn(0x7f536015a3b0 legacy=0x7f53600ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:05.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.168+0000 7f535effd640 1 -- 172.21.15.31:0/3049457685 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600c47e0 con 0x7f5360153c80 2023-10-30T13:59:05.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f535effd640 1 --2- 172.21.15.31:0/3049457685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5360153c80 0x7f536015cf90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f5350009870 tx=0x7f535001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6bbd4cb9032c01 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:05.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f535e7fc640 1 -- 172.21.15.31:0/3049457685 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53500a89b0 con 0x7f5360153c80 2023-10-30T13:59:05.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f535e7fc640 1 -- 172.21.15.31:0/3049457685 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53500a8b10 con 0x7f5360153c80 2023-10-30T13:59:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f535e7fc640 1 -- 172.21.15.31:0/3049457685 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53500a8ea0 con 0x7f5360153c80 2023-10-30T13:59:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 -- 172.21.15.31:0/3049457685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5360153c80 msgr2=0x7f536015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 --2- 172.21.15.31:0/3049457685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5360153c80 0x7f536015cf90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f5350009870 tx=0x7f535001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 -- 172.21.15.31:0/3049457685 shutdown_connections 2023-10-30T13:59:05.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 --2- 172.21.15.31:0/3049457685 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f53600c4a10 0x7f53600c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 --2- 172.21.15.31:0/3049457685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5360153c80 0x7f536015cf90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 -- 172.21.15.31:0/3049457685 >> 172.21.15.31:0/3049457685 conn(0x7f53600715c0 msgr2=0x7f53600719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:05.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 -- 172.21.15.31:0/3049457685 shutdown_connections 2023-10-30T13:59:05.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.172+0000 7f536553e640 1 -- 172.21.15.31:0/3049457685 wait complete. 2023-10-30T13:59:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 Processor -- start 2023-10-30T13:59:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 -- start start 2023-10-30T13:59:05.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53600c4a10 0x7f536014fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5360153c80 0x7f53600b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f536015a3b0 0x7f53600c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536014f170 con 0x7f53600c4a10 2023-10-30T13:59:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536014e9e0 con 0x7f5360153c80 2023-10-30T13:59:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536014ec80 con 0x7f536015a3b0 2023-10-30T13:59:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53600c4a10 0x7f536014fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53600c4a10 0x7f536014fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35626/0 (socket says 172.21.15.31:35626) 2023-10-30T13:59:05.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 -- 172.21.15.31:0/2326443127 learned_addr learned my addr 172.21.15.31:0/2326443127 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:05.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f5357fff640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5360153c80 0x7f53600b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535f7fe640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f536015a3b0 0x7f53600c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 -- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f536015a3b0 msgr2=0x7f53600c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f536015a3b0 0x7f53600c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 -- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5360153c80 msgr2=0x7f53600b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5360153c80 0x7f53600b32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 -- 172.21.15.31:0/2326443127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500094d0 con 0x7f53600c4a10 2023-10-30T13:59:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f535effd640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53600c4a10 0x7f536014fe80 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f53500b3040 tx=0x7f535001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f53577fe640 1 -- 172.21.15.31:0/2326443127 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53500a89b0 con 0x7f53600c4a10 2023-10-30T13:59:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f53577fe640 1 -- 172.21.15.31:0/2326443127 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53500bb480 con 0x7f53600c4a10 2023-10-30T13:59:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f53577fe640 1 -- 172.21.15.31:0/2326443127 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53500bb830 con 0x7f53600c4a10 2023-10-30T13:59:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536014f3a0 con 0x7f53600c4a10 2023-10-30T13:59:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.176+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536014f960 con 0x7f53600c4a10 2023-10-30T13:59:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.180+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536004e170 con 0x7f53600c4a10 2023-10-30T13:59:05.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.180+0000 7f53577fe640 1 -- 172.21.15.31:0/2326443127 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53500bb990 con 0x7f53600c4a10 2023-10-30T13:59:05.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.180+0000 7f53577fe640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f53400664d0 0x7f5340068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.180+0000 7f53577fe640 1 -- 172.21.15.31:0/2326443127 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f53501228e0 con 0x7f53600c4a10 2023-10-30T13:59:05.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.180+0000 7f5357fff640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f53400664d0 0x7f5340068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.180+0000 7f5357fff640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f53400664d0 0x7f5340068990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f53601f3030 tx=0x7f5348008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:05.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.184+0000 7f53577fe640 1 -- 172.21.15.31:0/2326443127 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53500ef920 con 0x7f53600c4a10 2023-10-30T13:59:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.336+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f536015d900 con 0x7f53400664d0 2023-10-30T13:59:05.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.360+0000 7f53577fe640 1 -- 172.21.15.31:0/2326443127 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292756 (secure 0 0 0) 0x7f534800b200 con 0x7f53400664d0 2023-10-30T13:59:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:05.366 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:59:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f53400664d0 msgr2=0x7f5340068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f53400664d0 0x7f5340068990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f53601f3030 tx=0x7f5348008040 comp rx=0 tx=0).stop 2023-10-30T13:59:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53600c4a10 msgr2=0x7f536014fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53600c4a10 0x7f536014fe80 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f53500b3040 tx=0x7f535001ba90 comp rx=0 tx=0).stop 2023-10-30T13:59:05.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 shutdown_connections 2023-10-30T13:59:05.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f53400664d0 0x7f5340068990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f536015a3b0 0x7f53600c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5360153c80 0x7f53600b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 --2- 172.21.15.31:0/2326443127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53600c4a10 0x7f536014fe80 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 >> 172.21.15.31:0/2326443127 conn(0x7f53600715c0 msgr2=0x7f53600cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:05.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 shutdown_connections 2023-10-30T13:59:05.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.364+0000 7f536553e640 1 -- 172.21.15.31:0/2326443127 wait complete. 2023-10-30T13:59:05.385 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":774,"stamp":"2023-10-30T13:59:03.714457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1214691686,"num_objects":9684,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9684,"num_whiteouts":0,"num_read":8937,"num_read_kb":1843152,"num_write":57224,"num_write_kb":2316674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60045,"ondisk_log_size":60045,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3138908,"kb_used_data":2417628,"kb_used_omap":12,"kb_used_meta":721203,"kb_avail":751835812,"statfs":{"total":773094113280,"available":769879871488,"internally_reserved":0,"allocated":2475651072,"data_stored":2423363610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":738512440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28176770,"num_objects":827,"num_object_clones":0,"num_object_copies":1654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1154,"num_write_kb":28302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004989"},"pg_stats":[{"pgid":"3.3f","version":"56'647","reported_seq":2866,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.625135+0000","last_change":"2023-10-30T13:45:28.601107+0000","last_active":"2023-10-30T13:58:47.625135+0000","last_peered":"2023-10-30T13:58:47.625135+0000","last_clean":"2023-10-30T13:58:47.625135+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:58:47.625135+0000","last_undegraded":"2023-10-30T13:58:47.625135+0000","last_fullsized":"2023-10-30T13:58:47.625135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:58.462635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098613959999999997,"stat_sum":{"num_bytes":30381033,"num_objects":125,"num_object_clones":0,"num_object_copies":250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":124,"num_read_kb":8207,"num_write":615,"num_write_kb":38998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"56'626","reported_seq":2825,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.650883+0000","last_change":"2023-10-30T13:45:27.935552+0000","last_active":"2023-10-30T13:58:47.650883+0000","last_peered":"2023-10-30T13:58:47.650883+0000","last_clean":"2023-10-30T13:58:47.650883+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:58:47.650883+0000","last_undegraded":"2023-10-30T13:58:47.650883+0000","last_fullsized":"2023-10-30T13:58:47.650883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:50.606021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001955118,"stat_sum":{"num_bytes":9798818,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":104,"num_read_kb":11,"num_write":587,"num_write_kb":23396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"56'679","reported_seq":2881,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.382336+0000","last_change":"2023-10-30T13:45:27.939453+0000","last_active":"2023-10-30T13:59:02.382336+0000","last_peered":"2023-10-30T13:59:02.382336+0000","last_clean":"2023-10-30T13:59:02.382336+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:59:02.382336+0000","last_undegraded":"2023-10-30T13:59:02.382336+0000","last_fullsized":"2023-10-30T13:59:02.382336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:59.423451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021120940000000001,"stat_sum":{"num_bytes":23390266,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":110,"num_read_kb":20497,"num_write":638,"num_write_kb":47001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"56'663","reported_seq":2888,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.599578+0000","last_change":"2023-10-30T13:45:28.487484+0000","last_active":"2023-10-30T13:58:47.599578+0000","last_peered":"2023-10-30T13:58:47.599578+0000","last_clean":"2023-10-30T13:58:47.599578+0000","last_became_active":"2023-10-30T13:35:48.779909+0000","last_became_peered":"2023-10-30T13:35:48.779909+0000","last_unstale":"2023-10-30T13:58:47.599578+0000","last_undegraded":"2023-10-30T13:58:47.599578+0000","last_fullsized":"2023-10-30T13:58:47.599578+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:51.343491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001317775,"stat_sum":{"num_bytes":9891362,"num_objects":139,"num_object_clones":0,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":133,"num_read_kb":8205,"num_write":636,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"56'666","reported_seq":2869,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.626617+0000","last_change":"2023-10-30T13:45:28.597729+0000","last_active":"2023-10-30T13:58:47.626617+0000","last_peered":"2023-10-30T13:58:47.626617+0000","last_clean":"2023-10-30T13:58:47.626617+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:58:47.626617+0000","last_undegraded":"2023-10-30T13:58:47.626617+0000","last_fullsized":"2023-10-30T13:58:47.626617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:29.242932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007614844,"stat_sum":{"num_bytes":19656703,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":110,"num_read_kb":8208,"num_write":638,"num_write_kb":39882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"56'661","reported_seq":2858,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.395628+0000","last_change":"2023-10-30T13:45:27.941070+0000","last_active":"2023-10-30T13:59:02.395628+0000","last_peered":"2023-10-30T13:59:02.395628+0000","last_clean":"2023-10-30T13:59:02.395628+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:59:02.395628+0000","last_undegraded":"2023-10-30T13:59:02.395628+0000","last_fullsized":"2023-10-30T13:59:02.395628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:40.443395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030849649999999998,"stat_sum":{"num_bytes":19227960,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":104,"num_read_kb":7536,"num_write":633,"num_write_kb":20026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"56'662","reported_seq":2846,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.643842+0000","last_change":"2023-10-30T13:45:28.597665+0000","last_active":"2023-10-30T13:58:47.643842+0000","last_peered":"2023-10-30T13:58:47.643842+0000","last_clean":"2023-10-30T13:58:47.643842+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:58:47.643842+0000","last_undegraded":"2023-10-30T13:58:47.643842+0000","last_fullsized":"2023-10-30T13:58:47.643842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:44.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082986029999999999,"stat_sum":{"num_bytes":9880254,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":91,"num_read_kb":11,"num_write":635,"num_write_kb":29079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"56'645","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.642291+0000","last_change":"2023-10-30T13:45:28.780454+0000","last_active":"2023-10-30T13:58:47.642291+0000","last_peered":"2023-10-30T13:58:47.642291+0000","last_clean":"2023-10-30T13:58:47.642291+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:58:47.642291+0000","last_undegraded":"2023-10-30T13:58:47.642291+0000","last_fullsized":"2023-10-30T13:58:47.642291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:57.938730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021728070000000001,"stat_sum":{"num_bytes":10096886,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":132,"num_read_kb":1147,"num_write":620,"num_write_kb":24646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"56'631","reported_seq":2847,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.628018+0000","last_change":"2023-10-30T13:45:28.601219+0000","last_active":"2023-10-30T13:58:47.628018+0000","last_peered":"2023-10-30T13:58:47.628018+0000","last_clean":"2023-10-30T13:58:47.628018+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:58:47.628018+0000","last_undegraded":"2023-10-30T13:58:47.628018+0000","last_fullsized":"2023-10-30T13:58:47.628018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:53.268188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010045174,"stat_sum":{"num_bytes":10190840,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":124,"num_read_kb":4115,"num_write":586,"num_write_kb":23657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"56'612","reported_seq":2818,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.344375+0000","last_change":"2023-10-30T13:45:28.050787+0000","last_active":"2023-10-30T13:59:02.344375+0000","last_peered":"2023-10-30T13:59:02.344375+0000","last_clean":"2023-10-30T13:59:02.344375+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:59:02.344375+0000","last_undegraded":"2023-10-30T13:59:02.344375+0000","last_fullsized":"2023-10-30T13:59:02.344375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:02.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047277760000000004,"stat_sum":{"num_bytes":20230336,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":113,"num_read_kb":28,"num_write":585,"num_write_kb":25967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"56'610","reported_seq":2800,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.302667+0000","last_change":"2023-10-30T13:45:28.050006+0000","last_active":"2023-10-30T13:59:02.302667+0000","last_peered":"2023-10-30T13:59:02.302667+0000","last_clean":"2023-10-30T13:59:02.302667+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:59:02.302667+0000","last_undegraded":"2023-10-30T13:59:02.302667+0000","last_fullsized":"2023-10-30T13:59:02.302667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:30.724902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00429868,"stat_sum":{"num_bytes":2384023,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":97,"num_read_kb":17,"num_write":586,"num_write_kb":9055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"56'626","reported_seq":2835,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.294793+0000","last_change":"2023-10-30T13:45:27.946083+0000","last_active":"2023-10-30T13:59:02.294793+0000","last_peered":"2023-10-30T13:59:02.294793+0000","last_clean":"2023-10-30T13:59:02.294793+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:59:02.294793+0000","last_undegraded":"2023-10-30T13:59:02.294793+0000","last_fullsized":"2023-10-30T13:59:02.294793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:19.253547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066533010000000004,"stat_sum":{"num_bytes":18341132,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":8208,"num_write":596,"num_write_kb":32418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"56'610","reported_seq":2816,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.362815+0000","last_change":"2023-10-30T13:45:28.780475+0000","last_active":"2023-10-30T13:58:51.362815+0000","last_peered":"2023-10-30T13:58:51.362815+0000","last_clean":"2023-10-30T13:58:51.362815+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:58:51.362815+0000","last_undegraded":"2023-10-30T13:58:51.362815+0000","last_fullsized":"2023-10-30T13:58:51.362815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:10:02.225437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002338237,"stat_sum":{"num_bytes":13957458,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":114,"num_read_kb":12311,"num_write":572,"num_write_kb":36850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"56'731","reported_seq":2944,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.434802+0000","last_change":"2023-10-30T13:45:27.940425+0000","last_active":"2023-10-30T13:59:02.434802+0000","last_peered":"2023-10-30T13:59:02.434802+0000","last_clean":"2023-10-30T13:59:02.434802+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:59:02.434802+0000","last_undegraded":"2023-10-30T13:59:02.434802+0000","last_fullsized":"2023-10-30T13:59:02.434802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:57.204443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029248669999999998,"stat_sum":{"num_bytes":22804734,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":119,"num_read_kb":24041,"num_write":691,"num_write_kb":57342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"56'690","reported_seq":2896,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.631995+0000","last_change":"2023-10-30T13:45:27.934641+0000","last_active":"2023-10-30T13:58:47.631995+0000","last_peered":"2023-10-30T13:58:47.631995+0000","last_clean":"2023-10-30T13:58:47.631995+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:58:47.631995+0000","last_undegraded":"2023-10-30T13:58:47.631995+0000","last_fullsized":"2023-10-30T13:58:47.631995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:06:31.910235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021050410000000002,"stat_sum":{"num_bytes":18067659,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":112,"num_read_kb":7334,"num_write":654,"num_write_kb":39561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"56'726","reported_seq":2943,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.419953+0000","last_change":"2023-10-30T13:45:27.943054+0000","last_active":"2023-10-30T13:59:02.419953+0000","last_peered":"2023-10-30T13:59:02.419953+0000","last_clean":"2023-10-30T13:59:02.419953+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:59:02.419953+0000","last_undegraded":"2023-10-30T13:59:02.419953+0000","last_fullsized":"2023-10-30T13:59:02.419953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:29.930187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052579080000000004,"stat_sum":{"num_bytes":35259912,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":123,"num_read_kb":12306,"num_write":693,"num_write_kb":43349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"56'699","reported_seq":2895,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.379018+0000","last_change":"2023-10-30T13:45:27.950182+0000","last_active":"2023-10-30T13:59:02.379018+0000","last_peered":"2023-10-30T13:59:02.379018+0000","last_clean":"2023-10-30T13:59:02.379018+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:59:02.379018+0000","last_undegraded":"2023-10-30T13:59:02.379018+0000","last_fullsized":"2023-10-30T13:59:02.379018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:09.257269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052923609999999998,"stat_sum":{"num_bytes":14461920,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":103,"num_read_kb":4113,"num_write":659,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"56'628","reported_seq":2840,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.619440+0000","last_change":"2023-10-30T13:45:28.487735+0000","last_active":"2023-10-30T13:58:47.619440+0000","last_peered":"2023-10-30T13:58:47.619440+0000","last_clean":"2023-10-30T13:58:47.619440+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:58:47.619440+0000","last_undegraded":"2023-10-30T13:58:47.619440+0000","last_fullsized":"2023-10-30T13:58:47.619440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:05.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001369274,"stat_sum":{"num_bytes":15189051,"num_objects":119,"num_object_clones":0,"num_object_copies":238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":118,"num_read_kb":4123,"num_write":600,"num_write_kb":31137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"56'685","reported_seq":2912,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.624637+0000","last_change":"2023-10-30T13:45:28.600018+0000","last_active":"2023-10-30T13:58:47.624637+0000","last_peered":"2023-10-30T13:58:47.624637+0000","last_clean":"2023-10-30T13:58:47.624637+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:58:47.624637+0000","last_undegraded":"2023-10-30T13:58:47.624637+0000","last_fullsized":"2023-10-30T13:58:47.624637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:15.777688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092525119999999992,"stat_sum":{"num_bytes":8625831,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":134,"num_read_kb":8209,"num_write":656,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"56'660","reported_seq":2853,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.287934+0000","last_change":"2023-10-30T13:45:28.051528+0000","last_active":"2023-10-30T13:59:02.287934+0000","last_peered":"2023-10-30T13:59:02.287934+0000","last_clean":"2023-10-30T13:59:02.287934+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:59:02.287934+0000","last_undegraded":"2023-10-30T13:59:02.287934+0000","last_fullsized":"2023-10-30T13:59:02.287934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:24.699943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00502631,"stat_sum":{"num_bytes":6211385,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":100,"num_read_kb":8209,"num_write":620,"num_write_kb":19966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"56'684","reported_seq":2889,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.481517+0000","last_change":"2023-10-30T13:45:27.940012+0000","last_active":"2023-10-30T13:59:02.481517+0000","last_peered":"2023-10-30T13:59:02.481517+0000","last_clean":"2023-10-30T13:59:02.481517+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:59:02.481517+0000","last_undegraded":"2023-10-30T13:59:02.481517+0000","last_fullsized":"2023-10-30T13:59:02.481517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:13.888934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026024469999999999,"stat_sum":{"num_bytes":18097476,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":111,"num_read_kb":21,"num_write":651,"num_write_kb":32402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"56'700","reported_seq":2899,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.370220+0000","last_change":"2023-10-30T13:45:27.947213+0000","last_active":"2023-10-30T13:59:02.370220+0000","last_peered":"2023-10-30T13:59:02.370220+0000","last_clean":"2023-10-30T13:59:02.370220+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:59:02.370220+0000","last_undegraded":"2023-10-30T13:59:02.370220+0000","last_fullsized":"2023-10-30T13:59:02.370220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:24.312446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054353270000000002,"stat_sum":{"num_bytes":10272429,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":105,"num_read_kb":4117,"num_write":665,"num_write_kb":16082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"56'675","reported_seq":2882,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.598533+0000","last_change":"2023-10-30T13:45:28.781012+0000","last_active":"2023-10-30T13:58:47.598533+0000","last_peered":"2023-10-30T13:58:47.598533+0000","last_clean":"2023-10-30T13:58:47.598533+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:58:47.598533+0000","last_undegraded":"2023-10-30T13:58:47.598533+0000","last_fullsized":"2023-10-30T13:58:47.598533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:40:44.651276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025617359999999998,"stat_sum":{"num_bytes":13960173,"num_objects":128,"num_object_clones":0,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":114,"num_read_kb":8213,"num_write":647,"num_write_kb":36096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"56'705","reported_seq":2926,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.388279+0000","last_change":"2023-10-30T13:45:27.942487+0000","last_active":"2023-10-30T13:59:02.388279+0000","last_peered":"2023-10-30T13:59:02.388279+0000","last_clean":"2023-10-30T13:59:02.388279+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:59:02.388279+0000","last_undegraded":"2023-10-30T13:59:02.388279+0000","last_fullsized":"2023-10-30T13:59:02.388279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:39.369789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004034279,"stat_sum":{"num_bytes":1595822,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":126,"num_read_kb":16,"num_write":676,"num_write_kb":12551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"56'645","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.596722+0000","last_change":"2023-10-30T13:45:28.782023+0000","last_active":"2023-10-30T13:58:47.596722+0000","last_peered":"2023-10-30T13:58:47.596722+0000","last_clean":"2023-10-30T13:58:47.596722+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:58:47.596722+0000","last_undegraded":"2023-10-30T13:58:47.596722+0000","last_fullsized":"2023-10-30T13:58:47.596722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:09:53.916658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030593479999999999,"stat_sum":{"num_bytes":7588015,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":117,"num_read_kb":4108,"num_write":613,"num_write_kb":17885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"56'712","reported_seq":2918,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.378475+0000","last_change":"2023-10-30T13:45:28.051170+0000","last_active":"2023-10-30T13:59:02.378475+0000","last_peered":"2023-10-30T13:59:02.378475+0000","last_clean":"2023-10-30T13:59:02.378475+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:59:02.378475+0000","last_undegraded":"2023-10-30T13:59:02.378475+0000","last_fullsized":"2023-10-30T13:59:02.378475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:05.790369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049754609999999996,"stat_sum":{"num_bytes":9799460,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":111,"num_read_kb":24,"num_write":679,"num_write_kb":16350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"56'641","reported_seq":2846,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.609884+0000","last_change":"2023-10-30T13:45:27.935505+0000","last_active":"2023-10-30T13:58:47.609884+0000","last_peered":"2023-10-30T13:58:47.609884+0000","last_clean":"2023-10-30T13:58:47.609884+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:58:47.609884+0000","last_undegraded":"2023-10-30T13:58:47.609884+0000","last_fullsized":"2023-10-30T13:58:47.609884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:28.046138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028468439999999999,"stat_sum":{"num_bytes":33204376,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":110,"num_read_kb":16400,"num_write":604,"num_write_kb":35545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"56'610","reported_seq":2816,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.349462+0000","last_change":"2023-10-30T13:45:28.781733+0000","last_active":"2023-10-30T13:58:51.349462+0000","last_peered":"2023-10-30T13:58:51.349462+0000","last_clean":"2023-10-30T13:58:51.349462+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:58:51.349462+0000","last_undegraded":"2023-10-30T13:58:51.349462+0000","last_fullsized":"2023-10-30T13:58:51.349462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:10.458130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002839121,"stat_sum":{"num_bytes":9452766,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":112,"num_read_kb":10701,"num_write":586,"num_write_kb":14944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"56'679","reported_seq":2883,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.489266+0000","last_change":"2023-10-30T13:45:27.941228+0000","last_active":"2023-10-30T13:59:02.489266+0000","last_peered":"2023-10-30T13:59:02.489266+0000","last_clean":"2023-10-30T13:59:02.489266+0000","last_became_active":"2023-10-30T13:35:48.847825+0000","last_became_peered":"2023-10-30T13:35:48.847825+0000","last_unstale":"2023-10-30T13:59:02.489266+0000","last_undegraded":"2023-10-30T13:59:02.489266+0000","last_fullsized":"2023-10-30T13:59:02.489266+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:46.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028647770000000002,"stat_sum":{"num_bytes":7231756,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":102,"num_read_kb":4848,"num_write":653,"num_write_kb":26441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"56'688","reported_seq":2903,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.382620+0000","last_change":"2023-10-30T13:45:27.948917+0000","last_active":"2023-10-30T13:59:02.382620+0000","last_peered":"2023-10-30T13:59:02.382620+0000","last_clean":"2023-10-30T13:59:02.382620+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:59:02.382620+0000","last_undegraded":"2023-10-30T13:59:02.382620+0000","last_fullsized":"2023-10-30T13:59:02.382620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:51.840613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060200970000000003,"stat_sum":{"num_bytes":22333767,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":120,"num_read_kb":4116,"num_write":656,"num_write_kb":43887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"56'753","reported_seq":2987,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.633520+0000","last_change":"2023-10-30T13:45:28.487621+0000","last_active":"2023-10-30T13:58:47.633520+0000","last_peered":"2023-10-30T13:58:47.633520+0000","last_clean":"2023-10-30T13:58:47.633520+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:58:47.633520+0000","last_undegraded":"2023-10-30T13:58:47.633520+0000","last_fullsized":"2023-10-30T13:58:47.633520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:58:52.921718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001263171,"stat_sum":{"num_bytes":10399401,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":139,"num_read_kb":4118,"num_write":726,"num_write_kb":34561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"56'704","reported_seq":2909,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.381095+0000","last_change":"2023-10-30T13:45:28.048779+0000","last_active":"2023-10-30T13:59:02.381095+0000","last_peered":"2023-10-30T13:59:02.381095+0000","last_clean":"2023-10-30T13:59:02.381095+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:59:02.381095+0000","last_undegraded":"2023-10-30T13:59:02.381095+0000","last_fullsized":"2023-10-30T13:59:02.381095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:18.375625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036881090000000002,"stat_sum":{"num_bytes":14815628,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":110,"num_read_kb":20,"num_write":677,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"56'677","reported_seq":2878,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.248436+0000","last_change":"2023-10-30T13:45:27.944400+0000","last_active":"2023-10-30T13:59:02.248436+0000","last_peered":"2023-10-30T13:59:02.248436+0000","last_clean":"2023-10-30T13:59:02.248436+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:59:02.248436+0000","last_undegraded":"2023-10-30T13:59:02.248436+0000","last_fullsized":"2023-10-30T13:59:02.248436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:12.964999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065387930000000002,"stat_sum":{"num_bytes":14536361,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":108,"num_read_kb":8117,"num_write":641,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"56'695","reported_seq":764,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382522+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:58:51.382522+0000","last_peered":"2023-10-30T13:58:51.382522+0000","last_clean":"2023-10-30T13:58:51.382522+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:58:51.382522+0000","last_undegraded":"2023-10-30T13:58:51.382522+0000","last_fullsized":"2023-10-30T13:58:51.382522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:41.053695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45057,"num_write":652,"num_write_kb":16749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"56'665","reported_seq":2885,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.488063+0000","last_change":"2023-10-30T13:45:27.941002+0000","last_active":"2023-10-30T13:59:02.488063+0000","last_peered":"2023-10-30T13:59:02.488063+0000","last_clean":"2023-10-30T13:59:02.488063+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:59:02.488063+0000","last_undegraded":"2023-10-30T13:59:02.488063+0000","last_fullsized":"2023-10-30T13:59:02.488063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:33.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031306289999999998,"stat_sum":{"num_bytes":22886210,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":127,"num_read_kb":6230,"num_write":644,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"56'743","reported_seq":874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.377388+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:58:51.377388+0000","last_peered":"2023-10-30T13:58:51.377388+0000","last_clean":"2023-10-30T13:58:51.377388+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:58:51.377388+0000","last_undegraded":"2023-10-30T13:58:51.377388+0000","last_fullsized":"2023-10-30T13:58:51.377388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21047876,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":114393,"num_write":693,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"56'141","reported_seq":249,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.571367+0000","last_change":"2023-10-30T13:35:07.293467+0000","last_active":"2023-10-30T13:58:52.571367+0000","last_peered":"2023-10-30T13:58:52.571367+0000","last_clean":"2023-10-30T13:58:52.571367+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:58:52.571367+0000","last_undegraded":"2023-10-30T13:58:52.571367+0000","last_fullsized":"2023-10-30T13:58:52.571367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:47.405473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297526,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":20619,"num_write":102,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"56'752","reported_seq":2988,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.668049+0000","last_change":"2023-10-30T13:45:28.781311+0000","last_active":"2023-10-30T13:58:47.668049+0000","last_peered":"2023-10-30T13:58:47.668049+0000","last_clean":"2023-10-30T13:58:47.668049+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:58:47.668049+0000","last_undegraded":"2023-10-30T13:58:47.668049+0000","last_fullsized":"2023-10-30T13:58:47.668049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:42.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027758349999999999,"stat_sum":{"num_bytes":1481853,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":144,"num_read_kb":4120,"num_write":721,"num_write_kb":40298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"56'594","reported_seq":676,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:31.559442+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:58:31.559442+0000","last_peered":"2023-10-30T13:58:31.559442+0000","last_clean":"2023-10-30T13:58:31.559442+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:58:31.559442+0000","last_undegraded":"2023-10-30T13:58:31.559442+0000","last_fullsized":"2023-10-30T13:58:31.559442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":96432,"num_write":540,"num_write_kb":20769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"56'616","reported_seq":2823,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.628582+0000","last_change":"2023-10-30T13:45:27.934297+0000","last_active":"2023-10-30T13:58:47.628582+0000","last_peered":"2023-10-30T13:58:47.628582+0000","last_clean":"2023-10-30T13:58:47.628582+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:58:47.628582+0000","last_undegraded":"2023-10-30T13:58:47.628582+0000","last_fullsized":"2023-10-30T13:58:47.628582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:48.732244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017556029999999999,"stat_sum":{"num_bytes":5986884,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":115,"num_read_kb":8211,"num_write":580,"num_write_kb":32401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"56'127","reported_seq":186,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382513+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:58:51.382513+0000","last_peered":"2023-10-30T13:58:51.382513+0000","last_clean":"2023-10-30T13:58:51.382513+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:58:51.382513+0000","last_undegraded":"2023-10-30T13:58:51.382513+0000","last_fullsized":"2023-10-30T13:58:51.382513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":91,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"56'666","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.619983+0000","last_change":"2023-10-30T13:45:28.487297+0000","last_active":"2023-10-30T13:58:47.619983+0000","last_peered":"2023-10-30T13:58:47.619983+0000","last_clean":"2023-10-30T13:58:47.619983+0000","last_became_active":"2023-10-30T13:35:49.007835+0000","last_became_peered":"2023-10-30T13:35:49.007835+0000","last_unstale":"2023-10-30T13:58:47.619983+0000","last_undegraded":"2023-10-30T13:58:47.619983+0000","last_fullsized":"2023-10-30T13:58:47.619983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:17:24.830286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011632000000000001,"stat_sum":{"num_bytes":15463549,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":101,"num_read_kb":5707,"num_write":628,"num_write_kb":26855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"56'384","reported_seq":457,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.335510+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:59:02.335510+0000","last_peered":"2023-10-30T13:59:02.335510+0000","last_clean":"2023-10-30T13:59:02.335510+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:59:02.335510+0000","last_undegraded":"2023-10-30T13:59:02.335510+0000","last_fullsized":"2023-10-30T13:59:02.335510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20383142,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":61443,"num_write":344,"num_write_kb":20149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"56'654","reported_seq":2875,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.418220+0000","last_change":"2023-10-30T13:45:27.941812+0000","last_active":"2023-10-30T13:59:02.418220+0000","last_peered":"2023-10-30T13:59:02.418220+0000","last_clean":"2023-10-30T13:59:02.418220+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:59:02.418220+0000","last_undegraded":"2023-10-30T13:59:02.418220+0000","last_fullsized":"2023-10-30T13:59:02.418220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:17.717541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028607060000000002,"stat_sum":{"num_bytes":18966683,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":128,"num_read_kb":11963,"num_write":623,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"56'713","reported_seq":798,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.576749+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:58:52.576749+0000","last_peered":"2023-10-30T13:58:52.576749+0000","last_clean":"2023-10-30T13:58:52.576749+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:58:52.576749+0000","last_undegraded":"2023-10-30T13:58:52.576749+0000","last_fullsized":"2023-10-30T13:58:52.576749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:40.279245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":49185,"num_write":677,"num_write_kb":16860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"56'662","reported_seq":2855,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.355115+0000","last_change":"2023-10-30T13:45:27.947116+0000","last_active":"2023-10-30T13:59:02.355115+0000","last_peered":"2023-10-30T13:59:02.355115+0000","last_clean":"2023-10-30T13:59:02.355115+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:59:02.355115+0000","last_undegraded":"2023-10-30T13:59:02.355115+0000","last_fullsized":"2023-10-30T13:59:02.355115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:44.472162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070741379999999998,"stat_sum":{"num_bytes":9996128,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":99,"num_read_kb":8209,"num_write":632,"num_write_kb":23925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"56'1127","reported_seq":1231,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.351736+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:58:56.351736+0000","last_peered":"2023-10-30T13:58:56.351736+0000","last_clean":"2023-10-30T13:58:56.351736+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:58:56.351736+0000","last_undegraded":"2023-10-30T13:58:56.351736+0000","last_fullsized":"2023-10-30T13:58:56.351736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1127,"log_dups_size":0,"ondisk_log_size":1127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":65,"num_read_kb":86059,"num_write":1076,"num_write_kb":17029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"56'645","reported_seq":2842,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.611995+0000","last_change":"2023-10-30T13:45:28.596691+0000","last_active":"2023-10-30T13:58:47.611995+0000","last_peered":"2023-10-30T13:58:47.611995+0000","last_clean":"2023-10-30T13:58:47.611995+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:58:47.611995+0000","last_undegraded":"2023-10-30T13:58:47.611995+0000","last_fullsized":"2023-10-30T13:58:47.611995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:49:41.351825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070810969999999997,"stat_sum":{"num_bytes":18060472,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":104,"num_read_kb":12299,"num_write":619,"num_write_kb":31647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"56'430","reported_seq":527,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:46.388525+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:58:46.388525+0000","last_peered":"2023-10-30T13:58:46.388525+0000","last_clean":"2023-10-30T13:58:46.388525+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:58:46.388525+0000","last_undegraded":"2023-10-30T13:58:46.388525+0000","last_fullsized":"2023-10-30T13:58:46.388525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:13.193224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":71,"num_read_kb":77906,"num_write":383,"num_write_kb":12555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"56'673","reported_seq":2900,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.616338+0000","last_change":"2023-10-30T13:45:28.487276+0000","last_active":"2023-10-30T13:58:47.616338+0000","last_peered":"2023-10-30T13:58:47.616338+0000","last_clean":"2023-10-30T13:58:47.616338+0000","last_became_active":"2023-10-30T13:35:49.010460+0000","last_became_peered":"2023-10-30T13:35:49.010460+0000","last_unstale":"2023-10-30T13:58:47.616338+0000","last_undegraded":"2023-10-30T13:58:47.616338+0000","last_fullsized":"2023-10-30T13:58:47.616338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:47:08.712606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001187553,"stat_sum":{"num_bytes":13551160,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":138,"num_read_kb":8210,"num_write":640,"num_write_kb":55768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"56'177","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.380435+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:58:51.380435+0000","last_peered":"2023-10-30T13:58:51.380435+0000","last_clean":"2023-10-30T13:58:51.380435+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:58:51.380435+0000","last_undegraded":"2023-10-30T13:58:51.380435+0000","last_fullsized":"2023-10-30T13:58:51.380435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":632,"num_write":147,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"56'672","reported_seq":2891,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.617238+0000","last_change":"2023-10-30T13:45:28.599507+0000","last_active":"2023-10-30T13:58:47.617238+0000","last_peered":"2023-10-30T13:58:47.617238+0000","last_clean":"2023-10-30T13:58:47.617238+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:58:47.617238+0000","last_undegraded":"2023-10-30T13:58:47.617238+0000","last_fullsized":"2023-10-30T13:58:47.617238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:11.218437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092364379999999996,"stat_sum":{"num_bytes":15694014,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":126,"num_read_kb":8225,"num_write":638,"num_write_kb":34297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"56'171","reported_seq":234,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.363936+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:58:51.363936+0000","last_peered":"2023-10-30T13:58:51.363936+0000","last_clean":"2023-10-30T13:58:51.363936+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:58:51.363936+0000","last_undegraded":"2023-10-30T13:58:51.363936+0000","last_fullsized":"2023-10-30T13:58:51.363936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4103,"num_write":119,"num_write_kb":4220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"56'655","reported_seq":2848,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.372721+0000","last_change":"2023-10-30T13:45:28.049648+0000","last_active":"2023-10-30T13:59:02.372721+0000","last_peered":"2023-10-30T13:59:02.372721+0000","last_clean":"2023-10-30T13:59:02.372721+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:59:02.372721+0000","last_undegraded":"2023-10-30T13:59:02.372721+0000","last_fullsized":"2023-10-30T13:59:02.372721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:03.640232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045350950000000003,"stat_sum":{"num_bytes":14031523,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":99,"num_read_kb":8208,"num_write":613,"num_write_kb":14857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"56'613","reported_seq":722,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.572185+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:58:51.572185+0000","last_peered":"2023-10-30T13:58:51.572185+0000","last_clean":"2023-10-30T13:58:51.572185+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:58:51.572185+0000","last_undegraded":"2023-10-30T13:58:51.572185+0000","last_fullsized":"2023-10-30T13:58:51.572185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:24.462329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":130,"num_read_kb":41105,"num_write":605,"num_write_kb":12685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"56'710","reported_seq":2920,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.607871+0000","last_change":"2023-10-30T13:45:28.598560+0000","last_active":"2023-10-30T13:58:47.607871+0000","last_peered":"2023-10-30T13:58:47.607871+0000","last_clean":"2023-10-30T13:58:47.607871+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:58:47.607871+0000","last_undegraded":"2023-10-30T13:58:47.607871+0000","last_fullsized":"2023-10-30T13:58:47.607871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:25.254829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089036940000000002,"stat_sum":{"num_bytes":5954339,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":119,"num_read_kb":8210,"num_write":672,"num_write_kb":34961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'351","reported_seq":425,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.537629+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:58:32.537629+0000","last_peered":"2023-10-30T13:58:32.537629+0000","last_clean":"2023-10-30T13:58:32.537629+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:58:32.537629+0000","last_undegraded":"2023-10-30T13:58:32.537629+0000","last_fullsized":"2023-10-30T13:58:32.537629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":28724,"num_write":357,"num_write_kb":8477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"56'621","reported_seq":2813,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.600642+0000","last_change":"2023-10-30T13:45:28.600359+0000","last_active":"2023-10-30T13:58:47.600642+0000","last_peered":"2023-10-30T13:58:47.600642+0000","last_clean":"2023-10-30T13:58:47.600642+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:58:47.600642+0000","last_undegraded":"2023-10-30T13:58:47.600642+0000","last_fullsized":"2023-10-30T13:58:47.600642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:40.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094781629999999995,"stat_sum":{"num_bytes":5933515,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":99,"num_read_kb":8205,"num_write":588,"num_write_kb":15719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"56'638","reported_seq":715,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:49.590466+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:58:49.590466+0000","last_peered":"2023-10-30T13:58:49.590466+0000","last_clean":"2023-10-30T13:58:49.590466+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:58:49.590466+0000","last_undegraded":"2023-10-30T13:58:49.590466+0000","last_fullsized":"2023-10-30T13:58:49.590466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:05.297512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33457,"num_write":581,"num_write_kb":12711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"56'639","reported_seq":2843,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.631413+0000","last_change":"2023-10-30T13:45:28.601986+0000","last_active":"2023-10-30T13:58:47.631413+0000","last_peered":"2023-10-30T13:58:47.631413+0000","last_clean":"2023-10-30T13:58:47.631413+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:58:47.631413+0000","last_undegraded":"2023-10-30T13:58:47.631413+0000","last_fullsized":"2023-10-30T13:58:47.631413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:59.052456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010320058,"stat_sum":{"num_bytes":13685218,"num_objects":132,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":111,"num_read_kb":8210,"num_write":603,"num_write_kb":35843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"56'400","reported_seq":474,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.554499+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:58:32.554499+0000","last_peered":"2023-10-30T13:58:32.554499+0000","last_clean":"2023-10-30T13:58:32.554499+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:58:32.554499+0000","last_undegraded":"2023-10-30T13:58:32.554499+0000","last_fullsized":"2023-10-30T13:58:32.554499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:40.806018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582982,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":36872,"num_write":369,"num_write_kb":12516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"56'744","reported_seq":2973,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.369177+0000","last_change":"2023-10-30T13:45:27.948762+0000","last_active":"2023-10-30T13:59:02.369177+0000","last_peered":"2023-10-30T13:59:02.369177+0000","last_clean":"2023-10-30T13:59:02.369177+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:59:02.369177+0000","last_undegraded":"2023-10-30T13:59:02.369177+0000","last_fullsized":"2023-10-30T13:59:02.369177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.722813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066640720000000001,"stat_sum":{"num_bytes":18395347,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":135,"num_read_kb":4114,"num_write":711,"num_write_kb":24326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"56'222","reported_seq":290,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.553986+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:58:32.553986+0000","last_peered":"2023-10-30T13:58:32.553986+0000","last_clean":"2023-10-30T13:58:32.553986+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:58:32.553986+0000","last_undegraded":"2023-10-30T13:58:32.553986+0000","last_fullsized":"2023-10-30T13:58:32.553986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:26.728394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":20494,"num_write":198,"num_write_kb":4363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"56'645","reported_seq":2844,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.612268+0000","last_change":"2023-10-30T13:45:28.598725+0000","last_active":"2023-10-30T13:58:47.612268+0000","last_peered":"2023-10-30T13:58:47.612268+0000","last_clean":"2023-10-30T13:58:47.612268+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:58:47.612268+0000","last_undegraded":"2023-10-30T13:58:47.612268+0000","last_fullsized":"2023-10-30T13:58:47.612268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:29.217863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084474579999999997,"stat_sum":{"num_bytes":17958459,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":106,"num_read_kb":8207,"num_write":607,"num_write_kb":40298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:57:42.172646+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:57:42.172646+0000","last_peered":"2023-10-30T13:57:42.172646+0000","last_clean":"2023-10-30T13:57:42.172646+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:57:42.172646+0000","last_undegraded":"2023-10-30T13:57:42.172646+0000","last_fullsized":"2023-10-30T13:57:42.172646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'346","reported_seq":413,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.460048+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:58:56.460048+0000","last_peered":"2023-10-30T13:58:56.460048+0000","last_clean":"2023-10-30T13:58:56.460048+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:58:56.460048+0000","last_undegraded":"2023-10-30T13:58:56.460048+0000","last_fullsized":"2023-10-30T13:58:56.460048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388674,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":20491,"num_write":340,"num_write_kb":8443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'643","reported_seq":2845,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.646867+0000","last_change":"2023-10-30T13:45:28.780750+0000","last_active":"2023-10-30T13:58:47.646867+0000","last_peered":"2023-10-30T13:58:47.646867+0000","last_clean":"2023-10-30T13:58:47.646867+0000","last_became_active":"2023-10-30T13:35:48.847905+0000","last_became_peered":"2023-10-30T13:35:48.847905+0000","last_unstale":"2023-10-30T13:58:47.646867+0000","last_undegraded":"2023-10-30T13:58:47.646867+0000","last_fullsized":"2023-10-30T13:58:47.646867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:37.243152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002389721,"stat_sum":{"num_bytes":10609565,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":100,"num_read_kb":11,"num_write":618,"num_write_kb":20098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"56'352","reported_seq":439,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.575618+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:58:51.575618+0000","last_peered":"2023-10-30T13:58:51.575618+0000","last_clean":"2023-10-30T13:58:51.575618+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:58:51.575618+0000","last_undegraded":"2023-10-30T13:58:51.575618+0000","last_fullsized":"2023-10-30T13:58:51.575618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":102412,"num_write":306,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"56'696","reported_seq":2908,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.336775+0000","last_change":"2023-10-30T13:45:27.939160+0000","last_active":"2023-10-30T13:59:02.336775+0000","last_peered":"2023-10-30T13:59:02.336775+0000","last_clean":"2023-10-30T13:59:02.336775+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:59:02.336775+0000","last_undegraded":"2023-10-30T13:59:02.336775+0000","last_fullsized":"2023-10-30T13:59:02.336775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:28:16.251173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018610090000000001,"stat_sum":{"num_bytes":24271212,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":117,"num_read_kb":10,"num_write":671,"num_write_kb":33782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"56'1498","reported_seq":2022,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.567569+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:58:52.567569+0000","last_peered":"2023-10-30T13:58:52.567569+0000","last_clean":"2023-10-30T13:58:52.567569+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:58:52.567569+0000","last_undegraded":"2023-10-30T13:58:52.567569+0000","last_fullsized":"2023-10-30T13:58:52.567569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:39:13.178991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959716,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":475,"num_read_kb":12479,"num_write":1460,"num_write_kb":17319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'629","reported_seq":2840,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.636121+0000","last_change":"2023-10-30T13:45:27.934841+0000","last_active":"2023-10-30T13:58:47.636121+0000","last_peered":"2023-10-30T13:58:47.636121+0000","last_clean":"2023-10-30T13:58:47.636121+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:58:47.636121+0000","last_undegraded":"2023-10-30T13:58:47.636121+0000","last_fullsized":"2023-10-30T13:58:47.636121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:35.639553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022228600000000001,"stat_sum":{"num_bytes":9645659,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":116,"num_read_kb":4117,"num_write":587,"num_write_kb":24506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'628","reported_seq":701,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.549494+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:58:32.549494+0000","last_peered":"2023-10-30T13:58:32.549494+0000","last_clean":"2023-10-30T13:58:32.549494+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:58:32.549494+0000","last_undegraded":"2023-10-30T13:58:32.549494+0000","last_fullsized":"2023-10-30T13:58:32.549494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":61440,"num_write":565,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'712","reported_seq":2908,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.318656+0000","last_change":"2023-10-30T13:45:27.945262+0000","last_active":"2023-10-30T13:59:02.318656+0000","last_peered":"2023-10-30T13:59:02.318656+0000","last_clean":"2023-10-30T13:59:02.318656+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:59:02.318656+0000","last_undegraded":"2023-10-30T13:59:02.318656+0000","last_fullsized":"2023-10-30T13:59:02.318656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:31.086225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066434720000000001,"stat_sum":{"num_bytes":26554596,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":101,"num_read_kb":3528,"num_write":689,"num_write_kb":37408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"56'726","reported_seq":791,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.543895+0000","last_change":"2023-10-30T13:35:07.278533+0000","last_active":"2023-10-30T13:58:32.543895+0000","last_peered":"2023-10-30T13:58:32.543895+0000","last_clean":"2023-10-30T13:58:32.543895+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:58:32.543895+0000","last_undegraded":"2023-10-30T13:58:32.543895+0000","last_fullsized":"2023-10-30T13:58:32.543895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:03.154390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":18914,"num_write":694,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"56'695","reported_seq":2898,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.390308+0000","last_change":"2023-10-30T13:45:27.949717+0000","last_active":"2023-10-30T13:59:02.390308+0000","last_peered":"2023-10-30T13:59:02.390308+0000","last_clean":"2023-10-30T13:59:02.390308+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:59:02.390308+0000","last_undegraded":"2023-10-30T13:59:02.390308+0000","last_fullsized":"2023-10-30T13:59:02.390308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:55.918379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064085849999999996,"stat_sum":{"num_bytes":10071990,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":4109,"num_write":659,"num_write_kb":40765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"56'756","reported_seq":826,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.368827+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:58:51.368827+0000","last_peered":"2023-10-30T13:58:51.368827+0000","last_clean":"2023-10-30T13:58:51.368827+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:58:51.368827+0000","last_undegraded":"2023-10-30T13:58:51.368827+0000","last_fullsized":"2023-10-30T13:58:51.368827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":707,"num_write_kb":16664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"56'647","reported_seq":2850,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.626256+0000","last_change":"2023-10-30T13:45:27.934883+0000","last_active":"2023-10-30T13:58:47.626256+0000","last_peered":"2023-10-30T13:58:47.626256+0000","last_clean":"2023-10-30T13:58:47.626256+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:58:47.626256+0000","last_undegraded":"2023-10-30T13:58:47.626256+0000","last_fullsized":"2023-10-30T13:58:47.626256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:16.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001735594,"stat_sum":{"num_bytes":6219710,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":108,"num_read_kb":4114,"num_write":624,"num_write_kb":12697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"56'277","reported_seq":355,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.425915+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:58:56.425915+0000","last_peered":"2023-10-30T13:58:56.425915+0000","last_clean":"2023-10-30T13:58:56.425915+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:58:56.425915+0000","last_undegraded":"2023-10-30T13:58:56.425915+0000","last_fullsized":"2023-10-30T13:58:56.425915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:16.312748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":698,"num_write":513,"num_write_kb":2808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"56'586","reported_seq":2787,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:48.029365+0000","last_change":"2023-10-30T13:45:28.781685+0000","last_active":"2023-10-30T13:58:48.029365+0000","last_peered":"2023-10-30T13:58:48.029365+0000","last_clean":"2023-10-30T13:58:48.029365+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:58:48.029365+0000","last_undegraded":"2023-10-30T13:58:48.029365+0000","last_fullsized":"2023-10-30T13:58:48.029365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:24.835692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002858771,"stat_sum":{"num_bytes":14189536,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":106,"num_read_kb":4110,"num_write":557,"num_write_kb":24121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"56'119","reported_seq":196,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.385695+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:58:51.385695+0000","last_peered":"2023-10-30T13:58:51.385695+0000","last_clean":"2023-10-30T13:58:51.385695+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:58:51.385695+0000","last_undegraded":"2023-10-30T13:58:51.385695+0000","last_fullsized":"2023-10-30T13:58:51.385695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":20522,"num_write":112,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"56'681","reported_seq":2888,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.392109+0000","last_change":"2023-10-30T13:45:27.941313+0000","last_active":"2023-10-30T13:59:02.392109+0000","last_peered":"2023-10-30T13:59:02.392109+0000","last_clean":"2023-10-30T13:59:02.392109+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:59:02.392109+0000","last_undegraded":"2023-10-30T13:59:02.392109+0000","last_fullsized":"2023-10-30T13:59:02.392109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:13.966713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027315170000000001,"stat_sum":{"num_bytes":18657914,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":112,"num_read_kb":8214,"num_write":646,"num_write_kb":32163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"56'144","reported_seq":213,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:41.727964+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:58:41.727964+0000","last_peered":"2023-10-30T13:58:41.727964+0000","last_clean":"2023-10-30T13:58:41.727964+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:58:41.727964+0000","last_undegraded":"2023-10-30T13:58:41.727964+0000","last_fullsized":"2023-10-30T13:58:41.727964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":30625,"num_write":105,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"56'677","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.279925+0000","last_change":"2023-10-30T13:45:27.946293+0000","last_active":"2023-10-30T13:59:02.279925+0000","last_peered":"2023-10-30T13:59:02.279925+0000","last_clean":"2023-10-30T13:59:02.279925+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:59:02.279925+0000","last_undegraded":"2023-10-30T13:59:02.279925+0000","last_fullsized":"2023-10-30T13:59:02.279925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:04.199638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075826330000000001,"stat_sum":{"num_bytes":5755271,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":98,"num_read_kb":7,"num_write":639,"num_write_kb":16186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'941","reported_seq":1007,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382666+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:58:51.382666+0000","last_peered":"2023-10-30T13:58:51.382666+0000","last_clean":"2023-10-30T13:58:51.382666+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:58:51.382666+0000","last_undegraded":"2023-10-30T13:58:51.382666+0000","last_fullsized":"2023-10-30T13:58:51.382666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:26.294394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":20487,"num_write":909,"num_write_kb":8631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"56'696","reported_seq":2903,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.359470+0000","last_change":"2023-10-30T13:45:28.782068+0000","last_active":"2023-10-30T13:58:51.359470+0000","last_peered":"2023-10-30T13:58:51.359470+0000","last_clean":"2023-10-30T13:58:51.359470+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:58:51.359470+0000","last_undegraded":"2023-10-30T13:58:51.359470+0000","last_fullsized":"2023-10-30T13:58:51.359470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:04:56.579303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029726169999999999,"stat_sum":{"num_bytes":19759817,"num_objects":119,"num_object_clones":0,"num_object_copies":238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":112,"num_read_kb":12306,"num_write":660,"num_write_kb":31083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"56'477","reported_seq":549,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:31.537236+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:58:31.537236+0000","last_peered":"2023-10-30T13:58:31.537236+0000","last_clean":"2023-10-30T13:58:31.537236+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:58:31.537236+0000","last_undegraded":"2023-10-30T13:58:31.537236+0000","last_fullsized":"2023-10-30T13:58:31.537236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:15.655883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":53262,"num_write":506,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"56'660","reported_seq":2865,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.640353+0000","last_change":"2023-10-30T13:45:28.781107+0000","last_active":"2023-10-30T13:58:47.640353+0000","last_peered":"2023-10-30T13:58:47.640353+0000","last_clean":"2023-10-30T13:58:47.640353+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:58:47.640353+0000","last_undegraded":"2023-10-30T13:58:47.640353+0000","last_fullsized":"2023-10-30T13:58:47.640353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:36.715896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002600925,"stat_sum":{"num_bytes":9507297,"num_objects":132,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":111,"num_read_kb":4112,"num_write":621,"num_write_kb":24574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"56'1285","reported_seq":1372,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.561985+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:58:32.561985+0000","last_peered":"2023-10-30T13:58:32.561985+0000","last_clean":"2023-10-30T13:58:32.561985+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:58:32.561985+0000","last_undegraded":"2023-10-30T13:58:32.561985+0000","last_fullsized":"2023-10-30T13:58:32.561985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":81960,"num_write":1252,"num_write_kb":21130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"56'708","reported_seq":2911,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.279723+0000","last_change":"2023-10-30T13:45:28.050416+0000","last_active":"2023-10-30T13:59:02.279723+0000","last_peered":"2023-10-30T13:59:02.279723+0000","last_clean":"2023-10-30T13:59:02.279723+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:59:02.279723+0000","last_undegraded":"2023-10-30T13:59:02.279723+0000","last_fullsized":"2023-10-30T13:59:02.279723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:34.848723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051869250000000002,"stat_sum":{"num_bytes":18626842,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":5015,"num_write":667,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"56'654","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.352749+0000","last_change":"2023-10-30T13:45:27.949465+0000","last_active":"2023-10-30T13:59:02.352749+0000","last_peered":"2023-10-30T13:59:02.352749+0000","last_clean":"2023-10-30T13:59:02.352749+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:59:02.352749+0000","last_undegraded":"2023-10-30T13:59:02.352749+0000","last_fullsized":"2023-10-30T13:59:02.352749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:57.261270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061609869999999997,"stat_sum":{"num_bytes":14245271,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":123,"num_read_kb":12313,"num_write":620,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"56'230","reported_seq":302,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.389069+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:58:51.389069+0000","last_peered":"2023-10-30T13:58:51.389069+0000","last_clean":"2023-10-30T13:58:51.389069+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:58:51.389069+0000","last_undegraded":"2023-10-30T13:58:51.389069+0000","last_fullsized":"2023-10-30T13:58:51.389069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":24606,"num_write":205,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"56'700","reported_seq":2905,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.336024+0000","last_change":"2023-10-30T13:45:27.948077+0000","last_active":"2023-10-30T13:59:02.336024+0000","last_peered":"2023-10-30T13:59:02.336024+0000","last_clean":"2023-10-30T13:59:02.336024+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:59:02.336024+0000","last_undegraded":"2023-10-30T13:59:02.336024+0000","last_fullsized":"2023-10-30T13:59:02.336024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:48.941885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061457359999999997,"stat_sum":{"num_bytes":11313833,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":112,"num_read_kb":16,"num_write":670,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"56'950","reported_seq":1044,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.465087+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:58:56.465087+0000","last_peered":"2023-10-30T13:58:56.465087+0000","last_clean":"2023-10-30T13:58:56.465087+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:58:56.465087+0000","last_undegraded":"2023-10-30T13:58:56.465087+0000","last_fullsized":"2023-10-30T13:58:56.465087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":103644,"num_write":908,"num_write_kb":16901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"56'670","reported_seq":2886,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:02.415474+0000","last_change":"2023-10-30T13:45:27.941557+0000","last_active":"2023-10-30T13:59:02.415474+0000","last_peered":"2023-10-30T13:59:02.415474+0000","last_clean":"2023-10-30T13:59:02.415474+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:59:02.415474+0000","last_undegraded":"2023-10-30T13:59:02.415474+0000","last_fullsized":"2023-10-30T13:59:02.415474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:51.049883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029730300000000002,"stat_sum":{"num_bytes":19886092,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":121,"num_read_kb":4113,"num_write":647,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"56'585","reported_seq":664,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:41.720146+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:58:41.720146+0000","last_peered":"2023-10-30T13:58:41.720146+0000","last_clean":"2023-10-30T13:58:41.720146+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:58:41.720146+0000","last_undegraded":"2023-10-30T13:58:41.720146+0000","last_fullsized":"2023-10-30T13:58:41.720146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:57.758990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":72355,"num_write":540,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"56'629","reported_seq":2822,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:47.591316+0000","last_change":"2023-10-30T13:45:28.781420+0000","last_active":"2023-10-30T13:58:47.591316+0000","last_peered":"2023-10-30T13:58:47.591316+0000","last_clean":"2023-10-30T13:58:47.591316+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:58:47.591316+0000","last_undegraded":"2023-10-30T13:58:47.591316+0000","last_fullsized":"2023-10-30T13:58:47.591316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:15.218245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026777319999999999,"stat_sum":{"num_bytes":7610379,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":98,"num_read_kb":4107,"num_write":602,"num_write_kb":17238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"56'356","reported_seq":451,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382987+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:58:51.382987+0000","last_peered":"2023-10-30T13:58:51.382987+0000","last_clean":"2023-10-30T13:58:51.382987+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:58:51.382987+0000","last_undegraded":"2023-10-30T13:58:51.382987+0000","last_fullsized":"2023-10-30T13:58:51.382987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:50.825591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4418509,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":57427,"num_write":339,"num_write_kb":8706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":916305331,"num_objects":9467,"num_object_clones":0,"num_object_copies":18934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9467,"num_whiteouts":0,"num_read":7268,"num_read_kb":400185,"num_write":40670,"num_write_kb":1941038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880854528,"data_stored":1831121052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42732,"ondisk_log_size":42732,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":297795987,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1553,"num_read_kb":1442867,"num_write":16395,"num_write_kb":373220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595755008,"data_stored":595591974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17196,"ondisk_log_size":17196,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509734,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447344,"kb_used_data":339816,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93924496,"statfs":{"total":96636764160,"available":96178683904,"internally_reserved":0,"allocated":347971584,"data_stored":341021827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":110098891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85199999999999998,"15min":0.81200000000000006},"min":{"1min":0.433,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.63},{"interface":"front","average":{"1min":1.1830000000000001,"5min":0.96299999999999997,"15min":0.90600000000000003},"min":{"1min":0.53500000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.3499999999999996,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.0900000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.86199999999999999,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.0289999999999999,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.02,"5min":0.95499999999999996,"15min":0.92300000000000004},"min":{"1min":0.57399999999999995,"5min":0.45200000000000001,"15min":0.33400000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.625},"last":0.74099999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0389999999999999,"15min":0.97299999999999998},"min":{"1min":0.626,"5min":0.51300000000000001,"15min":0.32000000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.877},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.91700000000000004,"15min":0.94499999999999995},"min":{"1min":0.63300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":4.1120000000000001},"last":0.998}]},{"osd":3,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.75600000000000001,"5min":0.55300000000000005,"15min":0.45900000000000002},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":4.2069999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":4.4189999999999996,"5min":1.645,"15min":1.204},"min":{"1min":0.83299999999999996,"5min":0.47199999999999998,"15min":0.44400000000000001},"max":{"1min":59.844999999999999,"5min":59.844999999999999,"15min":59.844999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1080000000000001,"15min":1.089},"min":{"1min":0.85899999999999999,"5min":0.623,"15min":0.38100000000000001},"max":{"1min":1.853,"5min":1.853,"15min":3.1059999999999999},"last":1.329},{"interface":"front","average":{"1min":1.335,"5min":1.3220000000000001,"15min":1.2769999999999999},"min":{"1min":1.137,"5min":0.754,"15min":0.42399999999999999},"max":{"1min":1.907,"5min":1.907,"15min":3.198},"last":1.401}]},{"osd":5,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0049999999999999,"15min":1.093},"min":{"1min":0.65600000000000003,"5min":0.53100000000000003,"15min":0.40300000000000002},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":4.2759999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.228,"5min":1.258,"15min":1.238},"min":{"1min":0.80000000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5329999999999999,"5min":1.8380000000000001,"15min":3.4380000000000002},"last":1.252}]},{"osd":6,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92200000000000004,"15min":1.091},"min":{"1min":0.55400000000000005,"5min":0.48999999999999999,"15min":0.48399999999999999},"max":{"1min":1.3109999999999999,"5min":1.429,"15min":3.9630000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.458,"5min":1.131,"15min":1.216},"min":{"1min":0.84399999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":7.7060000000000004,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.95599999999999996}]}]},{"osd":3,"up_from":15,"seq":64424509733,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410640,"kb_used_data":311432,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93961200,"statfs":{"total":96636764160,"available":96216268800,"internally_reserved":0,"allocated":318906368,"data_stored":311949814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":101579208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":9.3320000000000007},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0369999999999999,"15min":0.72399999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.26100000000000001},"max":{"1min":0.71599999999999997,"5min":38.082999999999998,"15min":38.082999999999998},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.70199999999999996,"15min":0.60699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":11.599,"15min":11.599},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.70899999999999996,"15min":0.66100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":15.715999999999999,"15min":15.715999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.72699999999999998,"15min":0.66700000000000004},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.28999999999999998},"max":{"1min":0.83899999999999997,"5min":11.646000000000001,"15min":12.351000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.755,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.83399999999999996,"5min":15.641999999999999,"15min":15.641999999999999},"last":0.502}]},{"osd":4,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.016,"15min":0.92000000000000004},"min":{"1min":0.59999999999999998,"5min":0.55400000000000005,"15min":0.51600000000000001},"max":{"1min":0.98899999999999999,"5min":15.759,"15min":15.759},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.298,"15min":1.0640000000000001},"min":{"1min":0.58499999999999996,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":38.137999999999998,"15min":38.137999999999998},"last":0.88400000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.4139999999999999,"15min":1.097},"min":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.52600000000000002},"max":{"1min":1.036,"5min":40.808,"15min":40.808},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.0309999999999999,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.55800000000000005,"15min":0.49299999999999999},"max":{"1min":1.0660000000000001,"5min":16.498000000000001,"15min":19.835999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.96499999999999997,"15min":0.91800000000000004},"min":{"1min":0.621,"5min":0.56799999999999995,"15min":0.53500000000000003},"max":{"1min":1.2370000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.998,"15min":0.96099999999999997},"min":{"1min":0.68500000000000005,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":1.091,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1020000000000001,"15min":0.95799999999999996},"min":{"1min":0.72499999999999998,"5min":0.60599999999999998,"15min":0.59999999999999998},"max":{"1min":1.2549999999999999,"5min":16.082999999999998,"15min":16.082999999999998},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.88,"5min":1.5609999999999999,"15min":1.111},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.58199999999999996},"max":{"1min":1.228,"5min":52.902999999999999,"15min":52.902999999999999},"last":0.98299999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232516,"kb_used_data":187452,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94139324,"statfs":{"total":96636764160,"available":96398667776,"internally_reserved":0,"allocated":191950848,"data_stored":187647968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.45700000000000002},"min":{"1min":0.32100000000000001,"5min":0.251,"15min":0.246},"max":{"1min":0.83199999999999996,"5min":3.407,"15min":3.8969999999999998},"last":0.495},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.371,"5min":0.35299999999999998,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":5.6980000000000004,"15min":5.6980000000000004},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51000000000000001,"15min":0.60099999999999998},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68100000000000005,"5min":5.5750000000000002,"15min":36.109000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58899999999999997,"5min":5.4569999999999999,"15min":8.1890000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.63500000000000001,"15min":0.68999999999999995},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.29199999999999998},"max":{"1min":0.73999999999999999,"5min":5.532,"15min":35.890999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66800000000000004,"15min":0.69799999999999995},"min":{"1min":0.48199999999999998,"5min":0.42899999999999999,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":5.2830000000000004,"15min":22.846},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84099999999999997},"min":{"1min":0.67000000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0549999999999999,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.496,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":5.6059999999999999,"15min":7.8220000000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.93999999999999995,"5min":5.4260000000000002,"15min":9.9109999999999996},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.86499999999999999},"min":{"1min":0.70799999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":5.4870000000000001,"15min":5.4870000000000001},"last":0.874}]},{"osd":6,"last update":"Mon Oct 30 13:58:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":1.2030000000000001,"15min":0.95999999999999996},"min":{"1min":0.64100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0960000000000001,"15min":0.97599999999999998},"min":{"1min":0.68300000000000005,"5min":0.60099999999999998,"15min":0.56599999999999995},"max":{"1min":15.975,"5min":15.975,"15min":15.975},"last":0.86299999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.75700000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.169,"5min":5.3899999999999997,"15min":5.3899999999999997},"last":1.036},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.91900000000000004,"15min":0.91600000000000004},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53700000000000003},"max":{"1min":1.0800000000000001,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":0.84199999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509732,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365392,"kb_used_data":257928,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94006448,"statfs":{"total":96636764160,"available":96262602752,"internally_reserved":0,"allocated":264118272,"data_stored":259563065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.437},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.245},"max":{"1min":0.67200000000000004,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55100000000000005,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":0.66700000000000004,"5min":2.5920000000000001,"15min":22.149000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.315},"max":{"1min":0.749,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.443}]},{"osd":3,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.75900000000000001},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":2.8130000000000002,"15min":54.530000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.67800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34100000000000003},"max":{"1min":0.83799999999999997,"5min":2.879,"15min":25.457000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78300000000000003,"15min":0.81299999999999994},"min":{"1min":0.53900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94699999999999995,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.79500000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":1.054,"5min":2.101,"15min":2.101},"last":0.70199999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83699999999999997,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.505},"max":{"1min":1.012,"5min":2.7810000000000001,"15min":11.164999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83599999999999997},"min":{"1min":0.624,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":0.94399999999999995,"5min":2.8460000000000001,"15min":2.8820000000000001},"last":0.79000000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.83699999999999997},"min":{"1min":0.57299999999999995,"5min":0.55600000000000005,"15min":0.497},"max":{"1min":0.95399999999999996,"5min":2.903,"15min":5.9180000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.85299999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":1.9930000000000001,"15min":2.2629999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:58:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.002,"15min":0.96899999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":12.221,"5min":12.221,"15min":12.221},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.373,"5min":1.0209999999999999,"15min":0.94499999999999995},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":12.153,"5min":12.153,"15min":12.153},"last":0.77800000000000002}]}]},{"osd":6,"up_from":14,"seq":60129542439,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333848,"kb_used_data":240272,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94037992,"statfs":{"total":96636764160,"available":96294903808,"internally_reserved":0,"allocated":246038528,"data_stored":240883050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":95812039},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.1240000000000001,"15min":1.042},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.41999999999999998},"max":{"1min":1.0389999999999999,"5min":19.741,"15min":19.741},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.131,"15min":0.99099999999999999},"min":{"1min":0.69599999999999995,"5min":0.55800000000000005,"15min":0.34899999999999998},"max":{"1min":1.3600000000000001,"5min":13.863,"15min":13.863},"last":0.98199999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93700000000000006,"15min":0.86799999999999999},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.46100000000000002},"max":{"1min":1.0669999999999999,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.1819999999999999,"15min":1.0289999999999999},"min":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.46800000000000003},"max":{"1min":1.2929999999999999,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96199999999999997,"15min":0.97699999999999998},"min":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.335,"5min":3.5329999999999999,"15min":3.7709999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.095,"15min":1.083},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.4059999999999999,"5min":3.407,"15min":4.6840000000000002},"last":1.1519999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.97599999999999998,"15min":1.014},"min":{"1min":0.67100000000000004,"5min":0.52800000000000002,"15min":0.46700000000000003},"max":{"1min":1.359,"5min":3.7999999999999998,"15min":4.7629999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.171,"15min":1.0840000000000001},"min":{"1min":0.92900000000000005,"5min":0.59299999999999997,"15min":0.54600000000000004},"max":{"1min":1.4179999999999999,"5min":3.968,"15min":3.968},"last":1.0589999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2290000000000001,"15min":1.208},"min":{"1min":0.79600000000000004,"5min":0.55800000000000005,"15min":0.51000000000000001},"max":{"1min":1.4710000000000001,"5min":3.8900000000000001,"15min":16.204000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1519999999999999,"15min":1.2649999999999999},"min":{"1min":0.63700000000000001,"5min":0.622,"15min":0.42699999999999999},"max":{"1min":1.399,"5min":19.398,"15min":43.198},"last":1.246}]},{"osd":5,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.98499999999999999,"15min":1.0489999999999999},"min":{"1min":0.70899999999999996,"5min":0.47699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98399999999999999,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3129999999999999,"15min":1.3009999999999999},"min":{"1min":1.1879999999999999,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":1.522,"5min":3.7269999999999999,"15min":3.802},"last":1.196}]},{"osd":7,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.833,"15min":1.5880000000000001},"min":{"1min":0.83499999999999996,"5min":0.58899999999999997,"15min":0.55400000000000005},"max":{"1min":1.2709999999999999,"5min":70.335999999999999,"15min":73.228999999999999},"last":1.026},{"interface":"front","average":{"1min":1.079,"5min":1.675,"15min":1.637},"min":{"1min":0.82699999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.381,"5min":42.164999999999999,"15min":73.289000000000001},"last":0.76300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542437,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446168,"kb_used_data":352528,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93925672,"statfs":{"total":96636764160,"available":96179888128,"internally_reserved":0,"allocated":360988672,"data_stored":352973919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.89900000000000002,"15min":0.879},"min":{"1min":0.54600000000000004,"5min":0.46999999999999997,"15min":0.40400000000000003},"max":{"1min":3.9159999999999999,"5min":3.9159999999999999,"15min":5.0599999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.83099999999999996,"15min":0.872},"min":{"1min":0.52900000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0569999999999999,"5min":1.5620000000000001,"15min":3.4849999999999999},"last":1.724}]},{"osd":1,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.92600000000000005,"15min":0.871},"min":{"1min":0.55300000000000005,"5min":0.40600000000000003,"15min":0.33600000000000002},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.85099999999999998,"15min":0.90000000000000002},"min":{"1min":0.50700000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95899999999999996,"15min":0.95599999999999996},"min":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.51100000000000001},"max":{"1min":1.462,"5min":1.897,"15min":5.3710000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1279999999999999,"15min":1.056},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.53200000000000003},"max":{"1min":1.369,"5min":2.081,"15min":2.887},"last":0.70099999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.095,"15min":1.0620000000000001},"min":{"1min":0.82999999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":5.476},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.97599999999999998,"15min":1.0349999999999999},"min":{"1min":0.67200000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1779999999999999,"5min":1.52,"15min":3.8500000000000001},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95599999999999996,"15min":0.98699999999999999},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.38400000000000001},"max":{"1min":1.3360000000000001,"5min":1.847,"15min":4.1710000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.66900000000000004,"5min":0.49199999999999999,"15min":0.41799999999999998},"max":{"1min":1.3009999999999999,"5min":1.7270000000000001,"15min":4.2519999999999998},"last":0.752}]},{"osd":6,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1899999999999999,"15min":1.1739999999999999},"min":{"1min":0.747,"5min":0.65000000000000002,"15min":0.621},"max":{"1min":1.546,"5min":2.177,"15min":5.6029999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1459999999999999,"15min":1.1739999999999999},"min":{"1min":0.69499999999999995,"5min":0.56100000000000005,"15min":0.41899999999999998},"max":{"1min":1.403,"5min":1.839,"15min":4.077},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.147,"15min":1.23},"min":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.45100000000000001},"max":{"1min":1.4650000000000001,"5min":1.7949999999999999,"15min":9.9390000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.2430000000000001,"15min":1.2889999999999999},"min":{"1min":0.97399999999999998,"5min":0.78000000000000003,"15min":0.48499999999999999},"max":{"1min":1.415,"5min":1.907,"15min":4.3399999999999999},"last":0.5}]}]},{"osd":4,"up_from":14,"seq":60129542437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433140,"kb_used_data":351980,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93938700,"statfs":{"total":96636764160,"available":96193228800,"internally_reserved":0,"allocated":360427520,"data_stored":352125363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.91100000000000003,"15min":0.873},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.167,"5min":8.7680000000000007,"15min":8.7680000000000007},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.89000000000000001,"15min":0.86099999999999999},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.125,"5min":6.5990000000000002,"15min":6.5990000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82899999999999996,"15min":0.89700000000000002},"min":{"1min":0.51600000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.236,"5min":1.375,"15min":5.8010000000000002},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.85899999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.262,"5min":1.6970000000000001,"15min":3.0499999999999998},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88500000000000001,"15min":0.91600000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.212,"5min":1.331,"15min":3.323},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.98599999999999999,"15min":0.97099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.252,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.1100000000000001,"15min":1.0209999999999999},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.151,"5min":8.0570000000000004,"15min":8.0570000000000004},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.056,"15min":1.0009999999999999},"min":{"1min":0.65900000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.381,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.018,"15min":1.0600000000000001},"min":{"1min":0.82699999999999996,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.325,"5min":4.0030000000000001,"15min":8.4550000000000001},"last":1.149},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.175,"15min":1.171},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.57699999999999996},"max":{"1min":1.3959999999999999,"5min":4.0609999999999999,"15min":8.5399999999999991},"last":0.92300000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1859999999999999,"15min":1.083},"min":{"1min":0.76300000000000001,"5min":0.503,"15min":0.45100000000000001},"max":{"1min":1.4870000000000001,"5min":16.989999999999998,"15min":16.989999999999998},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.173,"15min":1.149},"min":{"1min":0.86699999999999999,"5min":0.41299999999999998,"15min":0.36699999999999999},"max":{"1min":1.599,"5min":4.4530000000000003,"15min":4.4530000000000003},"last":1.1819999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:42 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1719999999999999,"15min":1.163},"min":{"1min":0.84199999999999997,"5min":0.45100000000000001,"15min":0.42199999999999999},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":3.4510000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.298,"15min":1.3049999999999999},"min":{"1min":1.0269999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.5470000000000002},"last":1.2450000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469860,"kb_used_data":376220,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93901980,"statfs":{"total":96636764160,"available":96155627520,"internally_reserved":0,"allocated":385249280,"data_stored":377198604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.98899999999999999,"15min":0.63900000000000001},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":0.63900000000000001,"5min":45.311,"15min":45.311},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.59599999999999997},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.27200000000000002},"max":{"1min":0.72599999999999998,"5min":2.2410000000000001,"15min":27.616},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.29099999999999998},"max":{"1min":0.82999999999999996,"5min":0.88100000000000001,"15min":2.5449999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.44,"5min":0.40899999999999997,"15min":0.35199999999999998},"max":{"1min":0.752,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.57599999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.67500000000000004,"15min":0.56100000000000005},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":0.79300000000000004,"5min":12.368,"15min":12.368},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.75800000000000001,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.40899999999999997},"max":{"1min":0.83299999999999996,"5min":12.442,"15min":12.442},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84799999999999998,"15min":0.85999999999999999},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.41599999999999998},"max":{"1min":1,"5min":2.6949999999999998,"15min":9.4979999999999993},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.871,"15min":0.86499999999999999},"min":{"1min":0.629,"5min":0.60199999999999998,"15min":0.46700000000000003},"max":{"1min":1.1220000000000001,"5min":2.6440000000000001,"15min":9.5429999999999993},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.88,"15min":0.878},"min":{"1min":0.72199999999999998,"5min":0.47099999999999997,"15min":0.317},"max":{"1min":1.087,"5min":1.163,"15min":11.026},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.375,"15min":1.0309999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.46999999999999997},"max":{"1min":1.1759999999999999,"5min":44.225999999999999,"15min":44.225999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88,"15min":0.873},"min":{"1min":0.73899999999999999,"5min":0.63,"15min":0.48799999999999999},"max":{"1min":1.0649999999999999,"5min":2.5169999999999999,"15min":2.9300000000000002},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.435},"max":{"1min":1.133,"5min":1.236,"15min":11.087999999999999},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.91400000000000003,"15min":0.879},"min":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.50900000000000001},"max":{"1min":1.0740000000000001,"5min":5.6849999999999996,"15min":5.6849999999999996},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.93200000000000005,"15min":0.93300000000000005},"min":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.57199999999999995},"max":{"1min":1.0840000000000001,"5min":2.6030000000000002,"15min":9.5890000000000004},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58961920,"data_stored":58944539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50540544,"data_stored":50514908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75808768,"data_stored":75783932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":88170496,"data_stored":88156866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":87515136,"data_stored":87493416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":70914048,"data_stored":70895130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79933440,"data_stored":79916159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325066752,"data_stored":317352305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179568640,"data_stored":175353072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140771328,"data_stored":136810091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242458624,"data_stored":235842913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":273088512,"data_stored":265087328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":275906560,"data_stored":268118820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":177188864,"data_stored":172364192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266805248,"data_stored":260192331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:59:05.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:59:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f519cd94640 1 Processor -- start 2023-10-30T13:59:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f519cd94640 1 -- start start 2023-10-30T13:59:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f519cd94640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f5198070520 0x7f5198070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f519cd94640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51980c2310 con 0x7f51980cb7d0 2023-10-30T13:59:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f519cd94640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5198156710 con 0x7f5198150830 2023-10-30T13:59:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f519cd94640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519806bc10 con 0x7f5198070520 2023-10-30T13:59:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f518ffff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f51980cb7d0 0x7f51980c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49474/0 (socket says 172.21.15.31:49474) 2023-10-30T13:59:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f518ffff640 1 -- 172.21.15.31:0/2529218628 learned_addr learned my addr 172.21.15.31:0/2529218628 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.572+0000 7f5196d9d640 1 --2- 172.21.15.31:0/2529218628 >> v2:172.21.15.31:3300/0 conn(0x7f5198070520 0x7f5198070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5196d9d640 1 -- 172.21.15.31:0/2529218628 >> v1:172.21.15.33:6789/0 conn(0x7f5198150830 legacy=0x7f519806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5196d9d640 1 -- 172.21.15.31:0/2529218628 >> v1:172.21.15.31:6789/0 conn(0x7f51980cb7d0 legacy=0x7f51980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5196d9d640 1 -- 172.21.15.31:0/2529218628 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51980c4360 con 0x7f5198070520 2023-10-30T13:59:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5195d9b640 1 -- 172.21.15.31:0/2529218628 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658479346 0 0) 0x7f51980c2310 con 0x7f51980cb7d0 2023-10-30T13:59:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5196d9d640 1 --2- 172.21.15.31:0/2529218628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198070520 0x7f5198070990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f5190021340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fbb78ef1b3f7e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5195d9b640 1 -- 172.21.15.31:0/2529218628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51900b1600 con 0x7f5198070520 2023-10-30T13:59:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5195d9b640 1 -- 172.21.15.31:0/2529218628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51900ad050 con 0x7f5198070520 2023-10-30T13:59:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.576+0000 7f5195d9b640 1 -- 172.21.15.31:0/2529218628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51900b1980 con 0x7f5198070520 2023-10-30T13:59:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- 172.21.15.31:0/2529218628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198070520 msgr2=0x7f5198070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 --2- 172.21.15.31:0/2529218628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198070520 0x7f5198070990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f5190021340 comp rx=0 tx=0).stop 2023-10-30T13:59:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- 172.21.15.31:0/2529218628 shutdown_connections 2023-10-30T13:59:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 --2- 172.21.15.31:0/2529218628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198070520 0x7f5198070990 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- 172.21.15.31:0/2529218628 >> 172.21.15.31:0/2529218628 conn(0x7f51980cac70 msgr2=0x7f51980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- 172.21.15.31:0/2529218628 shutdown_connections 2023-10-30T13:59:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- 172.21.15.31:0/2529218628 wait complete. 2023-10-30T13:59:05.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 Processor -- start 2023-10-30T13:59:05.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- start start 2023-10-30T13:59:05.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5198070520 0x7f51981f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51980cb7d0 0x7f519815a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198150830 0x7f519815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198152ec0 con 0x7f5198150830 2023-10-30T13:59:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198152760 con 0x7f51980cb7d0 2023-10-30T13:59:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519cd94640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51980c65b0 con 0x7f5198070520 2023-10-30T13:59:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198150830 0x7f519815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198150830 0x7f519815a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35644/0 (socket says 172.21.15.31:35644) 2023-10-30T13:59:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 -- 172.21.15.31:0/1670716416 learned_addr learned my addr 172.21.15.31:0/1670716416 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f518ffff640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51980cb7d0 0x7f519815a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f519659c640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5198070520 0x7f51981f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 -- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5198070520 msgr2=0x7f51981f0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5198070520 0x7f51981f0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 -- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51980cb7d0 msgr2=0x7f519815a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51980cb7d0 0x7f519815a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.580+0000 7f5196d9d640 1 -- 172.21.15.31:0/1670716416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5190009a70 con 0x7f5198150830 2023-10-30T13:59:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f5196d9d640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198150830 0x7f519815a600 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519001e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f518effd640 1 -- 172.21.15.31:0/1670716416 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519000e140 con 0x7f5198150830 2023-10-30T13:59:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f518effd640 1 -- 172.21.15.31:0/1670716416 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51900039a0 con 0x7f5198150830 2023-10-30T13:59:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f518effd640 1 -- 172.21.15.31:0/1670716416 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190003d10 con 0x7f5198150830 2023-10-30T13:59:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51981516c0 con 0x7f5198150830 2023-10-30T13:59:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5198151c80 con 0x7f5198150830 2023-10-30T13:59:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5198066510 con 0x7f5198150830 2023-10-30T13:59:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f518effd640 1 -- 172.21.15.31:0/1670716416 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5190004030 con 0x7f5198150830 2023-10-30T13:59:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f518effd640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51780664d0 0x7f5178068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.584+0000 7f518effd640 1 -- 172.21.15.31:0/1670716416 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f51900aae40 con 0x7f5198150830 2023-10-30T13:59:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.588+0000 7f519659c640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51780664d0 0x7f5178068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:05.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.588+0000 7f519659c640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51780664d0 0x7f5178068990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5188007fd0 tx=0x7f5188009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:05.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.588+0000 7f518effd640 1 -- 172.21.15.31:0/1670716416 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51900f2870 con 0x7f5198150830 2023-10-30T13:59:05.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.748+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f519804e170 con 0x7f5198150830 2023-10-30T13:59:05.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.748+0000 7f518effd640 1 -- 172.21.15.31:0/1670716416 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18184 (secure 0 0 0) 0x7f51900b6020 con 0x7f5198150830 2023-10-30T13:59:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:05.751 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":56,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:57:42.156993+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6836/2267972067":"2023-10-31T13:56:02.130126+0000","172.21.15.33:6836/3928996203":"2023-10-31T13:53:47.073133+0000","172.21.15.33:6834/3928996203":"2023-10-31T13:53:47.073133+0000","172.21.15.31:6837/327248983":"2023-10-31T13:51:22.020336+0000","172.21.15.31:6836/327248983":"2023-10-31T13:51:22.020336+0000","172.21.15.33:6834/2281022644":"2023-10-31T13:57:42.156901+0000","172.21.15.31:6837/2267972067":"2023-10-31T13:56:02.130126+0000","172.21.15.33:6836/2545654386":"2023-10-31T13:36:26.753730+0000","172.21.15.31:6836/1956809670":"2023-10-31T13:39:06.794112+0000","172.21.15.33:6836/15857146":"2023-10-31T13:49:36.991529+0000","172.21.15.31:6837/1956809670":"2023-10-31T13:39:06.794112+0000","172.21.15.33:6835/3402459191":"2023-10-31T13:40:56.840815+0000","172.21.15.33:6836/2281022644":"2023-10-31T13:57:42.156901+0000","172.21.15.33:6836/210349556":"2023-10-31T13:44:26.910290+0000","172.21.15.31:6836/349618961":"2023-10-31T13:47:11.952274+0000","172.21.15.33:6838/2419808617":"2023-10-31T13:43:31.895625+0000","172.21.15.33:6834/2545654386":"2023-10-31T13:36:26.753730+0000","172.21.15.33:6834/15857146":"2023-10-31T13:49:36.991529+0000","172.21.15.33:6839/2419808617":"2023-10-31T13:43:31.895625+0000","172.21.15.33:6837/3402459191":"2023-10-31T13:40:56.840815+0000","172.21.15.33:6834/210349556":"2023-10-31T13:44:26.910290+0000","172.21.15.31:6837/349618961":"2023-10-31T13:47:11.952274+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:59:05.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51780664d0 msgr2=0x7f5178068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51780664d0 0x7f5178068990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5188007fd0 tx=0x7f5188009040 comp rx=0 tx=0).stop 2023-10-30T13:59:05.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198150830 msgr2=0x7f519815a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:05.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198150830 0x7f519815a600 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519001e0e0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 shutdown_connections 2023-10-30T13:59:05.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f51780664d0 0x7f5178068990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198150830 0x7f519815a600 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51980cb7d0 0x7f519815a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 --2- 172.21.15.31:0/1670716416 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5198070520 0x7f51981f0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:05.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 >> 172.21.15.31:0/1670716416 conn(0x7f51980cac70 msgr2=0x7f519806bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:05.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 shutdown_connections 2023-10-30T13:59:05.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:05.752+0000 7f519cd94640 1 -- 172.21.15.31:0/1670716416 wait complete. 2023-10-30T13:59:06.773 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T13:59:06.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:06.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.920+0000 7f72a6446640 1 Processor -- start 2023-10-30T13:59:06.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a6446640 1 -- start start 2023-10-30T13:59:06.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a6446640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f72a0150690 0x7f72a006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:06.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a6446640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f72a00c7770 0x7f72a00c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:06.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a6446640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a00c80e0 con 0x7f72a00c0930 2023-10-30T13:59:06.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a6446640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a0072420 con 0x7f72a00c7770 2023-10-30T13:59:06.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a6446640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a01564d0 con 0x7f72a0150690 2023-10-30T13:59:06.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a49e3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f72a00c7770 0x7f72a00c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:06.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a49e3640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f72a00c7770 0x7f72a00c7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35652/0 (socket says 172.21.15.31:35652) 2023-10-30T13:59:06.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a49e3640 1 -- 172.21.15.31:0/416997927 learned_addr learned my addr 172.21.15.31:0/416997927 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:06.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f729f7fe640 1 --1- 172.21.15.31:0/416997927 >> v1:172.21.15.31:6789/0 conn(0x7f72a00c0930 0x7f72a0156c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49486/0 (socket says 172.21.15.31:49486) 2023-10-30T13:59:06.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f729ffff640 1 --2- 172.21.15.31:0/416997927 >> v2:172.21.15.33:3300/0 conn(0x7f72a0150690 0x7f72a006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:06.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a49e3640 1 -- 172.21.15.31:0/416997927 >> v2:172.21.15.33:3300/0 conn(0x7f72a0150690 msgr2=0x7f72a006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:06.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a49e3640 1 --2- 172.21.15.31:0/416997927 >> v2:172.21.15.33:3300/0 conn(0x7f72a0150690 0x7f72a006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:06.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a49e3640 1 -- 172.21.15.31:0/416997927 >> v1:172.21.15.31:6789/0 conn(0x7f72a00c0930 legacy=0x7f72a0156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:06.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f729effd640 1 -- 172.21.15.31:0/416997927 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329945627 0 0) 0x7f72a00c80e0 con 0x7f72a00c0930 2023-10-30T13:59:06.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.924+0000 7f72a49e3640 1 -- 172.21.15.31:0/416997927 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a00c74d0 con 0x7f72a00c7770 2023-10-30T13:59:06.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a49e3640 1 --2- 172.21.15.31:0/416997927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a00c7770 0x7f72a00c7ba0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f72a006c730 tx=0x7f72980213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fef81862a845b53 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f729effd640 1 -- 172.21.15.31:0/416997927 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729800b2d0 con 0x7f72a00c7770 2023-10-30T13:59:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f729effd640 1 -- 172.21.15.31:0/416997927 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72980ad050 con 0x7f72a00c7770 2023-10-30T13:59:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f729effd640 1 -- 172.21.15.31:0/416997927 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298009820 con 0x7f72a00c7770 2023-10-30T13:59:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 -- 172.21.15.31:0/416997927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a00c7770 msgr2=0x7f72a00c7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 --2- 172.21.15.31:0/416997927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a00c7770 0x7f72a00c7ba0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f72a006c730 tx=0x7f72980213c0 comp rx=0 tx=0).stop 2023-10-30T13:59:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 -- 172.21.15.31:0/416997927 shutdown_connections 2023-10-30T13:59:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 --2- 172.21.15.31:0/416997927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a00c7770 0x7f72a00c7ba0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 --2- 172.21.15.31:0/416997927 >> v2:172.21.15.33:3300/0 conn(0x7f72a0150690 0x7f72a006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 -- 172.21.15.31:0/416997927 >> 172.21.15.31:0/416997927 conn(0x7f72a006e150 msgr2=0x7f72a0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 -- 172.21.15.31:0/416997927 shutdown_connections 2023-10-30T13:59:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 -- 172.21.15.31:0/416997927 wait complete. 2023-10-30T13:59:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 Processor -- start 2023-10-30T13:59:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.928+0000 7f72a6446640 1 -- start start 2023-10-30T13:59:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a00c0930 0x7f72a0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f72a00c7770 0x7f72a01edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a0150690 0x7f72a014dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a01df900 con 0x7f72a0150690 2023-10-30T13:59:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a01e01a0 con 0x7f72a00c0930 2023-10-30T13:59:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a01e0630 con 0x7f72a00c7770 2023-10-30T13:59:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a0150690 0x7f72a014dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a0150690 0x7f72a014dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35666/0 (socket says 172.21.15.31:35666) 2023-10-30T13:59:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 -- 172.21.15.31:0/2496721955 learned_addr learned my addr 172.21.15.31:0/2496721955 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f729ffff640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a00c0930 0x7f72a0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:06.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f729f7fe640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f72a00c7770 0x7f72a01edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:06.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 -- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f72a00c7770 msgr2=0x7f72a01edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:06.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f72a00c7770 0x7f72a01edf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:06.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 -- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a00c0930 msgr2=0x7f72a0156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:06.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a00c0930 0x7f72a0156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:06.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 -- 172.21.15.31:0/2496721955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729800bd40 con 0x7f72a0150690 2023-10-30T13:59:06.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f729ffff640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a00c0930 0x7f72a0156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:06.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a49e3640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a0150690 0x7f72a014dec0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f72980b4040 tx=0x7f729801e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:06.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980c2070 con 0x7f72a0150690 2023-10-30T13:59:06.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7298009890 con 0x7f72a0150690 2023-10-30T13:59:06.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298009c20 con 0x7f72a0150690 2023-10-30T13:59:06.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a014fa50 con 0x7f72a0150690 2023-10-30T13:59:06.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a01dfb50 con 0x7f72a0150690 2023-10-30T13:59:06.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.932+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72a0071b80 con 0x7f72a0150690 2023-10-30T13:59:06.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72980b8040 con 0x7f72a0150690 2023-10-30T13:59:06.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729cff9640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f728c0664a0 0x7f728c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:06.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f7298124960 con 0x7f72a0150690 2023-10-30T13:59:06.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729cff9640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f728c06f060 0x7f728c071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:06.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 --> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f728c071bd0 con 0x7f728c06f060 2023-10-30T13:59:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7298124cd0 con 0x7f72a0150690 2023-10-30T13:59:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729ffff640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f728c0664a0 0x7f728c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729f7fe640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f728c06f060 0x7f728c071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729f7fe640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f728c06f060 0x7f728c071500 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:06.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729ffff640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f728c0664a0 0x7f728c068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f729000b9f0 tx=0x7f7290008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:06.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.936+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== osd.0 v2:172.21.15.31:6800/1469638853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f728c071bd0 con 0x7f728c06f060 2023-10-30T13:59:06.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.948+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 --> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f72a00718a0 con 0x7f728c06f060 2023-10-30T13:59:06.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.948+0000 7f729cff9640 1 -- 172.21.15.31:0/2496721955 <== osd.0 v2:172.21.15.31:6800/1469638853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f72940071f0 con 0x7f728c06f060 2023-10-30T13:59:06.953 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:06.953 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:06.953 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:06.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f728c06f060 msgr2=0x7f728c071500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:06.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:6800/1469638853,v1:172.21.15.31:6801/1469638853] conn(0x7f728c06f060 0x7f728c071500 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:06.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f728c0664a0 msgr2=0x7f728c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:06.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f728c0664a0 0x7f728c068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f729000b9f0 tx=0x7f7290008040 comp rx=0 tx=0).stop 2023-10-30T13:59:06.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a0150690 msgr2=0x7f72a014dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 --2- 172.21.15.31:0/2496721955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72a0150690 0x7f72a014dec0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f72980b4040 tx=0x7f729801e160 comp rx=0 tx=0).stop 2023-10-30T13:59:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a49e3640 1 -- 172.21.15.31:0/2496721955 reap_dead start 2023-10-30T13:59:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 shutdown_connections 2023-10-30T13:59:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 >> 172.21.15.31:0/2496721955 conn(0x7f72a006e150 msgr2=0x7f72a00c6150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:06.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 shutdown_connections 2023-10-30T13:59:06.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:06.952+0000 7f72a6446640 1 -- 172.21.15.31:0/2496721955 wait complete. 2023-10-30T13:59:06.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T13:59:07.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.112+0000 7fce07068640 1 Processor -- start 2023-10-30T13:59:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.112+0000 7fce07068640 1 -- start start 2023-10-30T13:59:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce07068640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fce00151840 0x7fce000c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce07068640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce07068640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fce000cb8b0 0x7fce00072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce07068640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce000c5030 con 0x7fce000cb8b0 2023-10-30T13:59:07.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce07068640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce00157830 con 0x7fce00151840 2023-10-30T13:59:07.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce07068640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce00157b10 con 0x7fce00151c20 2023-10-30T13:59:07.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fce000cb8b0 0x7fce00072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fce000cb8b0 0x7fce00072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35678/0 (socket says 172.21.15.31:35678) 2023-10-30T13:59:07.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 -- 172.21.15.31:0/2354771876 learned_addr learned my addr 172.21.15.31:0/2354771876 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:07.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fcdfffff640 1 --2- 172.21.15.31:0/2354771876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce04e04640 1 --2- 172.21.15.31:0/2354771876 >> v2:172.21.15.33:3300/0 conn(0x7fce00151840 0x7fce000c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 -- 172.21.15.31:0/2354771876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 msgr2=0x7fce00072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:07.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 --2- 172.21.15.31:0/2354771876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 -- 172.21.15.31:0/2354771876 >> v2:172.21.15.33:3300/0 conn(0x7fce00151840 msgr2=0x7fce000c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:07.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 --2- 172.21.15.31:0/2354771876 >> v2:172.21.15.33:3300/0 conn(0x7fce00151840 0x7fce000c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 -- 172.21.15.31:0/2354771876 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce00157df0 con 0x7fce000cb8b0 2023-10-30T13:59:07.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.116+0000 7fce05605640 1 --2- 172.21.15.31:0/2354771876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce000cb8b0 0x7fce00072540 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fce00072b40 tx=0x7fcdf801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954a094b8a2ab700 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fcdff7fe640 1 -- 172.21.15.31:0/2354771876 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf80af400 con 0x7fce000cb8b0 2023-10-30T13:59:07.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fcdff7fe640 1 -- 172.21.15.31:0/2354771876 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf80ab050 con 0x7fce000cb8b0 2023-10-30T13:59:07.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fcdff7fe640 1 -- 172.21.15.31:0/2354771876 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf80af780 con 0x7fce000cb8b0 2023-10-30T13:59:07.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- 172.21.15.31:0/2354771876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce000cb8b0 msgr2=0x7fce00072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 --2- 172.21.15.31:0/2354771876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce000cb8b0 0x7fce00072540 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fce00072b40 tx=0x7fcdf801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- 172.21.15.31:0/2354771876 shutdown_connections 2023-10-30T13:59:07.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 --2- 172.21.15.31:0/2354771876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce000cb8b0 0x7fce00072540 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 --2- 172.21.15.31:0/2354771876 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 --2- 172.21.15.31:0/2354771876 >> v2:172.21.15.33:3300/0 conn(0x7fce00151840 0x7fce000c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- 172.21.15.31:0/2354771876 >> 172.21.15.31:0/2354771876 conn(0x7fce000cac70 msgr2=0x7fce000cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:07.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- 172.21.15.31:0/2354771876 shutdown_connections 2023-10-30T13:59:07.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- 172.21.15.31:0/2354771876 wait complete. 2023-10-30T13:59:07.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 Processor -- start 2023-10-30T13:59:07.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- start start 2023-10-30T13:59:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce000cb8b0 0x7fce001e1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce00151840 0x7fce0014fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000c9060 con 0x7fce00151840 2023-10-30T13:59:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00147460 con 0x7fce000cb8b0 2023-10-30T13:59:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fce07068640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce00145060 con 0x7fce00151c20 2023-10-30T13:59:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.120+0000 7fcdfffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce00151840 0x7fce0014fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce00151840 0x7fce0014fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35692/0 (socket says 172.21.15.31:35692) 2023-10-30T13:59:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 -- 172.21.15.31:0/3756215807 learned_addr learned my addr 172.21.15.31:0/3756215807 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fce04e04640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce000cb8b0 0x7fce001e1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fce05605640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 -- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 msgr2=0x7fce00145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 -- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce000cb8b0 msgr2=0x7fce001e1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce000cb8b0 0x7fce001e1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 -- 172.21.15.31:0/3756215807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8009b30 con 0x7fce00151840 2023-10-30T13:59:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfffff640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce00151840 0x7fce0014fce0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00d7d0 tx=0x7fcdec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfd7fa640 1 -- 172.21.15.31:0/3756215807 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00b6c0 con 0x7fce00151840 2023-10-30T13:59:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfd7fa640 1 -- 172.21.15.31:0/3756215807 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec00b820 con 0x7fce00151840 2023-10-30T13:59:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfd7fa640 1 -- 172.21.15.31:0/3756215807 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00bbb0 con 0x7fce00151840 2023-10-30T13:59:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce00145300 con 0x7fce00151840 2023-10-30T13:59:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce000c9940 con 0x7fce00151840 2023-10-30T13:59:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0004e170 con 0x7fce00151840 2023-10-30T13:59:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfd7fa640 1 -- 172.21.15.31:0/3756215807 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcdec00bd10 con 0x7fce00151840 2023-10-30T13:59:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfd7fa640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde40664a0 0x7fcde4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.124+0000 7fcdfd7fa640 1 -- 172.21.15.31:0/3756215807 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fcdec0846d0 con 0x7fce00151840 2023-10-30T13:59:07.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.128+0000 7fce04e04640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde40664a0 0x7fcde4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.128+0000 7fce04e04640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde40664a0 0x7fcde4068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400ba30 tx=0x7fcdf4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.128+0000 7fcdfd7fa640 1 -- 172.21.15.31:0/3756215807 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcdec051600 con 0x7fce00151840 2023-10-30T13:59:07.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.288+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fce001583b0 con 0x7fcde40664a0 2023-10-30T13:59:07.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fcdfd7fa640 1 -- 172.21.15.31:0/3756215807 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcdf400b620 con 0x7fcde40664a0 2023-10-30T13:59:07.293 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T13:59:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde40664a0 msgr2=0x7fcde4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde40664a0 0x7fcde4068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400ba30 tx=0x7fcdf4008040 comp rx=0 tx=0).stop 2023-10-30T13:59:07.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce00151840 msgr2=0x7fce0014fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce00151840 0x7fce0014fce0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00d7d0 tx=0x7fcdec00dc90 comp rx=0 tx=0).stop 2023-10-30T13:59:07.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 shutdown_connections 2023-10-30T13:59:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fcde40664a0 0x7fcde4068960 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fce00151c20 0x7fce00145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce00151840 0x7fce0014fce0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 --2- 172.21.15.31:0/3756215807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce000cb8b0 0x7fce001e1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 >> 172.21.15.31:0/3756215807 conn(0x7fce000cac70 msgr2=0x7fce001e40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 shutdown_connections 2023-10-30T13:59:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.292+0000 7fce07068640 1 -- 172.21.15.31:0/3756215807 wait complete. 2023-10-30T13:59:07.315 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T13:59:07.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.460+0000 7f4918f0f640 1 Processor -- start 2023-10-30T13:59:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.460+0000 7f4918f0f640 1 -- start start 2023-10-30T13:59:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.460+0000 7f4918f0f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f49140c7e80 0x7f491406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.460+0000 7f4918f0f640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49140c74d0 con 0x7f49140c0930 2023-10-30T13:59:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.460+0000 7f4918f0f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49140718a0 con 0x7f4914150690 2023-10-30T13:59:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4918f0f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4914072420 con 0x7f49140c7e80 2023-10-30T13:59:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f490bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f49140c0930 0x7f49140c7770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49500/0 (socket says 172.21.15.31:49500) 2023-10-30T13:59:07.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f490bfff640 1 -- 172.21.15.31:0/1441567447 learned_addr learned my addr 172.21.15.31:0/1441567447 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:07.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4912d9d640 1 --2- 172.21.15.31:0/1441567447 >> v2:172.21.15.33:3300/0 conn(0x7f49140c7e80 0x7f491406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566611265 0 0) 0x7f48fc001120 con 0x7f49140c0930 2023-10-30T13:59:07.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4900005740 con 0x7f49140c0930 2023-10-30T13:59:07.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 598542185 0 0) 0x7f4900005740 con 0x7f49140c0930 2023-10-30T13:59:07.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 >> v2:172.21.15.33:3300/0 conn(0x7f49140c7e80 msgr2=0x7f491406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:07.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 --2- 172.21.15.31:0/1441567447 >> v2:172.21.15.33:3300/0 conn(0x7f49140c7e80 0x7f491406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 >> v1:172.21.15.33:6789/0 conn(0x7f4914150690 legacy=0x7f4914156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4914071b80 con 0x7f49140c0930 2023-10-30T13:59:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f48fc001810 con 0x7f49140c0930 2023-10-30T13:59:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 >> v1:172.21.15.31:6789/0 conn(0x7f49140c0930 legacy=0x7f49140c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4900005b90 0x7f4900005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4900006590 0x7f4900006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f490000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4914072420 con 0x7f4900007260 2023-10-30T13:59:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490000e110 con 0x7f4900005b90 2023-10-30T13:59:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490000e2d0 con 0x7f4900006590 2023-10-30T13:59:07.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4912d9d640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f490000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f490bfff640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4900006590 0x7f4900006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f491259c640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4900005b90 0x7f4900005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4912d9d640 1 -- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4900006590 msgr2=0x7f4900006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:07.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4912d9d640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4900006590 0x7f4900006a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4912d9d640 1 -- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4900005b90 msgr2=0x7f4900005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:07.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4912d9d640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4900005b90 0x7f4900005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f4912d9d640 1 -- 172.21.15.31:0/1441567447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4900005740 con 0x7f4900007260 2023-10-30T13:59:07.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.464+0000 7f490bfff640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4900006590 0x7f4900006a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:07.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4912d9d640 1 -- 172.21.15.31:0/1441567447 reap_dead start 2023-10-30T13:59:07.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4912d9d640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f490000dba0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f490c009fc0 tx=0x7f490c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2f93a02a4c8ce1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c0a5cb0 con 0x7f4900007260 2023-10-30T13:59:07.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c00f040 con 0x7f4900007260 2023-10-30T13:59:07.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4911d9b640 1 -- 172.21.15.31:0/1441567447 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c003be0 con 0x7f4900007260 2023-10-30T13:59:07.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 -- 172.21.15.31:0/1441567447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 msgr2=0x7f490000dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f490000dba0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f490c009fc0 tx=0x7f490c01cca0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 -- 172.21.15.31:0/1441567447 shutdown_connections 2023-10-30T13:59:07.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 --2- 172.21.15.31:0/1441567447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f490000dba0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 -- 172.21.15.31:0/1441567447 >> 172.21.15.31:0/1441567447 conn(0x7f491406e150 msgr2=0x7f4914069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:07.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 -- 172.21.15.31:0/1441567447 shutdown_connections 2023-10-30T13:59:07.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 -- 172.21.15.31:0/1441567447 wait complete. 2023-10-30T13:59:07.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 Processor -- start 2023-10-30T13:59:07.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.468+0000 7f4918f0f640 1 -- start start 2023-10-30T13:59:07.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f49140ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49140c1af0 0x7f491414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f49140c4e50 0x7f491414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491414ddc0 con 0x7f4900007260 2023-10-30T13:59:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49140c2350 con 0x7f49140c1af0 2023-10-30T13:59:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49140c2630 con 0x7f49140c4e50 2023-10-30T13:59:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f49140ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f49140ca4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35702/0 (socket says 172.21.15.31:35702) 2023-10-30T13:59:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 -- 172.21.15.31:0/2124914715 learned_addr learned my addr 172.21.15.31:0/2124914715 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f490bfff640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49140c1af0 0x7f491414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4912d9d640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f49140c4e50 0x7f491414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 -- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f49140c4e50 msgr2=0x7f491414fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f49140c4e50 0x7f491414fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 -- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49140c1af0 msgr2=0x7f491414f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49140c1af0 0x7f491414f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 -- 172.21.15.31:0/2124914715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49140c2910 con 0x7f4900007260 2023-10-30T13:59:07.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f491259c640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f49140ca4d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f490400e800 tx=0x7f490400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49040050d0 con 0x7f4900007260 2023-10-30T13:59:07.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904005250 con 0x7f4900007260 2023-10-30T13:59:07.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49140caa10 con 0x7f4900007260 2023-10-30T13:59:07.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490400c6c0 con 0x7f4900007260 2023-10-30T13:59:07.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491414d7f0 con 0x7f4900007260 2023-10-30T13:59:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4914141fc0 con 0x7f4900007260 2023-10-30T13:59:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.472+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f490400c820 con 0x7f4900007260 2023-10-30T13:59:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490affd640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f48f4066520 0x7f48f40689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f4904084090 con 0x7f4900007260 2023-10-30T13:59:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490affd640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f48f406f160 0x7f48f4071600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 --> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48f4071cd0 con 0x7f48f406f160 2023-10-30T13:59:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49040867a0 con 0x7f4900007260 2023-10-30T13:59:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490bfff640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f48f4066520 0x7f48f40689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f4912d9d640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f48f406f160 0x7f48f4071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f4912d9d640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f48f406f160 0x7f48f4071600 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490bfff640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f48f4066520 0x7f48f40689e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f48fc001280 tx=0x7f48fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.476+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== osd.1 v2:172.21.15.31:6802/1190064951 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f48f4071cd0 con 0x7f48f406f160 2023-10-30T13:59:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 --> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4914071e60 con 0x7f48f406f160 2023-10-30T13:59:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f490affd640 1 -- 172.21.15.31:0/2124914715 <== osd.1 v2:172.21.15.31:6802/1190064951 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f490c0a6c60 con 0x7f48f406f160 2023-10-30T13:59:07.492 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:07.492 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:07.493 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:07.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f48f406f160 msgr2=0x7f48f4071600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:6802/1190064951,v1:172.21.15.31:6804/1190064951] conn(0x7f48f406f160 0x7f48f4071600 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f48f4066520 msgr2=0x7f48f40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f48f4066520 0x7f48f40689e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f48fc001280 tx=0x7f48fc009040 comp rx=0 tx=0).stop 2023-10-30T13:59:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 msgr2=0x7f49140ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 --2- 172.21.15.31:0/2124914715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4900007260 0x7f49140ca4d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f490400e800 tx=0x7f490400ecc0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4912d9d640 1 -- 172.21.15.31:0/2124914715 reap_dead start 2023-10-30T13:59:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 shutdown_connections 2023-10-30T13:59:07.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 >> 172.21.15.31:0/2124914715 conn(0x7f491406e150 msgr2=0x7f49140bfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:07.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 shutdown_connections 2023-10-30T13:59:07.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.488+0000 7f4918f0f640 1 -- 172.21.15.31:0/2124914715 wait complete. 2023-10-30T13:59:07.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T13:59:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.652+0000 7f4e77e6b640 1 Processor -- start 2023-10-30T13:59:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.652+0000 7f4e77e6b640 1 -- start start 2023-10-30T13:59:07.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.652+0000 7f4e77e6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e701516c0 0x7f4e700c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.652+0000 7f4e77e6b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4e700c2500 0x7f4e700c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.652+0000 7f4e77e6b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e70151aa0 con 0x7f4e700c82d0 2023-10-30T13:59:07.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.652+0000 7f4e77e6b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e700c74a0 con 0x7f4e700c2500 2023-10-30T13:59:07.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.652+0000 7f4e77e6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700c2e90 con 0x7f4e701516c0 2023-10-30T13:59:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4e700c2500 0x7f4e700c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f4e700c2500 0x7f4e700c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35718/0 (socket says 172.21.15.31:35718) 2023-10-30T13:59:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 -- 172.21.15.31:0/3976651586 learned_addr learned my addr 172.21.15.31:0/3976651586 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e75c07640 1 --2- 172.21.15.31:0/3976651586 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e701516c0 0x7f4e700c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 -- 172.21.15.31:0/3976651586 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e701516c0 msgr2=0x7f4e700c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 --2- 172.21.15.31:0/3976651586 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e701516c0 0x7f4e700c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 -- 172.21.15.31:0/3976651586 >> v1:172.21.15.33:6789/0 conn(0x7f4e700c82d0 legacy=0x7f4e700ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 -- 172.21.15.31:0/3976651586 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70158af0 con 0x7f4e700c2500 2023-10-30T13:59:07.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e75c07640 1 --2- 172.21.15.31:0/3976651586 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e701516c0 0x7f4e700c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e76408640 1 --2- 172.21.15.31:0/3976651586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e700c2950 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009e80 tx=0x7f4e6801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2dedb7c1ebfa0b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e74c05640 1 -- 172.21.15.31:0/3976651586 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680a56c0 con 0x7f4e700c2500 2023-10-30T13:59:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e74c05640 1 -- 172.21.15.31:0/3976651586 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e680a5820 con 0x7f4e700c2500 2023-10-30T13:59:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.656+0000 7f4e74c05640 1 -- 172.21.15.31:0/3976651586 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680a5bb0 con 0x7f4e700c2500 2023-10-30T13:59:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- 172.21.15.31:0/3976651586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 msgr2=0x7f4e700c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/3976651586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e700c2950 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009e80 tx=0x7f4e6801c9f0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- 172.21.15.31:0/3976651586 shutdown_connections 2023-10-30T13:59:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/3976651586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e700c2950 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/3976651586 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e701516c0 0x7f4e700c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- 172.21.15.31:0/3976651586 >> 172.21.15.31:0/3976651586 conn(0x7f4e70071fb0 msgr2=0x7f4e700723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- 172.21.15.31:0/3976651586 shutdown_connections 2023-10-30T13:59:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- 172.21.15.31:0/3976651586 wait complete. 2023-10-30T13:59:07.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 Processor -- start 2023-10-30T13:59:07.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- start start 2023-10-30T13:59:07.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e70145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e700c82d0 0x7f4e701f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e701516c0 0x7f4e70160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e701f23e0 con 0x7f4e700c2500 2023-10-30T13:59:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e701f20e0 con 0x7f4e701516c0 2023-10-30T13:59:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e77e6b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e70160d60 con 0x7f4e700c82d0 2023-10-30T13:59:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e70145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e70145140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35728/0 (socket says 172.21.15.31:35728) 2023-10-30T13:59:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 -- 172.21.15.31:0/1781672746 learned_addr learned my addr 172.21.15.31:0/1781672746 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:07.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75406640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e700c82d0 0x7f4e701f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e76408640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e701516c0 0x7f4e70160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 -- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e700c82d0 msgr2=0x7f4e701f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e700c82d0 0x7f4e701f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 -- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e701516c0 msgr2=0x7f4e70160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e701516c0 0x7f4e70160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.660+0000 7f4e75c07640 1 -- 172.21.15.31:0/1781672746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68009b30 con 0x7f4e700c2500 2023-10-30T13:59:07.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e75c07640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e70145140 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c00e810 tx=0x7f4e6c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e667fc640 1 -- 172.21.15.31:0/1781672746 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c00c8d0 con 0x7f4e700c2500 2023-10-30T13:59:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e667fc640 1 -- 172.21.15.31:0/1781672746 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c00ca30 con 0x7f4e700c2500 2023-10-30T13:59:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e70161000 con 0x7f4e700c2500 2023-10-30T13:59:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e667fc640 1 -- 172.21.15.31:0/1781672746 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c00cdc0 con 0x7f4e700c2500 2023-10-30T13:59:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e70145680 con 0x7f4e700c2500 2023-10-30T13:59:07.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7004e170 con 0x7f4e700c2500 2023-10-30T13:59:07.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e667fc640 1 -- 172.21.15.31:0/1781672746 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e6c013410 con 0x7f4e700c2500 2023-10-30T13:59:07.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e667fc640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4e540664a0 0x7f4e54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:07.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e667fc640 1 -- 172.21.15.31:0/1781672746 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f4e6c0840d0 con 0x7f4e700c2500 2023-10-30T13:59:07.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.664+0000 7f4e75406640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4e540664a0 0x7f4e54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:07.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.668+0000 7f4e75406640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4e540664a0 0x7f4e54068960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4e7006f9e0 tx=0x7f4e60009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:07.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.668+0000 7f4e667fc640 1 -- 172.21.15.31:0/1781672746 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e6c051140 con 0x7f4e700c2500 2023-10-30T13:59:07.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.828+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e70158230 con 0x7f4e540664a0 2023-10-30T13:59:07.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.828+0000 7f4e667fc640 1 -- 172.21.15.31:0/1781672746 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4e60001450 con 0x7f4e540664a0 2023-10-30T13:59:07.832 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T13:59:07.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4e540664a0 msgr2=0x7f4e54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4e540664a0 0x7f4e54068960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4e7006f9e0 tx=0x7f4e60009040 comp rx=0 tx=0).stop 2023-10-30T13:59:07.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 msgr2=0x7f4e70145140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e70145140 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c00e810 tx=0x7f4e6c00ecd0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 shutdown_connections 2023-10-30T13:59:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f4e540664a0 0x7f4e54068960 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e701516c0 0x7f4e70160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4e700c82d0 0x7f4e701f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 --2- 172.21.15.31:0/1781672746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e700c2500 0x7f4e70145140 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 >> 172.21.15.31:0/1781672746 conn(0x7f4e70071fb0 msgr2=0x7f4e70158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 shutdown_connections 2023-10-30T13:59:07.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:07.832+0000 7f4e77e6b640 1 -- 172.21.15.31:0/1781672746 wait complete. 2023-10-30T13:59:07.854 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T13:59:07.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:08.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4478b1d640 1 Processor -- start 2023-10-30T13:59:08.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4478b1d640 1 -- start start 2023-10-30T13:59:08.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4478b1d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f44740c4340 0x7f44741599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4478b1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740ca470 0x7f44740ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4478b1d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447406af00 con 0x7f44740c4720 2023-10-30T13:59:08.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4478b1d640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44740c4cd0 con 0x7f44740c4340 2023-10-30T13:59:08.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4478b1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447406a660 con 0x7f44740ca470 2023-10-30T13:59:08.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f447259c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f44740c4340 0x7f44741599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f447259c640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f44740c4340 0x7f44741599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35740/0 (socket says 172.21.15.31:35740) 2023-10-30T13:59:08.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f447259c640 1 -- 172.21.15.31:0/923555713 learned_addr learned my addr 172.21.15.31:0/923555713 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:08.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f4472d9d640 1 --2- 172.21.15.31:0/923555713 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740ca470 0x7f44740ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f447259c640 1 -- 172.21.15.31:0/923555713 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740ca470 msgr2=0x7f44740ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:08.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f447259c640 1 --2- 172.21.15.31:0/923555713 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740ca470 0x7f44740ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f447259c640 1 -- 172.21.15.31:0/923555713 >> v1:172.21.15.33:6789/0 conn(0x7f44740c4720 legacy=0x7f4474069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:08.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.000+0000 7f447259c640 1 -- 172.21.15.31:0/923555713 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447406a940 con 0x7f44740c4340 2023-10-30T13:59:08.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f447259c640 1 --2- 172.21.15.31:0/923555713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740c4340 0x7f44741599e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f44740c4e10 tx=0x7f446401c160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7034d136f6555a3d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4471d9b640 1 -- 172.21.15.31:0/923555713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44640ab600 con 0x7f44740c4340 2023-10-30T13:59:08.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4471d9b640 1 -- 172.21.15.31:0/923555713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44640a7050 con 0x7f44740c4340 2023-10-30T13:59:08.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4471d9b640 1 -- 172.21.15.31:0/923555713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44640ab980 con 0x7f44740c4340 2023-10-30T13:59:08.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 -- 172.21.15.31:0/923555713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740c4340 msgr2=0x7f44741599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 --2- 172.21.15.31:0/923555713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740c4340 0x7f44741599e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f44740c4e10 tx=0x7f446401c160 comp rx=0 tx=0).stop 2023-10-30T13:59:08.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 -- 172.21.15.31:0/923555713 shutdown_connections 2023-10-30T13:59:08.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 --2- 172.21.15.31:0/923555713 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740ca470 0x7f44740ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 --2- 172.21.15.31:0/923555713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740c4340 0x7f44741599e0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 -- 172.21.15.31:0/923555713 >> 172.21.15.31:0/923555713 conn(0x7f44740cf1c0 msgr2=0x7f44740cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 -- 172.21.15.31:0/923555713 shutdown_connections 2023-10-30T13:59:08.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.004+0000 7f4478b1d640 1 -- 172.21.15.31:0/923555713 wait complete. 2023-10-30T13:59:08.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 Processor -- start 2023-10-30T13:59:08.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 -- start start 2023-10-30T13:59:08.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4340 0x7f4474148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4720 0x7f4474147380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740ca470 0x7f44741595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740c8690 con 0x7f44740ca470 2023-10-30T13:59:08.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740cd280 con 0x7f44740c4720 2023-10-30T13:59:08.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44740c77a0 con 0x7f44740c4340 2023-10-30T13:59:08.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740ca470 0x7f44741595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740ca470 0x7f44741595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35756/0 (socket says 172.21.15.31:35756) 2023-10-30T13:59:08.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 -- 172.21.15.31:0/1711955470 learned_addr learned my addr 172.21.15.31:0/1711955470 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:08.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f447259c640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4340 0x7f4474148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f446bfff640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4720 0x7f4474147380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 -- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4340 msgr2=0x7f4474148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f44740c4340 0x7f4474148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 -- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4720 msgr2=0x7f4474147380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44740c4720 0x7f4474147380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 -- 172.21.15.31:0/1711955470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44640094d0 con 0x7f44740ca470 2023-10-30T13:59:08.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4472d9d640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740ca470 0x7f44741595e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f446c007e40 tx=0x7f446c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c014620 con 0x7f44740ca470 2023-10-30T13:59:08.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c00ee90 con 0x7f44740ca470 2023-10-30T13:59:08.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44740c7a80 con 0x7f44740ca470 2023-10-30T13:59:08.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c0149e0 con 0x7f44740ca470 2023-10-30T13:59:08.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.008+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44740c8040 con 0x7f44740ca470 2023-10-30T13:59:08.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f44740c9f50 con 0x7f44740ca470 2023-10-30T13:59:08.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f446c014b40 con 0x7f44740ca470 2023-10-30T13:59:08.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446affd640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44540664a0 0x7f4454068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f446c084e30 con 0x7f44740ca470 2023-10-30T13:59:08.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446affd640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7f445406f060 0x7f4454071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 --> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4454071bd0 con 0x7f445406f060 2023-10-30T13:59:08.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f446c087410 con 0x7f44740ca470 2023-10-30T13:59:08.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f447259c640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44540664a0 0x7f4454068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446bfff640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7f445406f060 0x7f4454071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f446bfff640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7f445406f060 0x7f4454071500 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.012+0000 7f447259c640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44540664a0 0x7f4454068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f44640ad000 tx=0x7f44640a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.016+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== osd.2 v2:172.21.15.31:6824/3739146666 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4454071bd0 con 0x7f445406f060 2023-10-30T13:59:08.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 --> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f44740725b0 con 0x7f445406f060 2023-10-30T13:59:08.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f446affd640 1 -- 172.21.15.31:0/1711955470 <== osd.2 v2:172.21.15.31:6824/3739146666 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f44740725b0 con 0x7f445406f060 2023-10-30T13:59:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:08.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7f445406f060 msgr2=0x7f4454071500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:6824/3739146666,v1:172.21.15.31:6825/3739146666] conn(0x7f445406f060 0x7f4454071500 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44540664a0 msgr2=0x7f4454068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f44540664a0 0x7f4454068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f44640ad000 tx=0x7f44640a6000 comp rx=0 tx=0).stop 2023-10-30T13:59:08.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740ca470 msgr2=0x7f44741595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 --2- 172.21.15.31:0/1711955470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44740ca470 0x7f44741595e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f446c007e40 tx=0x7f446c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4472d9d640 1 -- 172.21.15.31:0/1711955470 reap_dead start 2023-10-30T13:59:08.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 shutdown_connections 2023-10-30T13:59:08.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 >> 172.21.15.31:0/1711955470 conn(0x7f44740cf1c0 msgr2=0x7f4474069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 shutdown_connections 2023-10-30T13:59:08.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.028+0000 7f4478b1d640 1 -- 172.21.15.31:0/1711955470 wait complete. 2023-10-30T13:59:08.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T13:59:08.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92f780640 1 Processor -- start 2023-10-30T13:59:08.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92f780640 1 -- start start 2023-10-30T13:59:08.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92f780640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fb9280c4a10 0x7fb92815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92f780640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb928159f20 con 0x7fb928153c80 2023-10-30T13:59:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92f780640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92815d620 con 0x7fb9280c42a0 2023-10-30T13:59:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92f780640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92815dec0 con 0x7fb9280c4a10 2023-10-30T13:59:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92d51c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fb928153c80 0x7fb92815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49512/0 (socket says 172.21.15.31:49512) 2023-10-30T13:59:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92d51c640 1 -- 172.21.15.31:0/1570592508 learned_addr learned my addr 172.21.15.31:0/1570592508 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92cd1b640 1 --2- 172.21.15.31:0/1570592508 >> v2:172.21.15.31:3300/0 conn(0x7fb9280c4a10 0x7fb92815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92cd1b640 1 -- 172.21.15.31:0/1570592508 >> v1:172.21.15.33:6789/0 conn(0x7fb9280c42a0 legacy=0x7fb9280ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92cd1b640 1 -- 172.21.15.31:0/1570592508 >> v1:172.21.15.31:6789/0 conn(0x7fb928153c80 legacy=0x7fb92815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.192+0000 7fb92cd1b640 1 -- 172.21.15.31:0/1570592508 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92815d900 con 0x7fb9280c4a10 2023-10-30T13:59:08.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.196+0000 7fb92cd1b640 1 --2- 172.21.15.31:0/1570592508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92815cf90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb9180097c0 tx=0x7fb91801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8ea490f57222d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.196+0000 7fb91ffff640 1 -- 172.21.15.31:0/1570592508 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9180a8970 con 0x7fb9280c4a10 2023-10-30T13:59:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.196+0000 7fb91ffff640 1 -- 172.21.15.31:0/1570592508 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9180a8ad0 con 0x7fb9280c4a10 2023-10-30T13:59:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.196+0000 7fb91ffff640 1 -- 172.21.15.31:0/1570592508 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9180a8e60 con 0x7fb9280c4a10 2023-10-30T13:59:08.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.196+0000 7fb92f780640 1 -- 172.21.15.31:0/1570592508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 msgr2=0x7fb92815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.196+0000 7fb92f780640 1 --2- 172.21.15.31:0/1570592508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92815cf90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb9180097c0 tx=0x7fb91801ec70 comp rx=0 tx=0).stop 2023-10-30T13:59:08.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- 172.21.15.31:0/1570592508 shutdown_connections 2023-10-30T13:59:08.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 --2- 172.21.15.31:0/1570592508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92815cf90 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- 172.21.15.31:0/1570592508 >> 172.21.15.31:0/1570592508 conn(0x7fb9280715c0 msgr2=0x7fb9280719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- 172.21.15.31:0/1570592508 shutdown_connections 2023-10-30T13:59:08.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- 172.21.15.31:0/1570592508 wait complete. 2023-10-30T13:59:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 Processor -- start 2023-10-30T13:59:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- start start 2023-10-30T13:59:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9280c42a0 0x7fb928149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92814a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb928153c80 0x7fb9280b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92815cc80 con 0x7fb9280c4a10 2023-10-30T13:59:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280b3470 con 0x7fb9280c42a0 2023-10-30T13:59:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92f780640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280b3b60 con 0x7fb928153c80 2023-10-30T13:59:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92814a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92814a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35768/0 (socket says 172.21.15.31:35768) 2023-10-30T13:59:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 -- 172.21.15.31:0/2197937013 learned_addr learned my addr 172.21.15.31:0/2197937013 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92d51c640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9280c42a0 0x7fb928149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92dd1d640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb928153c80 0x7fb9280b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 -- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb928153c80 msgr2=0x7fb9280b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb928153c80 0x7fb9280b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 -- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9280c42a0 msgr2=0x7fb928149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9280c42a0 0x7fb928149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 -- 172.21.15.31:0/2197937013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb918009420 con 0x7fb9280c4a10 2023-10-30T13:59:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.200+0000 7fb92cd1b640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92814a070 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb9180b3040 tx=0x7fb91801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb91dffb640 1 -- 172.21.15.31:0/2197937013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9180a8970 con 0x7fb9280c4a10 2023-10-30T13:59:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb91dffb640 1 -- 172.21.15.31:0/2197937013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9180b9410 con 0x7fb9280c4a10 2023-10-30T13:59:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb91dffb640 1 -- 172.21.15.31:0/2197937013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9180b97e0 con 0x7fb9280c4a10 2023-10-30T13:59:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92815d170 con 0x7fb9280c4a10 2023-10-30T13:59:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92815ce90 con 0x7fb9280c4a10 2023-10-30T13:59:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92804e170 con 0x7fb9280c4a10 2023-10-30T13:59:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb91dffb640 1 -- 172.21.15.31:0/2197937013 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9180b7040 con 0x7fb9280c4a10 2023-10-30T13:59:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb91dffb640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb90c0664a0 0x7fb90c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb91dffb640 1 -- 172.21.15.31:0/2197937013 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fb918122090 con 0x7fb9280c4a10 2023-10-30T13:59:08.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.204+0000 7fb92d51c640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb90c0664a0 0x7fb90c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.208+0000 7fb92d51c640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb90c0664a0 0x7fb90c068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb924005890 tx=0x7fb924009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.208+0000 7fb91dffb640 1 -- 172.21.15.31:0/2197937013 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9180ef050 con 0x7fb9280c4a10 2023-10-30T13:59:08.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb928159bb0 con 0x7fb90c0664a0 2023-10-30T13:59:08.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb91dffb640 1 -- 172.21.15.31:0/2197937013 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb924001380 con 0x7fb90c0664a0 2023-10-30T13:59:08.370 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T13:59:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb90c0664a0 msgr2=0x7fb90c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb90c0664a0 0x7fb90c068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb924005890 tx=0x7fb924009040 comp rx=0 tx=0).stop 2023-10-30T13:59:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 msgr2=0x7fb92814a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92814a070 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb9180b3040 tx=0x7fb91801ba10 comp rx=0 tx=0).stop 2023-10-30T13:59:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 shutdown_connections 2023-10-30T13:59:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fb90c0664a0 0x7fb90c068960 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fb928153c80 0x7fb9280b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9280c4a10 0x7fb92814a070 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 --2- 172.21.15.31:0/2197937013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9280c42a0 0x7fb928149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 >> 172.21.15.31:0/2197937013 conn(0x7fb9280715c0 msgr2=0x7fb9280c7720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 shutdown_connections 2023-10-30T13:59:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.368+0000 7fb92f780640 1 -- 172.21.15.31:0/2197937013 wait complete. 2023-10-30T13:59:08.391 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T13:59:08.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:08.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.536+0000 7fa4f3c46640 1 Processor -- start 2023-10-30T13:59:08.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.536+0000 7fa4f3c46640 1 -- start start 2023-10-30T13:59:08.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f3c46640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fa4ec151730 0x7fa4ec0c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f3c46640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa4ec151b10 0x7fa4ec072740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f3c46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 0x7fa4ec0c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f3c46640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec158560 con 0x7fa4ec151b10 2023-10-30T13:59:08.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f3c46640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec072c80 con 0x7fa4ec151730 2023-10-30T13:59:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f3c46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec157620 con 0x7fa4ec0c5fe0 2023-10-30T13:59:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa4ec151b10 0x7fa4ec072740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fa4ec151b10 0x7fa4ec072740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35780/0 (socket says 172.21.15.31:35780) 2023-10-30T13:59:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 -- 172.21.15.31:0/125752070 learned_addr learned my addr 172.21.15.31:0/125752070 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f19e2640 1 --2- 172.21.15.31:0/125752070 >> v2:172.21.15.33:3300/0 conn(0x7fa4ec151730 0x7fa4ec0c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f21e3640 1 --2- 172.21.15.31:0/125752070 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 0x7fa4ec0c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 -- 172.21.15.31:0/125752070 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 msgr2=0x7fa4ec0c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 --2- 172.21.15.31:0/125752070 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 0x7fa4ec0c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 -- 172.21.15.31:0/125752070 >> v2:172.21.15.33:3300/0 conn(0x7fa4ec151730 msgr2=0x7fa4ec0c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 --2- 172.21.15.31:0/125752070 >> v2:172.21.15.33:3300/0 conn(0x7fa4ec151730 0x7fa4ec0c57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 -- 172.21.15.31:0/125752070 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec1578e0 con 0x7fa4ec151b10 2023-10-30T13:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.540+0000 7fa4f11e1640 1 --2- 172.21.15.31:0/125752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec072740 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec06cab0 tx=0x7fa4dc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46b9fd539e93a79 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f09e0640 1 -- 172.21.15.31:0/125752070 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0a89b0 con 0x7fa4ec151b10 2023-10-30T13:59:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f09e0640 1 -- 172.21.15.31:0/125752070 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc0a8b10 con 0x7fa4ec151b10 2023-10-30T13:59:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f09e0640 1 -- 172.21.15.31:0/125752070 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0a8ea0 con 0x7fa4ec151b10 2023-10-30T13:59:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- 172.21.15.31:0/125752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 msgr2=0x7fa4ec072740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 --2- 172.21.15.31:0/125752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec072740 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec06cab0 tx=0x7fa4dc01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- 172.21.15.31:0/125752070 shutdown_connections 2023-10-30T13:59:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 --2- 172.21.15.31:0/125752070 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 0x7fa4ec0c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 --2- 172.21.15.31:0/125752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec072740 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 --2- 172.21.15.31:0/125752070 >> v2:172.21.15.33:3300/0 conn(0x7fa4ec151730 0x7fa4ec0c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- 172.21.15.31:0/125752070 >> 172.21.15.31:0/125752070 conn(0x7fa4ec06e150 msgr2=0x7fa4ec069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- 172.21.15.31:0/125752070 shutdown_connections 2023-10-30T13:59:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- 172.21.15.31:0/125752070 wait complete. 2023-10-30T13:59:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 Processor -- start 2023-10-30T13:59:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- start start 2023-10-30T13:59:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 0x7fa4ec1f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4ec151730 0x7fa4ec1ef530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec13f2a0 con 0x7fa4ec151b10 2023-10-30T13:59:08.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec147c60 con 0x7fa4ec151730 2023-10-30T13:59:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.544+0000 7fa4f3c46640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1f3120 con 0x7fa4ec0c5fe0 2023-10-30T13:59:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec1440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35790/0 (socket says 172.21.15.31:35790) 2023-10-30T13:59:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 -- 172.21.15.31:0/1862356344 learned_addr learned my addr 172.21.15.31:0/1862356344 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f11e1640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4ec151730 0x7fa4ec1ef530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f19e2640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 0x7fa4ec1f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 -- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 msgr2=0x7fa4ec1f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fa4ec0c5fe0 0x7fa4ec1f35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 -- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4ec151730 msgr2=0x7fa4ec1ef530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4ec151730 0x7fa4ec1ef530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 -- 172.21.15.31:0/1862356344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc0094d0 con 0x7fa4ec151b10 2023-10-30T13:59:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f21e3640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec1440c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4007d50 tx=0x7fa4e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4014420 con 0x7fa4ec151b10 2023-10-30T13:59:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e40145a0 con 0x7fa4ec151b10 2023-10-30T13:59:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4014950 con 0x7fa4ec151b10 2023-10-30T13:59:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec1f2b40 con 0x7fa4ec151b10 2023-10-30T13:59:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec0ca7e0 con 0x7fa4ec151b10 2023-10-30T13:59:08.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa4ec157ea0 con 0x7fa4ec151b10 2023-10-30T13:59:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.548+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4e4014ab0 con 0x7fa4ec151b10 2023-10-30T13:59:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4e27fc640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4d00664a0 0x7fa4d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fa4e4084b30 con 0x7fa4ec151b10 2023-10-30T13:59:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4e27fc640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7fa4d006f060 0x7fa4d0071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 --> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4d0071bd0 con 0x7fa4d006f060 2023-10-30T13:59:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4e40892a0 con 0x7fa4ec151b10 2023-10-30T13:59:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4f19e2640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4d00664a0 0x7fa4d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4f11e1640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7fa4d006f060 0x7fa4d0071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4f11e1640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7fa4d006f060 0x7fa4d0071500 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4f19e2640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4d00664a0 0x7fa4d0068960 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80079c0 tx=0x7fa4e8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.552+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== osd.3 v2:172.21.15.31:6816/2725375815 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa4d0071bd0 con 0x7fa4d006f060 2023-10-30T13:59:08.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 --> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa4ec157bc0 con 0x7fa4d006f060 2023-10-30T13:59:08.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4e27fc640 1 -- 172.21.15.31:0/1862356344 <== osd.3 v2:172.21.15.31:6816/2725375815 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa4dc0b0440 con 0x7fa4d006f060 2023-10-30T13:59:08.571 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:08.571 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:08.572 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:08.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7fa4d006f060 msgr2=0x7fa4d0071500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:6816/2725375815,v1:172.21.15.31:6817/2725375815] conn(0x7fa4d006f060 0x7fa4d0071500 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4d00664a0 msgr2=0x7fa4d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fa4d00664a0 0x7fa4d0068960 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80079c0 tx=0x7fa4e8008040 comp rx=0 tx=0).stop 2023-10-30T13:59:08.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 msgr2=0x7fa4ec1440c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 --2- 172.21.15.31:0/1862356344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4ec151b10 0x7fa4ec1440c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4007d50 tx=0x7fa4e400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f21e3640 1 -- 172.21.15.31:0/1862356344 reap_dead start 2023-10-30T13:59:08.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 shutdown_connections 2023-10-30T13:59:08.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 >> 172.21.15.31:0/1862356344 conn(0x7fa4ec06e150 msgr2=0x7fa4ec06bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 shutdown_connections 2023-10-30T13:59:08.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.568+0000 7fa4f3c46640 1 -- 172.21.15.31:0/1862356344 wait complete. 2023-10-30T13:59:08.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T13:59:08.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4172654640 1 Processor -- start 2023-10-30T13:59:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4172654640 1 -- start start 2023-10-30T13:59:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4172654640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f416c06bd10 0x7f416c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4172654640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0c5030 con 0x7f416c151840 2023-10-30T13:59:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4172654640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c157830 con 0x7f416c151c20 2023-10-30T13:59:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4172654640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c157b10 con 0x7f416c06bd10 2023-10-30T13:59:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416bfff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f416c151840 0x7f416c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49514/0 (socket says 172.21.15.31:49514) 2023-10-30T13:59:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416bfff640 1 -- 172.21.15.31:0/994613418 learned_addr learned my addr 172.21.15.31:0/994613418 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4170bf1640 1 --2- 172.21.15.31:0/994613418 >> v2:172.21.15.33:3300/0 conn(0x7f416c06bd10 0x7f416c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245386197 0 0) 0x7f416c0c5030 con 0x7f416c151840 2023-10-30T13:59:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4154003530 con 0x7f416c151840 2023-10-30T13:59:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3693399190 0 0) 0x7f4154003530 con 0x7f416c151840 2023-10-30T13:59:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 >> v2:172.21.15.33:3300/0 conn(0x7f416c06bd10 msgr2=0x7f416c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 --2- 172.21.15.31:0/994613418 >> v2:172.21.15.33:3300/0 conn(0x7f416c06bd10 0x7f416c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 >> v1:172.21.15.33:6789/0 conn(0x7f416c151c20 legacy=0x7f416c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c157df0 con 0x7f416c151840 2023-10-30T13:59:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f415c001680 con 0x7f416c151840 2023-10-30T13:59:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 >> v1:172.21.15.31:6789/0 conn(0x7f416c151840 legacy=0x7f416c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154003980 0x7f4154003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41540047d0 0x7f4154004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4154005190 0x7f415400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c157b10 con 0x7f4154005190 2023-10-30T13:59:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415400c040 con 0x7f41540047d0 2023-10-30T13:59:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415400c200 con 0x7f4154003980 2023-10-30T13:59:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4170bf1640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4154005190 0x7f415400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416b7fe640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41540047d0 0x7f4154004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416bfff640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154003980 0x7f4154003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4170bf1640 1 -- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154003980 msgr2=0x7f4154003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4170bf1640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154003980 0x7f4154003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4170bf1640 1 -- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41540047d0 msgr2=0x7f4154004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4170bf1640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41540047d0 0x7f4154004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f4170bf1640 1 -- 172.21.15.31:0/994613418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154003530 con 0x7f4154005190 2023-10-30T13:59:08.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.736+0000 7f416b7fe640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41540047d0 0x7f4154004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4170bf1640 1 -- 172.21.15.31:0/994613418 reap_dead start 2023-10-30T13:59:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4170bf1640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4154005190 0x7f415400b990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4164009fc0 tx=0x7f416401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469a2ec44e811fac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41640a5cb0 con 0x7f4154005190 2023-10-30T13:59:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416400f040 con 0x7f4154005190 2023-10-30T13:59:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f416affd640 1 -- 172.21.15.31:0/994613418 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4164003be0 con 0x7f4154005190 2023-10-30T13:59:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4172654640 1 -- 172.21.15.31:0/994613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4154005190 msgr2=0x7f415400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4172654640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4154005190 0x7f415400b990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4164009fc0 tx=0x7f416401cca0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4172654640 1 -- 172.21.15.31:0/994613418 shutdown_connections 2023-10-30T13:59:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4172654640 1 --2- 172.21.15.31:0/994613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4154005190 0x7f415400b990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4172654640 1 -- 172.21.15.31:0/994613418 >> 172.21.15.31:0/994613418 conn(0x7f416c0cac70 msgr2=0x7f416c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4172654640 1 -- 172.21.15.31:0/994613418 shutdown_connections 2023-10-30T13:59:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.740+0000 7f4172654640 1 -- 172.21.15.31:0/994613418 wait complete. 2023-10-30T13:59:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 Processor -- start 2023-10-30T13:59:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 -- start start 2023-10-30T13:59:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154005190 0x7f416c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f416c06fdf0 0x7f416c1f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416c0c7850 0x7f416c153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c152030 con 0x7f416c0c7850 2023-10-30T13:59:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c152800 con 0x7f416c06fdf0 2023-10-30T13:59:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c154bf0 con 0x7f4154005190 2023-10-30T13:59:08.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416c0c7850 0x7f416c153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416c0c7850 0x7f416c153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35804/0 (socket says 172.21.15.31:35804) 2023-10-30T13:59:08.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 -- 172.21.15.31:0/3937010618 learned_addr learned my addr 172.21.15.31:0/3937010618 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:08.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f416b7fe640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f416c06fdf0 0x7f416c1f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f416bfff640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154005190 0x7f416c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 -- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154005190 msgr2=0x7f416c0727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154005190 0x7f416c0727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 -- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f416c06fdf0 msgr2=0x7f416c1f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f416c06fdf0 0x7f416c1f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 -- 172.21.15.31:0/3937010618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c154ed0 con 0x7f416c0c7850 2023-10-30T13:59:08.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4170bf1640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416c0c7850 0x7f416c153100 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f416401c710 tx=0x7f4164009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4168ff9640 1 -- 172.21.15.31:0/3937010618 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41640aed60 con 0x7f416c0c7850 2023-10-30T13:59:08.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4168ff9640 1 -- 172.21.15.31:0/3937010618 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41640aeec0 con 0x7f416c0c7850 2023-10-30T13:59:08.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4168ff9640 1 -- 172.21.15.31:0/3937010618 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4164003bc0 con 0x7f416c0c7850 2023-10-30T13:59:08.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c152240 con 0x7f416c0c7850 2023-10-30T13:59:08.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.744+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c0c81f0 con 0x7f416c0c7850 2023-10-30T13:59:08.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.748+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c04e170 con 0x7f416c0c7850 2023-10-30T13:59:08.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.748+0000 7f4168ff9640 1 -- 172.21.15.31:0/3937010618 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4164003d20 con 0x7f416c0c7850 2023-10-30T13:59:08.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.748+0000 7f4168ff9640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41580664d0 0x7f4158068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:08.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.748+0000 7f4168ff9640 1 -- 172.21.15.31:0/3937010618 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f41640a6e40 con 0x7f416c0c7850 2023-10-30T13:59:08.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.748+0000 7f416bfff640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41580664d0 0x7f4158068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:08.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.748+0000 7f416bfff640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41580664d0 0x7f4158068990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f415c001060 tx=0x7f415c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:08.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.752+0000 7f4168ff9640 1 -- 172.21.15.31:0/3937010618 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41640ecfb0 con 0x7f416c0c7850 2023-10-30T13:59:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.908+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f416c1583b0 con 0x7f41580664d0 2023-10-30T13:59:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.908+0000 7f4168ff9640 1 -- 172.21.15.31:0/3937010618 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f415c010860 con 0x7f41580664d0 2023-10-30T13:59:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T13:59:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41580664d0 msgr2=0x7f4158068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41580664d0 0x7f4158068990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f415c001060 tx=0x7f415c009040 comp rx=0 tx=0).stop 2023-10-30T13:59:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416c0c7850 msgr2=0x7f416c153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416c0c7850 0x7f416c153100 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f416401c710 tx=0x7f4164009750 comp rx=0 tx=0).stop 2023-10-30T13:59:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 shutdown_connections 2023-10-30T13:59:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f41580664d0 0x7f4158068990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416c0c7850 0x7f416c153100 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f416c06fdf0 0x7f416c1f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 --2- 172.21.15.31:0/3937010618 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f4154005190 0x7f416c0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 >> 172.21.15.31:0/3937010618 conn(0x7f416c0cac70 msgr2=0x7f416c0cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 shutdown_connections 2023-10-30T13:59:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:08.912+0000 7f4172654640 1 -- 172.21.15.31:0/3937010618 wait complete. 2023-10-30T13:59:08.933 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T13:59:08.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:09.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2611232640 1 Processor -- start 2023-10-30T13:59:09.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2611232640 1 -- start start 2023-10-30T13:59:09.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2611232640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f260c0c4340 0x7f260c1599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2611232640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2611232640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260c06af00 con 0x7f260c0ca470 2023-10-30T13:59:09.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2611232640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f260c0c4cd0 con 0x7f260c0c4340 2023-10-30T13:59:09.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2611232640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c06a3d0 con 0x7f260c0c4720 2023-10-30T13:59:09.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260ad9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f260c0c4340 0x7f260c1599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260ad9d640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f260c0c4340 0x7f260c1599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35816/0 (socket says 172.21.15.31:35816) 2023-10-30T13:59:09.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260ad9d640 1 -- 172.21.15.31:0/1787822509 learned_addr learned my addr 172.21.15.31:0/1787822509 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:09.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260b59e640 1 --1- 172.21.15.31:0/1787822509 >> v1:172.21.15.31:6789/0 conn(0x7f260c0ca470 0x7f260c153880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49518/0 (socket says 172.21.15.31:49518) 2023-10-30T13:59:09.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260a59c640 1 --2- 172.21.15.31:0/1787822509 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260ad9d640 1 -- 172.21.15.31:0/1787822509 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 msgr2=0x7f260c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260ad9d640 1 --2- 172.21.15.31:0/1787822509 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260ad9d640 1 -- 172.21.15.31:0/1787822509 >> v1:172.21.15.31:6789/0 conn(0x7f260c0ca470 legacy=0x7f260c153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260ad9d640 1 -- 172.21.15.31:0/1787822509 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c06a660 con 0x7f260c0c4340 2023-10-30T13:59:09.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f260a59c640 1 --2- 172.21.15.31:0/1787822509 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:09.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.080+0000 7f2609d9b640 1 -- 172.21.15.31:0/1787822509 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365113075 0 0) 0x7f260c06af00 con 0x7f260c0ca470 2023-10-30T13:59:09.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f260ad9d640 1 --2- 172.21.15.31:0/1787822509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0c4340 0x7f260c1599e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009870 tx=0x7f25fc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ccb313e4d26779 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2609d9b640 1 -- 172.21.15.31:0/1787822509 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0ad640 con 0x7f260c0c4340 2023-10-30T13:59:09.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2609d9b640 1 -- 172.21.15.31:0/1787822509 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25fc0ad7a0 con 0x7f260c0c4340 2023-10-30T13:59:09.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2609d9b640 1 -- 172.21.15.31:0/1787822509 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0adb30 con 0x7f260c0c4340 2023-10-30T13:59:09.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 -- 172.21.15.31:0/1787822509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0c4340 msgr2=0x7f260c1599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 --2- 172.21.15.31:0/1787822509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0c4340 0x7f260c1599e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009870 tx=0x7f25fc01c480 comp rx=0 tx=0).stop 2023-10-30T13:59:09.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 -- 172.21.15.31:0/1787822509 shutdown_connections 2023-10-30T13:59:09.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 --2- 172.21.15.31:0/1787822509 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 --2- 172.21.15.31:0/1787822509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0c4340 0x7f260c1599e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 -- 172.21.15.31:0/1787822509 >> 172.21.15.31:0/1787822509 conn(0x7f260c0cf1c0 msgr2=0x7f260c0cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:09.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 -- 172.21.15.31:0/1787822509 shutdown_connections 2023-10-30T13:59:09.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.084+0000 7f2611232640 1 -- 172.21.15.31:0/1787822509 wait complete. 2023-10-30T13:59:09.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 Processor -- start 2023-10-30T13:59:09.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 -- start start 2023-10-30T13:59:09.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f260c0c4340 0x7f260c14caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0ca470 0x7f260c1f5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c1f2d00 con 0x7f260c0ca470 2023-10-30T13:59:09.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c1f54b0 con 0x7f260c0c4340 2023-10-30T13:59:09.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c1f2180 con 0x7f260c0c4720 2023-10-30T13:59:09.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0ca470 0x7f260c1f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0ca470 0x7f260c1f5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35828/0 (socket says 172.21.15.31:35828) 2023-10-30T13:59:09.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 -- 172.21.15.31:0/1398528193 learned_addr learned my addr 172.21.15.31:0/1398528193 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:09.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260a59c640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260ad9d640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f260c0c4340 0x7f260c14caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 -- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 msgr2=0x7f260c150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f260c0c4720 0x7f260c150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 -- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f260c0c4340 msgr2=0x7f260c14caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f260c0c4340 0x7f260c14caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 -- 172.21.15.31:0/1398528193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc0094d0 con 0x7f260c0ca470 2023-10-30T13:59:09.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f260b59e640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0ca470 0x7f260c1f5940 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f260400c780 tx=0x7f260400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260400ce90 con 0x7f260c0ca470 2023-10-30T13:59:09.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260400a5e0 con 0x7f260c0ca470 2023-10-30T13:59:09.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c1f2460 con 0x7f260c0ca470 2023-10-30T13:59:09.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260400a9b0 con 0x7f260c0ca470 2023-10-30T13:59:09.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.088+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c1f2a20 con 0x7f260c0ca470 2023-10-30T13:59:09.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f260c0c9f50 con 0x7f260c0ca470 2023-10-30T13:59:09.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f260400ab10 con 0x7f260c0ca470 2023-10-30T13:59:09.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f25f37fe640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25f80664a0 0x7f25f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f2604083ac0 con 0x7f260c0ca470 2023-10-30T13:59:09.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f25f37fe640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7f25f806eef0 0x7f25f8071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 --> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25f8071a60 con 0x7f25f806eef0 2023-10-30T13:59:09.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26040882a0 con 0x7f260c0ca470 2023-10-30T13:59:09.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f260ad9d640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25f80664a0 0x7f25f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.092+0000 7f260a59c640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7f25f806eef0 0x7f25f8071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.096+0000 7f260a59c640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7f25f806eef0 0x7f25f8071390 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.096+0000 7f260ad9d640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25f80664a0 0x7f25f8068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f25fc0af000 tx=0x7f25fc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.096+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== osd.4 v2:172.21.15.33:6800/465067075 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f25f8071a60 con 0x7f25f806eef0 2023-10-30T13:59:09.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 --> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f260c0725b0 con 0x7f25f806eef0 2023-10-30T13:59:09.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f25f37fe640 1 -- 172.21.15.31:0/1398528193 <== osd.4 v2:172.21.15.33:6800/465067075 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f260c0725b0 con 0x7f25f806eef0 2023-10-30T13:59:09.112 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:09.113 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:09.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7f25f806eef0 msgr2=0x7f25f8071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f2611232640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6800/465067075,v1:172.21.15.33:6801/465067075] conn(0x7f25f806eef0 0x7f25f8071390 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25f80664a0 msgr2=0x7f25f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f2611232640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f25f80664a0 0x7f25f8068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f25fc0af000 tx=0x7f25fc008040 comp rx=0 tx=0).stop 2023-10-30T13:59:09.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0ca470 msgr2=0x7f260c1f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f2611232640 1 --2- 172.21.15.31:0/1398528193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f260c0ca470 0x7f260c1f5940 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f260400c780 tx=0x7f260400cc40 comp rx=0 tx=0).stop 2023-10-30T13:59:09.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.108+0000 7f260b59e640 1 -- 172.21.15.31:0/1398528193 reap_dead start 2023-10-30T13:59:09.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.112+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 shutdown_connections 2023-10-30T13:59:09.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.112+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 >> 172.21.15.31:0/1398528193 conn(0x7f260c0cf1c0 msgr2=0x7f260c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:09.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.112+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 shutdown_connections 2023-10-30T13:59:09.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.112+0000 7f2611232640 1 -- 172.21.15.31:0/1398528193 wait complete. 2023-10-30T13:59:09.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T13:59:09.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 Processor -- start 2023-10-30T13:59:09.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 -- start start 2023-10-30T13:59:09.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed0151840 0x7feed00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7feed0151c20 0x7feed0072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7feed00cb8b0 0x7feed0072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed00c5030 con 0x7feed0151c20 2023-10-30T13:59:09.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed0157830 con 0x7feed00cb8b0 2023-10-30T13:59:09.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.272+0000 7feed4ed6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed0157b10 con 0x7feed0151840 2023-10-30T13:59:09.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7feed0151c20 0x7feed0072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7feed0151c20 0x7feed0072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35830/0 (socket says 172.21.15.31:35830) 2023-10-30T13:59:09.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 -- 172.21.15.31:0/1228039163 learned_addr learned my addr 172.21.15.31:0/1228039163 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:09.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feece59c640 1 --2- 172.21.15.31:0/1228039163 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed0151840 0x7feed00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feeced9d640 1 --2- 172.21.15.31:0/1228039163 >> v2:172.21.15.33:3300/0 conn(0x7feed00cb8b0 0x7feed0072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 -- 172.21.15.31:0/1228039163 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed0151840 msgr2=0x7feed00c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:09.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 --2- 172.21.15.31:0/1228039163 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed0151840 0x7feed00c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 -- 172.21.15.31:0/1228039163 >> v2:172.21.15.33:3300/0 conn(0x7feed00cb8b0 msgr2=0x7feed0072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:09.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 --2- 172.21.15.31:0/1228039163 >> v2:172.21.15.33:3300/0 conn(0x7feed00cb8b0 0x7feed0072540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 -- 172.21.15.31:0/1228039163 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0157df0 con 0x7feed0151c20 2023-10-30T13:59:09.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feec7fff640 1 --2- 172.21.15.31:0/1228039163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151c20 0x7feed0072000 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7feed00cb820 tx=0x7feeb801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fafa195087621d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feecdd9b640 1 -- 172.21.15.31:0/1228039163 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80a89b0 con 0x7feed0151c20 2023-10-30T13:59:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feecdd9b640 1 -- 172.21.15.31:0/1228039163 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb80a8b10 con 0x7feed0151c20 2023-10-30T13:59:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.276+0000 7feecdd9b640 1 -- 172.21.15.31:0/1228039163 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80a8ea0 con 0x7feed0151c20 2023-10-30T13:59:09.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- 172.21.15.31:0/1228039163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151c20 msgr2=0x7feed0072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1228039163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151c20 0x7feed0072000 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7feed00cb820 tx=0x7feeb801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- 172.21.15.31:0/1228039163 shutdown_connections 2023-10-30T13:59:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1228039163 >> v2:172.21.15.33:3300/0 conn(0x7feed00cb8b0 0x7feed0072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1228039163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151c20 0x7feed0072000 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1228039163 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed0151840 0x7feed00c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- 172.21.15.31:0/1228039163 >> 172.21.15.31:0/1228039163 conn(0x7feed00cac70 msgr2=0x7feed00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- 172.21.15.31:0/1228039163 shutdown_connections 2023-10-30T13:59:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- 172.21.15.31:0/1228039163 wait complete. 2023-10-30T13:59:09.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 Processor -- start 2023-10-30T13:59:09.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- start start 2023-10-30T13:59:09.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed00cb8b0 0x7feed01ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151840 0x7feed014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feed0151c20 0x7feed01df080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed01ec5e0 con 0x7feed0151840 2023-10-30T13:59:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed01ed550 con 0x7feed0151c20 2023-10-30T13:59:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feed4ed6640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed014e310 con 0x7feed00cb8b0 2023-10-30T13:59:09.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feec7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151840 0x7feed014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feec7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151840 0x7feed014f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35844/0 (socket says 172.21.15.31:35844) 2023-10-30T13:59:09.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feec7fff640 1 -- 172.21.15.31:0/1966071280 learned_addr learned my addr 172.21.15.31:0/1966071280 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:09.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feece59c640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed00cb8b0 0x7feed01ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.280+0000 7feeced9d640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feed0151c20 0x7feed01df080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec7fff640 1 -- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed00cb8b0 msgr2=0x7feed01ed010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec7fff640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed00cb8b0 0x7feed01ed010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec7fff640 1 -- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feed0151c20 msgr2=0x7feed01df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec7fff640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feed0151c20 0x7feed01df080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec7fff640 1 -- 172.21.15.31:0/1966071280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb80094d0 con 0x7feed0151840 2023-10-30T13:59:09.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec7fff640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151840 0x7feed014f2d0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7feeb801d3a0 tx=0x7feeb801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec6ffd640 1 -- 172.21.15.31:0/1966071280 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80b0d40 con 0x7feed0151840 2023-10-30T13:59:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec6ffd640 1 -- 172.21.15.31:0/1966071280 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb80bb460 con 0x7feed0151840 2023-10-30T13:59:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec6ffd640 1 -- 172.21.15.31:0/1966071280 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80b0ea0 con 0x7feed0151840 2023-10-30T13:59:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed014e5b0 con 0x7feed0151840 2023-10-30T13:59:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed01ec810 con 0x7feed0151840 2023-10-30T13:59:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed004e170 con 0x7feed0151840 2023-10-30T13:59:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.284+0000 7feec6ffd640 1 -- 172.21.15.31:0/1966071280 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feeb80b83f0 con 0x7feed0151840 2023-10-30T13:59:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.288+0000 7feec6ffd640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7feeb00664d0 0x7feeb0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.288+0000 7feec6ffd640 1 -- 172.21.15.31:0/1966071280 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7feeb8121b00 con 0x7feed0151840 2023-10-30T13:59:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.288+0000 7feece59c640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7feeb00664d0 0x7feeb0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.288+0000 7feece59c640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7feeb00664d0 0x7feeb0068990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7feec0009af0 tx=0x7feec0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.292+0000 7feec6ffd640 1 -- 172.21.15.31:0/1966071280 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feeb8123e60 con 0x7feed0151840 2023-10-30T13:59:09.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.456+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7feed01583b0 con 0x7feeb00664d0 2023-10-30T13:59:09.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.456+0000 7feec6ffd640 1 -- 172.21.15.31:0/1966071280 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feec00119b0 con 0x7feeb00664d0 2023-10-30T13:59:09.460 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T13:59:09.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7feeb00664d0 msgr2=0x7feeb0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7feeb00664d0 0x7feeb0068990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7feec0009af0 tx=0x7feec0008040 comp rx=0 tx=0).stop 2023-10-30T13:59:09.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151840 msgr2=0x7feed014f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151840 0x7feed014f2d0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7feeb801d3a0 tx=0x7feeb801ba90 comp rx=0 tx=0).stop 2023-10-30T13:59:09.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 shutdown_connections 2023-10-30T13:59:09.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7feeb00664d0 0x7feeb0068990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feed0151c20 0x7feed01df080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feed0151840 0x7feed014f2d0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 --2- 172.21.15.31:0/1966071280 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7feed00cb8b0 0x7feed01ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 >> 172.21.15.31:0/1966071280 conn(0x7feed00cac70 msgr2=0x7feed0146860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:09.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 shutdown_connections 2023-10-30T13:59:09.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.460+0000 7feed4ed6640 1 -- 172.21.15.31:0/1966071280 wait complete. 2023-10-30T13:59:09.482 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T13:59:09.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.628+0000 7f95d866b640 1 Processor -- start 2023-10-30T13:59:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.628+0000 7f95d866b640 1 -- start start 2023-10-30T13:59:09.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.628+0000 7f95d866b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f95d006bb60 0x7f95d00c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.628+0000 7f95d866b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d0157620 con 0x7f95d0151730 2023-10-30T13:59:09.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d866b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d01578e0 con 0x7f95d0151b10 2023-10-30T13:59:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d866b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d0158560 con 0x7f95d006bb60 2023-10-30T13:59:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d6c08640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f95d006bb60 0x7f95d00c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d6c08640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f95d006bb60 0x7f95d00c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35858/0 (socket says 172.21.15.31:35858) 2023-10-30T13:59:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d6c08640 1 -- 172.21.15.31:0/1227296296 learned_addr learned my addr 172.21.15.31:0/1227296296 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d6c08640 1 -- 172.21.15.31:0/1227296296 >> v1:172.21.15.33:6789/0 conn(0x7f95d0151b10 legacy=0x7f95d006b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d6c08640 1 -- 172.21.15.31:0/1227296296 >> v1:172.21.15.31:6789/0 conn(0x7f95d0151730 legacy=0x7f95d00c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d6c08640 1 -- 172.21.15.31:0/1227296296 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d0157bc0 con 0x7f95d006bb60 2023-10-30T13:59:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d5405640 1 -- 172.21.15.31:0/1227296296 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436079674 0 0) 0x7f95d0157620 con 0x7f95d0151730 2023-10-30T13:59:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d6c08640 1 --2- 172.21.15.31:0/1227296296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d006bb60 0x7f95d00c57e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009e50 tx=0x7f95c801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c2a0ce7a8de420 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d5405640 1 -- 172.21.15.31:0/1227296296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c80af400 con 0x7f95d006bb60 2023-10-30T13:59:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d5405640 1 -- 172.21.15.31:0/1227296296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c80ab050 con 0x7f95d006bb60 2023-10-30T13:59:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.632+0000 7f95d5405640 1 -- 172.21.15.31:0/1227296296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c80af780 con 0x7f95d006bb60 2023-10-30T13:59:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- 172.21.15.31:0/1227296296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d006bb60 msgr2=0x7f95d00c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 --2- 172.21.15.31:0/1227296296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d006bb60 0x7f95d00c57e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009e50 tx=0x7f95c801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- 172.21.15.31:0/1227296296 shutdown_connections 2023-10-30T13:59:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 --2- 172.21.15.31:0/1227296296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d006bb60 0x7f95d00c57e0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- 172.21.15.31:0/1227296296 >> 172.21.15.31:0/1227296296 conn(0x7f95d006e150 msgr2=0x7f95d0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- 172.21.15.31:0/1227296296 shutdown_connections 2023-10-30T13:59:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- 172.21.15.31:0/1227296296 wait complete. 2023-10-30T13:59:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 Processor -- start 2023-10-30T13:59:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- start start 2023-10-30T13:59:09.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95d006bb60 0x7f95d00c1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d0151730 0x7f95d01473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95d0151b10 0x7f95d0147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d01ef100 con 0x7f95d0151730 2023-10-30T13:59:09.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d01eee40 con 0x7f95d0151b10 2023-10-30T13:59:09.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d866b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d014b950 con 0x7f95d006bb60 2023-10-30T13:59:09.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d0151730 0x7f95d01473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d0151730 0x7f95d01473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35866/0 (socket says 172.21.15.31:35866) 2023-10-30T13:59:09.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 -- 172.21.15.31:0/3986142676 learned_addr learned my addr 172.21.15.31:0/3986142676 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:09.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d6407640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95d006bb60 0x7f95d00c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d6c08640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95d0151b10 0x7f95d0147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 -- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95d006bb60 msgr2=0x7f95d00c1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f95d006bb60 0x7f95d00c1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 -- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95d0151b10 msgr2=0x7f95d0147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95d0151b10 0x7f95d0147900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.636+0000 7f95d5c06640 1 -- 172.21.15.31:0/3986142676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c8009b30 con 0x7f95d0151730 2023-10-30T13:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95d5c06640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d0151730 0x7f95d01473c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c500 tx=0x7f95c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c0005220 con 0x7f95d0151730 2023-10-30T13:59:09.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c0005380 con 0x7f95d0151730 2023-10-30T13:59:09.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c0007270 con 0x7f95d0151730 2023-10-30T13:59:09.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d014bc30 con 0x7f95d0151730 2023-10-30T13:59:09.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d014c1f0 con 0x7f95d0151730 2023-10-30T13:59:09.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95d0158180 con 0x7f95d0151730 2023-10-30T13:59:09.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95c00073d0 con 0x7f95d0151730 2023-10-30T13:59:09.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95b40664d0 0x7f95b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f95c0084720 con 0x7f95d0151730 2023-10-30T13:59:09.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f95b406f090 0x7f95b4071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 --> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95b4071c00 con 0x7f95b406f090 2023-10-30T13:59:09.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95c0084b40 con 0x7f95d0151730 2023-10-30T13:59:09.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.640+0000 7f95d6407640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95b40664d0 0x7f95b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.644+0000 7f95d6c08640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f95b406f090 0x7f95b4071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.644+0000 7f95d6407640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95b40664d0 0x7f95b4068990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f95cc000f70 tx=0x7f95cc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.644+0000 7f95d6c08640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f95b406f090 0x7f95b4071530 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.644+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== osd.5 v2:172.21.15.33:6816/3902046602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f95b4071c00 con 0x7f95b406f090 2023-10-30T13:59:09.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.656+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 --> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f95d0157ea0 con 0x7f95b406f090 2023-10-30T13:59:09.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95c6ffd640 1 -- 172.21.15.31:0/3986142676 <== osd.5 v2:172.21.15.33:6816/3902046602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f95c80b9440 con 0x7f95b406f090 2023-10-30T13:59:09.661 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:09.661 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:09.662 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:09.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f95b406f090 msgr2=0x7f95b4071530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6816/3902046602,v1:172.21.15.33:6817/3902046602] conn(0x7f95b406f090 0x7f95b4071530 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95b40664d0 msgr2=0x7f95b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f95b40664d0 0x7f95b4068990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f95cc000f70 tx=0x7f95cc009040 comp rx=0 tx=0).stop 2023-10-30T13:59:09.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d0151730 msgr2=0x7f95d01473c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 --2- 172.21.15.31:0/3986142676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95d0151730 0x7f95d01473c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c500 tx=0x7f95c000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d6c08640 1 -- 172.21.15.31:0/3986142676 reap_dead start 2023-10-30T13:59:09.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 shutdown_connections 2023-10-30T13:59:09.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 >> 172.21.15.31:0/3986142676 conn(0x7f95d006e150 msgr2=0x7f95d0150350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:09.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 shutdown_connections 2023-10-30T13:59:09.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.660+0000 7f95d866b640 1 -- 172.21.15.31:0/3986142676 wait complete. 2023-10-30T13:59:09.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T13:59:09.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f686b640 1 Processor -- start 2023-10-30T13:59:09.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f686b640 1 -- start start 2023-10-30T13:59:09.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f686b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc1f01516c0 0x7fc1f00c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f686b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f00ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f686b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f0151aa0 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f686b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f00c74a0 con 0x7fc1f01516c0 2023-10-30T13:59:09.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f686b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0158510 con 0x7fc1f00c82d0 2023-10-30T13:59:09.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1effff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fc1f01516c0 0x7fc1f00c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.824+0000 7fc1f4e08640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fc1f00cd4a0 0x7fc1f00c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49534/0 (socket says 172.21.15.31:49534) 2023-10-30T13:59:09.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1f4e08640 1 -- 172.21.15.31:0/4234923304 learned_addr learned my addr 172.21.15.31:0/4234923304 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:09.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1ef7fe640 1 --2- 172.21.15.31:0/4234923304 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f00ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1effff640 1 -- 172.21.15.31:0/4234923304 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 msgr2=0x7fc1f00ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:09.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1effff640 1 --2- 172.21.15.31:0/4234923304 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f00ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1effff640 1 -- 172.21.15.31:0/4234923304 >> v1:172.21.15.31:6789/0 conn(0x7fc1f00cd4a0 legacy=0x7fc1f00c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1effff640 1 -- 172.21.15.31:0/4234923304 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f0158af0 con 0x7fc1f01516c0 2023-10-30T13:59:09.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1ef7fe640 1 --2- 172.21.15.31:0/4234923304 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f00ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:09.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1eeffd640 1 -- 172.21.15.31:0/4234923304 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347376040 0 0) 0x7fc1f0151aa0 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1effff640 1 --2- 172.21.15.31:0/4234923304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f01516c0 0x7fc1f00c7d90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00097c0 tx=0x7fc1e001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbfd6a3444d29b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1eeffd640 1 -- 172.21.15.31:0/4234923304 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e00ab600 con 0x7fc1f01516c0 2023-10-30T13:59:09.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1eeffd640 1 -- 172.21.15.31:0/4234923304 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e00a7050 con 0x7fc1f01516c0 2023-10-30T13:59:09.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.828+0000 7fc1eeffd640 1 -- 172.21.15.31:0/4234923304 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e00ab980 con 0x7fc1f01516c0 2023-10-30T13:59:09.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- 172.21.15.31:0/4234923304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f01516c0 msgr2=0x7fc1f00c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 --2- 172.21.15.31:0/4234923304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f01516c0 0x7fc1f00c7d90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00097c0 tx=0x7fc1e001c480 comp rx=0 tx=0).stop 2023-10-30T13:59:09.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- 172.21.15.31:0/4234923304 shutdown_connections 2023-10-30T13:59:09.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 --2- 172.21.15.31:0/4234923304 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f00ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 --2- 172.21.15.31:0/4234923304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f01516c0 0x7fc1f00c7d90 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- 172.21.15.31:0/4234923304 >> 172.21.15.31:0/4234923304 conn(0x7fc1f0071fb0 msgr2=0x7fc1f00723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:09.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- 172.21.15.31:0/4234923304 shutdown_connections 2023-10-30T13:59:09.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- 172.21.15.31:0/4234923304 wait complete. 2023-10-30T13:59:09.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 Processor -- start 2023-10-30T13:59:09.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- start start 2023-10-30T13:59:09.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f0143670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f00cd4a0 0x7fc1f0144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1f01516c0 0x7fc1f015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0144810 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f014bc60 con 0x7fc1f01516c0 2023-10-30T13:59:09.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f686b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f014bf40 con 0x7fc1f00c82d0 2023-10-30T13:59:09.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1ef7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f00cd4a0 0x7fc1f0144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1effff640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f0143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1ef7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f00cd4a0 0x7fc1f0144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35880/0 (socket says 172.21.15.31:35880) 2023-10-30T13:59:09.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1ef7fe640 1 -- 172.21.15.31:0/1122942351 learned_addr learned my addr 172.21.15.31:0/1122942351 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:09.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.832+0000 7fc1f4e08640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1f01516c0 0x7fc1f015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ef7fe640 1 -- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 msgr2=0x7fc1f0143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ef7fe640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f0143670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ef7fe640 1 -- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1f01516c0 msgr2=0x7fc1f015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:09.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ef7fe640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1f01516c0 0x7fc1f015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:09.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ef7fe640 1 -- 172.21.15.31:0/1122942351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0009420 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1f4e08640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1f01516c0 0x7fc1f015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:09.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ef7fe640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f00cd4a0 0x7fc1f0144ca0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400d800 tx=0x7fc1e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ecff9640 1 -- 172.21.15.31:0/1122942351 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40096c0 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ecff9640 1 -- 172.21.15.31:0/1122942351 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e4009820 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ecff9640 1 -- 172.21.15.31:0/1122942351 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4009bd0 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f01498c0 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f0149e80 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f014a040 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ecff9640 1 -- 172.21.15.31:0/1122942351 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1e4009d30 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ecff9640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1dc0664a0 0x7fc1dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:09.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1ecff9640 1 -- 172.21.15.31:0/1122942351 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fc1e4084a70 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.836+0000 7fc1effff640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1dc0664a0 0x7fc1dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:09.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.840+0000 7fc1effff640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1dc0664a0 0x7fc1dc068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00ad000 tx=0x7fc1e00a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:09.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.840+0000 7fc1ecff9640 1 -- 172.21.15.31:0/1122942351 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1e40519a0 con 0x7fc1f00cd4a0 2023-10-30T13:59:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.996+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1f0158230 con 0x7fc1dc0664a0 2023-10-30T13:59:10.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:09.996+0000 7fc1ecff9640 1 -- 172.21.15.31:0/1122942351 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc1e00ab760 con 0x7fc1dc0664a0 2023-10-30T13:59:10.000 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T13:59:10.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1dc0664a0 msgr2=0x7fc1dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1dc0664a0 0x7fc1dc068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00ad000 tx=0x7fc1e00a6000 comp rx=0 tx=0).stop 2023-10-30T13:59:10.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f00cd4a0 msgr2=0x7fc1f0144ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f00cd4a0 0x7fc1f0144ca0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400d800 tx=0x7fc1e400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 shutdown_connections 2023-10-30T13:59:10.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fc1dc0664a0 0x7fc1dc068960 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1f01516c0 0x7fc1f015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1f00cd4a0 0x7fc1f0144ca0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 --2- 172.21.15.31:0/1122942351 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fc1f00c82d0 0x7fc1f0143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 >> 172.21.15.31:0/1122942351 conn(0x7fc1f0071fb0 msgr2=0x7fc1f014f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 shutdown_connections 2023-10-30T13:59:10.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.000+0000 7fc1f686b640 1 -- 172.21.15.31:0/1122942351 wait complete. 2023-10-30T13:59:10.021 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T13:59:10.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:10.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed94817640 1 Processor -- start 2023-10-30T13:59:10.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed94817640 1 -- start start 2023-10-30T13:59:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed94817640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006fb20 0x7fed900c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed94817640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9015ded0 con 0x7fed90153bd0 2023-10-30T13:59:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed94817640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9006b130 con 0x7fed9006ff00 2023-10-30T13:59:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed94817640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9006ee00 con 0x7fed9006fb20 2023-10-30T13:59:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8ed9d640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fed90153bd0 0x7fed900c20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49536/0 (socket says 172.21.15.31:49536) 2023-10-30T13:59:10.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8ed9d640 1 -- 172.21.15.31:0/1519023260 learned_addr learned my addr 172.21.15.31:0/1519023260 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:10.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8e59c640 1 --2- 172.21.15.31:0/1519023260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006fb20 0x7fed900c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8d59a640 1 -- 172.21.15.31:0/1519023260 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398114375 0 0) 0x7fed9015ded0 con 0x7fed90153bd0 2023-10-30T13:59:10.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8d59a640 1 -- 172.21.15.31:0/1519023260 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed78003530 con 0x7fed90153bd0 2023-10-30T13:59:10.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8e59c640 1 -- 172.21.15.31:0/1519023260 >> v1:172.21.15.33:6789/0 conn(0x7fed9006ff00 legacy=0x7fed901534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8d59a640 1 -- 172.21.15.31:0/1519023260 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 399256492 0 0) 0x7fed78003530 con 0x7fed90153bd0 2023-10-30T13:59:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8e59c640 1 -- 172.21.15.31:0/1519023260 >> v1:172.21.15.31:6789/0 conn(0x7fed90153bd0 legacy=0x7fed900c20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.167+0000 7fed8e59c640 1 -- 172.21.15.31:0/1519023260 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9006cff0 con 0x7fed9006fb20 2023-10-30T13:59:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed8e59c640 1 --2- 172.21.15.31:0/1519023260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006fb20 0x7fed900c7990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fed80009770 tx=0x7fed8001c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a77b95b9a759ed1b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed8d59a640 1 -- 172.21.15.31:0/1519023260 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88001a00 con 0x7fed9006fb20 2023-10-30T13:59:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed8d59a640 1 -- 172.21.15.31:0/1519023260 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed800a4cb0 con 0x7fed9006fb20 2023-10-30T13:59:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed8d59a640 1 -- 172.21.15.31:0/1519023260 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed800ac640 con 0x7fed9006fb20 2023-10-30T13:59:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed94817640 1 -- 172.21.15.31:0/1519023260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006fb20 msgr2=0x7fed900c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed94817640 1 --2- 172.21.15.31:0/1519023260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006fb20 0x7fed900c7990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fed80009770 tx=0x7fed8001c180 comp rx=0 tx=0).stop 2023-10-30T13:59:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed94817640 1 -- 172.21.15.31:0/1519023260 shutdown_connections 2023-10-30T13:59:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed94817640 1 --2- 172.21.15.31:0/1519023260 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006fb20 0x7fed900c7990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed94817640 1 -- 172.21.15.31:0/1519023260 >> 172.21.15.31:0/1519023260 conn(0x7fed900cd010 msgr2=0x7fed900cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed94817640 1 -- 172.21.15.31:0/1519023260 shutdown_connections 2023-10-30T13:59:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.171+0000 7fed94817640 1 -- 172.21.15.31:0/1519023260 wait complete. 2023-10-30T13:59:10.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 Processor -- start 2023-10-30T13:59:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 -- start start 2023-10-30T13:59:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9006fb20 0x7fed9006bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006ff00 0x7fed900c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed90153bd0 0x7fed901488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9006ca50 con 0x7fed9006fb20 2023-10-30T13:59:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed90154880 con 0x7fed90153bd0 2023-10-30T13:59:10.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed901f6930 con 0x7fed9006ff00 2023-10-30T13:59:10.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9006fb20 0x7fed9006bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9006fb20 0x7fed9006bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35888/0 (socket says 172.21.15.31:35888) 2023-10-30T13:59:10.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 -- 172.21.15.31:0/3135261480 learned_addr learned my addr 172.21.15.31:0/3135261480 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:10.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8dd9b640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006ff00 0x7fed900c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8ed9d640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed90153bd0 0x7fed901488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 -- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006ff00 msgr2=0x7fed900c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fed9006ff00 0x7fed900c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 -- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed90153bd0 msgr2=0x7fed901488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed90153bd0 0x7fed901488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 -- 172.21.15.31:0/3135261480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80009420 con 0x7fed9006fb20 2023-10-30T13:59:10.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed8e59c640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9006fb20 0x7fed9006bb80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fed8001a830 tx=0x7fed80018f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80003a30 con 0x7fed9006fb20 2023-10-30T13:59:10.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80003b90 con 0x7fed9006fb20 2023-10-30T13:59:10.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed800ac440 con 0x7fed9006fb20 2023-10-30T13:59:10.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed901f5b90 con 0x7fed9006fb20 2023-10-30T13:59:10.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed901f5e50 con 0x7fed9006fb20 2023-10-30T13:59:10.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.175+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fed9006e740 con 0x7fed9006fb20 2023-10-30T13:59:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed800b3040 con 0x7fed9006fb20 2023-10-30T13:59:10.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed7effd640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed700664d0 0x7fed70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fed8011e280 con 0x7fed9006fb20 2023-10-30T13:59:10.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed7effd640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fed7006f090 0x7fed70071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 --> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed70071c00 con 0x7fed7006f090 2023-10-30T13:59:10.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed80120860 con 0x7fed9006fb20 2023-10-30T13:59:10.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed8dd9b640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed700664d0 0x7fed70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed8ed9d640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fed7006f090 0x7fed70071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed8ed9d640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fed7006f090 0x7fed70071530 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.179+0000 7fed8dd9b640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed700664d0 0x7fed70068990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fed900c8430 tx=0x7fed84009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.183+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== osd.6 v2:172.21.15.33:6808/931136877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fed70071c00 con 0x7fed7006f090 2023-10-30T13:59:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 --> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fed9006e460 con 0x7fed7006f090 2023-10-30T13:59:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed7effd640 1 -- 172.21.15.31:0/3135261480 <== osd.6 v2:172.21.15.33:6808/931136877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fed880015a0 con 0x7fed7006f090 2023-10-30T13:59:10.199 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:10.199 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:10.200 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:10.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fed7006f090 msgr2=0x7fed70071530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6808/931136877,v1:172.21.15.33:6809/931136877] conn(0x7fed7006f090 0x7fed70071530 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed700664d0 msgr2=0x7fed70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed700664d0 0x7fed70068990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fed900c8430 tx=0x7fed84009040 comp rx=0 tx=0).stop 2023-10-30T13:59:10.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9006fb20 msgr2=0x7fed9006bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 --2- 172.21.15.31:0/3135261480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9006fb20 0x7fed9006bb80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fed8001a830 tx=0x7fed80018f20 comp rx=0 tx=0).stop 2023-10-30T13:59:10.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed8ed9d640 1 -- 172.21.15.31:0/3135261480 reap_dead start 2023-10-30T13:59:10.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 shutdown_connections 2023-10-30T13:59:10.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 >> 172.21.15.31:0/3135261480 conn(0x7fed900cd010 msgr2=0x7fed9014ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 shutdown_connections 2023-10-30T13:59:10.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.195+0000 7fed94817640 1 -- 172.21.15.31:0/3135261480 wait complete. 2023-10-30T13:59:10.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T13:59:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46d62640 1 Processor -- start 2023-10-30T13:59:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46d62640 1 -- start start 2023-10-30T13:59:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46d62640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c400c8b00 0x7f8c400c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46d62640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8c400b22a0 0x7f8c400b3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46d62640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c400b27f0 con 0x7f8c4015cdf0 2023-10-30T13:59:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46d62640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4015d370 con 0x7f8c400b22a0 2023-10-30T13:59:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46d62640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4015d630 con 0x7f8c400c8b00 2023-10-30T13:59:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46561640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8c400b22a0 0x7f8c400b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46561640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8c400b22a0 0x7f8c400b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35900/0 (socket says 172.21.15.31:35900) 2023-10-30T13:59:10.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46561640 1 -- 172.21.15.31:0/466375269 learned_addr learned my addr 172.21.15.31:0/466375269 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:10.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c45d60640 1 --2- 172.21.15.31:0/466375269 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c400c8b00 0x7f8c400c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46561640 1 -- 172.21.15.31:0/466375269 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c400c8b00 msgr2=0x7f8c400c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46561640 1 --2- 172.21.15.31:0/466375269 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c400c8b00 0x7f8c400c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.367+0000 7f8c46561640 1 -- 172.21.15.31:0/466375269 >> v1:172.21.15.33:6789/0 conn(0x7f8c4015cdf0 legacy=0x7f8c400c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46561640 1 -- 172.21.15.31:0/466375269 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4015d910 con 0x7f8c400b22a0 2023-10-30T13:59:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46561640 1 --2- 172.21.15.31:0/466375269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c400b3240 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009e50 tx=0x7f8c3801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f52c7d2ea62ead server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c44d5e640 1 -- 172.21.15.31:0/466375269 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c380af400 con 0x7f8c400b22a0 2023-10-30T13:59:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c44d5e640 1 -- 172.21.15.31:0/466375269 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c380ab050 con 0x7f8c400b22a0 2023-10-30T13:59:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c44d5e640 1 -- 172.21.15.31:0/466375269 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c380af780 con 0x7f8c400b22a0 2023-10-30T13:59:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 -- 172.21.15.31:0/466375269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 msgr2=0x7f8c400b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 --2- 172.21.15.31:0/466375269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c400b3240 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009e50 tx=0x7f8c3801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 -- 172.21.15.31:0/466375269 shutdown_connections 2023-10-30T13:59:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 --2- 172.21.15.31:0/466375269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c400b3240 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 --2- 172.21.15.31:0/466375269 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c400c8b00 0x7f8c400c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 -- 172.21.15.31:0/466375269 >> 172.21.15.31:0/466375269 conn(0x7f8c4006bed0 msgr2=0x7f8c4006c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 -- 172.21.15.31:0/466375269 shutdown_connections 2023-10-30T13:59:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.371+0000 7f8c46d62640 1 -- 172.21.15.31:0/466375269 wait complete. 2023-10-30T13:59:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 Processor -- start 2023-10-30T13:59:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 -- start start 2023-10-30T13:59:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c401ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c400c8b00 0x7f8c400cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c4015cdf0 0x7f8c4015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c40150f20 con 0x7f8c400b22a0 2023-10-30T13:59:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c401ef9e0 con 0x7f8c400c8b00 2023-10-30T13:59:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c401500c0 con 0x7f8c4015cdf0 2023-10-30T13:59:10.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c401ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c401ef4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35914/0 (socket says 172.21.15.31:35914) 2023-10-30T13:59:10.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 -- 172.21.15.31:0/230431518 learned_addr learned my addr 172.21.15.31:0/230431518 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:10.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46561640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c4015cdf0 0x7f8c4015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c4555f640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c400c8b00 0x7f8c400cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 -- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c4015cdf0 msgr2=0x7f8c4015a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c4015cdf0 0x7f8c4015a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 -- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c400c8b00 msgr2=0x7f8c400cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c400c8b00 0x7f8c400cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 -- 172.21.15.31:0/230431518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c38009b30 con 0x7f8c400b22a0 2023-10-30T13:59:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c45d60640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c401ef4a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00d800 tx=0x7f8c3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c367fc640 1 -- 172.21.15.31:0/230431518 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c00b6c0 con 0x7f8c400b22a0 2023-10-30T13:59:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c367fc640 1 -- 172.21.15.31:0/230431518 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c00b820 con 0x7f8c400b22a0 2023-10-30T13:59:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c367fc640 1 -- 172.21.15.31:0/230431518 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c00bb90 con 0x7f8c400b22a0 2023-10-30T13:59:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c401503a0 con 0x7f8c400b22a0 2023-10-30T13:59:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.375+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c40150960 con 0x7f8c400b22a0 2023-10-30T13:59:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.379+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4014def0 con 0x7f8c400b22a0 2023-10-30T13:59:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.379+0000 7f8c367fc640 1 -- 172.21.15.31:0/230431518 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c3c00bcf0 con 0x7f8c400b22a0 2023-10-30T13:59:10.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.379+0000 7f8c367fc640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c240664d0 0x7f8c24068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.379+0000 7f8c367fc640 1 -- 172.21.15.31:0/230431518 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f8c3c0848e0 con 0x7f8c400b22a0 2023-10-30T13:59:10.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.379+0000 7f8c4555f640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c240664d0 0x7f8c24068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.379+0000 7f8c4555f640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c240664d0 0x7f8c24068990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8c400c7f00 tx=0x7f8c30009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.383+0000 7f8c367fc640 1 -- 172.21.15.31:0/230431518 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c3c0518a0 con 0x7f8c400b22a0 2023-10-30T13:59:10.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.539+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c400b2a40 con 0x7f8c240664d0 2023-10-30T13:59:10.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c367fc640 1 -- 172.21.15.31:0/230431518 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8c30001370 con 0x7f8c240664d0 2023-10-30T13:59:10.545 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T13:59:10.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c240664d0 msgr2=0x7f8c24068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c240664d0 0x7f8c24068990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8c400c7f00 tx=0x7f8c30009040 comp rx=0 tx=0).stop 2023-10-30T13:59:10.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 msgr2=0x7f8c401ef4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c401ef4a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00d800 tx=0x7f8c3c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 shutdown_connections 2023-10-30T13:59:10.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8c240664d0 0x7f8c24068990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8c4015cdf0 0x7f8c4015a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c400c8b00 0x7f8c400cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 --2- 172.21.15.31:0/230431518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c400b22a0 0x7f8c401ef4a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 >> 172.21.15.31:0/230431518 conn(0x7f8c4006bed0 msgr2=0x7f8c400c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 shutdown_connections 2023-10-30T13:59:10.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.543+0000 7f8c46d62640 1 -- 172.21.15.31:0/230431518 wait complete. 2023-10-30T13:59:10.566 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T13:59:10.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:59:10.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a4091d640 1 Processor -- start 2023-10-30T13:59:10.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a4091d640 1 -- start start 2023-10-30T13:59:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a4091d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 0x7f5a3c06fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a4091d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f5a3c0c7770 0x7f5a3c0c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a4091d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0c74d0 con 0x7f5a3c150690 2023-10-30T13:59:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a4091d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3c072420 con 0x7f5a3c0c7770 2023-10-30T13:59:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a4091d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c1564d0 con 0x7f5a3c0c0930 2023-10-30T13:59:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3a59c640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f5a3c150690 0x7f5a3c156c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49550/0 (socket says 172.21.15.31:49550) 2023-10-30T13:59:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3a59c640 1 -- 172.21.15.31:0/746063432 learned_addr learned my addr 172.21.15.31:0/746063432 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/746063432 >> v2:172.21.15.33:3300/0 conn(0x7f5a3c0c7770 0x7f5a3c0c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a39d9b640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 0x7f5a3c06fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785386148 0 0) 0x7f5a2c001120 con 0x7f5a3c150690 2023-10-30T13:59:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a24005740 con 0x7f5a3c150690 2023-10-30T13:59:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4197828272 0 0) 0x7f5a24005740 con 0x7f5a3c150690 2023-10-30T13:59:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 msgr2=0x7f5a3c06fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 0x7f5a3c06fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 >> v2:172.21.15.33:3300/0 conn(0x7f5a3c0c7770 msgr2=0x7f5a3c0c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 --2- 172.21.15.31:0/746063432 >> v2:172.21.15.33:3300/0 conn(0x7f5a3c0c7770 0x7f5a3c0c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0718a0 con 0x7f5a3c150690 2023-10-30T13:59:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f5a2c001810 con 0x7f5a3c150690 2023-10-30T13:59:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 >> v1:172.21.15.31:6789/0 conn(0x7f5a3c150690 legacy=0x7f5a3c156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24005b90 0x7f5a24005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 0x7f5a240069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a24006f20 0x7f5a240073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c072420 con 0x7f5a24006f20 2023-10-30T13:59:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c1564d0 con 0x7f5a24005b90 2023-10-30T13:59:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2400dba0 con 0x7f5a3c0c0930 2023-10-30T13:59:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a24006f20 0x7f5a240073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.711+0000 7f5a3a59c640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24005b90 0x7f5a24005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a39d9b640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 0x7f5a240069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 msgr2=0x7f5a240069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c0c0930 0x7f5a240069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24005b90 msgr2=0x7f5a24005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24005b90 0x7f5a24005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/746063432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a24005740 con 0x7f5a24006f20 2023-10-30T13:59:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a24006f20 0x7f5a240073a0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009fc0 tx=0x7f5a3401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcb0250b2e1a4df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340a5cb0 con 0x7f5a24006f20 2023-10-30T13:59:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3400f040 con 0x7f5a24006f20 2023-10-30T13:59:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3959a640 1 -- 172.21.15.31:0/746063432 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340ad680 con 0x7f5a24006f20 2023-10-30T13:59:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a4091d640 1 -- 172.21.15.31:0/746063432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a24006f20 msgr2=0x7f5a240073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a4091d640 1 --2- 172.21.15.31:0/746063432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a24006f20 0x7f5a240073a0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009fc0 tx=0x7f5a3401cca0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.715+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/746063432 reap_dead start 2023-10-30T13:59:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- 172.21.15.31:0/746063432 shutdown_connections 2023-10-30T13:59:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- 172.21.15.31:0/746063432 >> 172.21.15.31:0/746063432 conn(0x7f5a3c06e150 msgr2=0x7f5a3c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- 172.21.15.31:0/746063432 shutdown_connections 2023-10-30T13:59:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- 172.21.15.31:0/746063432 wait complete. 2023-10-30T13:59:10.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 Processor -- start 2023-10-30T13:59:10.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- start start 2023-10-30T13:59:10.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24006f20 0x7f5a3c158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c159160 0x7f5a3c1efd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a3c157690 0x7f5a3c1510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c157d20 con 0x7f5a3c157690 2023-10-30T13:59:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c1f0270 con 0x7f5a24006f20 2023-10-30T13:59:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a4091d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c1ef2b0 con 0x7f5a3c159160 2023-10-30T13:59:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a3c157690 0x7f5a3c1510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a3c157690 0x7f5a3c1510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35934/0 (socket says 172.21.15.31:35934) 2023-10-30T13:59:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/368245657 learned_addr learned my addr 172.21.15.31:0/368245657 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3a59c640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24006f20 0x7f5a3c158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a39d9b640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c159160 0x7f5a3c1efd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c159160 msgr2=0x7f5a3c1efd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f5a3c159160 0x7f5a3c1efd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24006f20 msgr2=0x7f5a3c158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a24006f20 0x7f5a3c158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/368245657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0caa60 con 0x7f5a3c157690 2023-10-30T13:59:10.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.719+0000 7f5a3ad9d640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a3c157690 0x7f5a3c1510a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b0040 tx=0x7f5a34009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340039a0 con 0x7f5a3c157690 2023-10-30T13:59:10.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a34003b20 con 0x7f5a3c157690 2023-10-30T13:59:10.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340ae890 con 0x7f5a3c157690 2023-10-30T13:59:10.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3c1537d0 con 0x7f5a3c157690 2023-10-30T13:59:10.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3c153d90 con 0x7f5a3c157690 2023-10-30T13:59:10.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5a3c071e60 con 0x7f5a3c157690 2023-10-30T13:59:10.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5a340b4040 con 0x7f5a3c157690 2023-10-30T13:59:10.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5a1c0664a0 0x7f5a1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f5a3411e140 con 0x7f5a3c157690 2023-10-30T13:59:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f5a1c06f060 0x7f5a1c071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 --> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a1c071bd0 con 0x7f5a1c06f060 2023-10-30T13:59:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a34120720 con 0x7f5a3c157690 2023-10-30T13:59:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a3a59c640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5a1c0664a0 0x7f5a1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.723+0000 7f5a39d9b640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f5a1c06f060 0x7f5a1c071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.727+0000 7f5a3a59c640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5a1c0664a0 0x7f5a1c068960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0012c0 tx=0x7f5a2c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.727+0000 7f5a39d9b640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f5a1c06f060 0x7f5a1c071500 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.727+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== osd.7 v2:172.21.15.33:6824/3463894958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5a1c071bd0 con 0x7f5a1c06f060 2023-10-30T13:59:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.739+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 --> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5a3c071b80 con 0x7f5a1c06f060 2023-10-30T13:59:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a2affd640 1 -- 172.21.15.31:0/368245657 <== osd.7 v2:172.21.15.33:6824/3463894958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5a30009c50 con 0x7f5a1c06f060 2023-10-30T13:59:10.745 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-30T13:59:10.745 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:59:10.745 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-30T13:59:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f5a1c06f060 msgr2=0x7f5a1c071500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6824/3463894958,v1:172.21.15.33:6825/3463894958] conn(0x7f5a1c06f060 0x7f5a1c071500 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5a1c0664a0 msgr2=0x7f5a1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f5a1c0664a0 0x7f5a1c068960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0012c0 tx=0x7f5a2c009040 comp rx=0 tx=0).stop 2023-10-30T13:59:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a3c157690 msgr2=0x7f5a3c1510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 --2- 172.21.15.31:0/368245657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a3c157690 0x7f5a3c1510a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b0040 tx=0x7f5a34009750 comp rx=0 tx=0).stop 2023-10-30T13:59:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a3ad9d640 1 -- 172.21.15.31:0/368245657 reap_dead start 2023-10-30T13:59:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 shutdown_connections 2023-10-30T13:59:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 >> 172.21.15.31:0/368245657 conn(0x7f5a3c06e150 msgr2=0x7f5a3c06bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 shutdown_connections 2023-10-30T13:59:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.743+0000 7f5a4091d640 1 -- 172.21.15.31:0/368245657 wait complete. 2023-10-30T13:59:10.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T13:59:10.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a85047640 1 Processor -- start 2023-10-30T13:59:10.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a85047640 1 -- start start 2023-10-30T13:59:10.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a85047640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f6a800c8b00 0x7f6a800c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a85047640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6a8015cdf0 0x7f6a800c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a85047640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a800b27f0 con 0x7f6a800b22a0 2023-10-30T13:59:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a85047640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8015d370 con 0x7f6a8015cdf0 2023-10-30T13:59:10.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a85047640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8015d630 con 0x7f6a800c8b00 2023-10-30T13:59:10.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a7f7fe640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f6a8015cdf0 0x7f6a800c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a84846640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f6a800b22a0 0x7f6a800b3240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49566/0 (socket says 172.21.15.31:49566) 2023-10-30T13:59:10.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a84846640 1 -- 172.21.15.31:0/3559896698 learned_addr learned my addr 172.21.15.31:0/3559896698 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:10.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a7ffff640 1 --2- 172.21.15.31:0/3559896698 >> v2:172.21.15.33:3300/0 conn(0x7f6a800c8b00 0x7f6a800c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a7f7fe640 1 -- 172.21.15.31:0/3559896698 >> v2:172.21.15.33:3300/0 conn(0x7f6a800c8b00 msgr2=0x7f6a800c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:10.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a7f7fe640 1 --2- 172.21.15.31:0/3559896698 >> v2:172.21.15.33:3300/0 conn(0x7f6a800c8b00 0x7f6a800c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a7f7fe640 1 -- 172.21.15.31:0/3559896698 >> v1:172.21.15.31:6789/0 conn(0x7f6a800b22a0 legacy=0x7f6a800b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.911+0000 7f6a7f7fe640 1 -- 172.21.15.31:0/3559896698 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8015d910 con 0x7f6a8015cdf0 2023-10-30T13:59:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a7f7fe640 1 --2- 172.21.15.31:0/3559896698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a800c9e30 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6a8006ca60 tx=0x7f6a7401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2d27bdde668836 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a7effd640 1 -- 172.21.15.31:0/3559896698 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a740a7070 con 0x7f6a8015cdf0 2023-10-30T13:59:10.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a7effd640 1 -- 172.21.15.31:0/3559896698 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a740ab430 con 0x7f6a8015cdf0 2023-10-30T13:59:10.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a7effd640 1 -- 172.21.15.31:0/3559896698 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a740ab7c0 con 0x7f6a8015cdf0 2023-10-30T13:59:10.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 -- 172.21.15.31:0/3559896698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 msgr2=0x7f6a800c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 --2- 172.21.15.31:0/3559896698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a800c9e30 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6a8006ca60 tx=0x7f6a7401c430 comp rx=0 tx=0).stop 2023-10-30T13:59:10.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 -- 172.21.15.31:0/3559896698 shutdown_connections 2023-10-30T13:59:10.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 --2- 172.21.15.31:0/3559896698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a800c9e30 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 --2- 172.21.15.31:0/3559896698 >> v2:172.21.15.33:3300/0 conn(0x7f6a800c8b00 0x7f6a800c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 -- 172.21.15.31:0/3559896698 >> 172.21.15.31:0/3559896698 conn(0x7f6a8006bed0 msgr2=0x7f6a8006c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 -- 172.21.15.31:0/3559896698 shutdown_connections 2023-10-30T13:59:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.915+0000 7f6a85047640 1 -- 172.21.15.31:0/3559896698 wait complete. 2023-10-30T13:59:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 Processor -- start 2023-10-30T13:59:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 -- start start 2023-10-30T13:59:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a800b22a0 0x7f6a8015c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6a800c8b00 0x7f6a800ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a801f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a801f6140 con 0x7f6a8015cdf0 2023-10-30T13:59:10.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8015b490 con 0x7f6a800b22a0 2023-10-30T13:59:10.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a80157790 con 0x7f6a800c8b00 2023-10-30T13:59:10.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a801f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a801f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35950/0 (socket says 172.21.15.31:35950) 2023-10-30T13:59:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 -- 172.21.15.31:0/968308140 learned_addr learned my addr 172.21.15.31:0/968308140 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a7ffff640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a800b22a0 0x7f6a8015c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a7f7fe640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6a800c8b00 0x7f6a800ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 -- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6a800c8b00 msgr2=0x7f6a800ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6a800c8b00 0x7f6a800ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 -- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a800b22a0 msgr2=0x7f6a8015c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a800b22a0 0x7f6a8015c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 -- 172.21.15.31:0/968308140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74009420 con 0x7f6a8015cdf0 2023-10-30T13:59:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a7ffff640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a800b22a0 0x7f6a8015c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a84846640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a801f6560 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800ca80 tx=0x7f6a7800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a7cff9640 1 -- 172.21.15.31:0/968308140 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800ac30 con 0x7f6a8015cdf0 2023-10-30T13:59:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a7cff9640 1 -- 172.21.15.31:0/968308140 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7800ad90 con 0x7f6a8015cdf0 2023-10-30T13:59:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a7cff9640 1 -- 172.21.15.31:0/968308140 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78012600 con 0x7f6a8015cdf0 2023-10-30T13:59:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a80157a70 con 0x7f6a8015cdf0 2023-10-30T13:59:10.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.919+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a80158030 con 0x7f6a8015cdf0 2023-10-30T13:59:10.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.923+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8004e170 con 0x7f6a8015cdf0 2023-10-30T13:59:10.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.923+0000 7f6a7cff9640 1 -- 172.21.15.31:0/968308140 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a78007070 con 0x7f6a8015cdf0 2023-10-30T13:59:10.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.923+0000 7f6a7cff9640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6a6c0664a0 0x7f6a6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:10.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.923+0000 7f6a7cff9640 1 -- 172.21.15.31:0/968308140 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f6a78084150 con 0x7f6a8015cdf0 2023-10-30T13:59:10.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.923+0000 7f6a7ffff640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6a6c0664a0 0x7f6a6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:10.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.923+0000 7f6a7ffff640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6a6c0664a0 0x7f6a6c068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6a70007a70 tx=0x7f6a70008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:10.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:10.927+0000 7f6a7cff9640 1 -- 172.21.15.31:0/968308140 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a780511c0 con 0x7f6a8015cdf0 2023-10-30T13:59:11.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a800b2d20 con 0x7f6a6c0664a0 2023-10-30T13:59:11.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a7cff9640 1 -- 172.21.15.31:0/968308140 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6a70007710 con 0x7f6a6c0664a0 2023-10-30T13:59:11.086 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T13:59:11.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6a6c0664a0 msgr2=0x7f6a6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6a6c0664a0 0x7f6a6c068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6a70007a70 tx=0x7f6a70008040 comp rx=0 tx=0).stop 2023-10-30T13:59:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 msgr2=0x7f6a801f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a801f6560 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800ca80 tx=0x7f6a7800cf40 comp rx=0 tx=0).stop 2023-10-30T13:59:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 shutdown_connections 2023-10-30T13:59:11.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f6a6c0664a0 0x7f6a6c068960 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a8015cdf0 0x7f6a801f6560 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f6a800c8b00 0x7f6a800ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 --2- 172.21.15.31:0/968308140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a800b22a0 0x7f6a8015c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 >> 172.21.15.31:0/968308140 conn(0x7f6a8006bed0 msgr2=0x7f6a800c6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:11.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 shutdown_connections 2023-10-30T13:59:11.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.083+0000 7f6a85047640 1 -- 172.21.15.31:0/968308140 wait complete. 2023-10-30T13:59:11.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:59:11.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.247+0000 7f1467b9d640 1 Processor -- start 2023-10-30T13:59:11.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.247+0000 7f1467b9d640 1 -- start start 2023-10-30T13:59:11.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1467b9d640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f146015d840 0x7f14600ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1467b9d640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600c7b00 con 0x7f1460153aa0 2023-10-30T13:59:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1467b9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600c2050 con 0x7f14600cd420 2023-10-30T13:59:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1467b9d640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1460070a50 con 0x7f146015d840 2023-10-30T13:59:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f1460153aa0 0x7f14600701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:49576/0 (socket says 172.21.15.31:49576) 2023-10-30T13:59:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 -- 172.21.15.31:0/644161024 learned_addr learned my addr 172.21.15.31:0/644161024 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T13:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465138640 1 --2- 172.21.15.31:0/644161024 >> v2:172.21.15.33:3300/0 conn(0x7f146015d840 0x7f14600ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903466272 0 0) 0x7f14600c7b00 con 0x7f1460153aa0 2023-10-30T13:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c003530 con 0x7f1460153aa0 2023-10-30T13:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 484499041 0 0) 0x7f144c003530 con 0x7f1460153aa0 2023-10-30T13:59:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 >> v2:172.21.15.33:3300/0 conn(0x7f146015d840 msgr2=0x7f14600ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 --2- 172.21.15.31:0/644161024 >> v2:172.21.15.33:3300/0 conn(0x7f146015d840 0x7f14600ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 >> v1:172.21.15.33:6789/0 conn(0x7f14600cd420 legacy=0x7f146006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600c2330 con 0x7f1460153aa0 2023-10-30T13:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f145c001680 con 0x7f1460153aa0 2023-10-30T13:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 >> v1:172.21.15.31:6789/0 conn(0x7f1460153aa0 legacy=0x7f14600701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f144c003980 0x7f144c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c0047d0 0x7f144c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f144c005190 0x7f144c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1460070a50 con 0x7f144c003980 2023-10-30T13:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c00c040 con 0x7f144c0047d0 2023-10-30T13:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c00c200 con 0x7f144c005190 2023-10-30T13:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f144c003980 0x7f144c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465138640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c0047d0 0x7f144c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f146613a640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f144c005190 0x7f144c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 -- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f144c005190 msgr2=0x7f144c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f144c005190 0x7f144c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 -- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c0047d0 msgr2=0x7f144c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:11.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c0047d0 0x7f144c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465939640 1 -- 172.21.15.31:0/644161024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c003530 con 0x7f144c003980 2023-10-30T13:59:11.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f1465138640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c0047d0 0x7f144c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:59:11.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.251+0000 7f146613a640 1 -- 172.21.15.31:0/644161024 reap_dead start 2023-10-30T13:59:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1465939640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f144c003980 0x7f144c003da0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c1c0 tx=0x7f145c01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3c80c71f63aec9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c005ae0 con 0x7f144c003980 2023-10-30T13:59:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c0ab070 con 0x7f144c003980 2023-10-30T13:59:11.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1464937640 1 -- 172.21.15.31:0/644161024 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c007070 con 0x7f144c003980 2023-10-30T13:59:11.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1467b9d640 1 -- 172.21.15.31:0/644161024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f144c003980 msgr2=0x7f144c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1467b9d640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f144c003980 0x7f144c003da0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c1c0 tx=0x7f145c01f390 comp rx=0 tx=0).stop 2023-10-30T13:59:11.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1467b9d640 1 -- 172.21.15.31:0/644161024 shutdown_connections 2023-10-30T13:59:11.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1467b9d640 1 --2- 172.21.15.31:0/644161024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f144c003980 0x7f144c003da0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1467b9d640 1 -- 172.21.15.31:0/644161024 >> 172.21.15.31:0/644161024 conn(0x7f1460071fb0 msgr2=0x7f14600723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:11.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1467b9d640 1 -- 172.21.15.31:0/644161024 shutdown_connections 2023-10-30T13:59:11.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.255+0000 7f1467b9d640 1 -- 172.21.15.31:0/644161024 wait complete. 2023-10-30T13:59:11.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 Processor -- start 2023-10-30T13:59:11.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 -- start start 2023-10-30T13:59:11.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c003980 0x7f1460159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f14601eef30 0x7f14601ef350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14600c4c40 0x7f14600c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14601590b0 con 0x7f14600c4c40 2023-10-30T13:59:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1460158530 con 0x7f144c003980 2023-10-30T13:59:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14601580e0 con 0x7f14601eef30 2023-10-30T13:59:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14600c4c40 0x7f14600c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14600c4c40 0x7f14600c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35974/0 (socket says 172.21.15.31:35974) 2023-10-30T13:59:11.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 -- 172.21.15.31:0/1232208696 learned_addr learned my addr 172.21.15.31:0/1232208696 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:11.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1465138640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f14601eef30 0x7f14601ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1465939640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c003980 0x7f1460159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 -- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f14601eef30 msgr2=0x7f14601ef350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f14601eef30 0x7f14601ef350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 -- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c003980 msgr2=0x7f1460159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c003980 0x7f1460159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 -- 172.21.15.31:0/1232208696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600c4490 con 0x7f14600c4c40 2023-10-30T13:59:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f146613a640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14600c4c40 0x7f14600c50c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1460071870 tx=0x7f145800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f14567fc640 1 -- 172.21.15.31:0/1232208696 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458003980 con 0x7f14600c4c40 2023-10-30T13:59:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f14567fc640 1 -- 172.21.15.31:0/1232208696 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458003b00 con 0x7f14600c4c40 2023-10-30T13:59:11.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f14567fc640 1 -- 172.21.15.31:0/1232208696 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580050d0 con 0x7f14600c4c40 2023-10-30T13:59:11.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1460158780 con 0x7f14600c4c40 2023-10-30T13:59:11.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1460158d40 con 0x7f14600c4c40 2023-10-30T13:59:11.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.259+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14601eed10 con 0x7f14600c4c40 2023-10-30T13:59:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.263+0000 7f14567fc640 1 -- 172.21.15.31:0/1232208696 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1458005230 con 0x7f14600c4c40 2023-10-30T13:59:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.263+0000 7f14567fc640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f14440664a0 0x7f1444068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.263+0000 7f14567fc640 1 -- 172.21.15.31:0/1232208696 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f1458083fe0 con 0x7f14600c4c40 2023-10-30T13:59:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.263+0000 7f1465939640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f14440664a0 0x7f1444068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:11.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.263+0000 7f1465939640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f14440664a0 0x7f1444068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f145c0af000 tx=0x7f145c0aa000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:11.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.267+0000 7f14567fc640 1 -- 172.21.15.31:0/1232208696 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1458051100 con 0x7f14600c4c40 2023-10-30T13:59:11.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.439+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14600c28f0 con 0x7f14440664a0 2023-10-30T13:59:11.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.471+0000 7f14567fc640 1 -- 172.21.15.31:0/1232208696 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291132 (secure 0 0 0) 0x7f145c001600 con 0x7f14440664a0 2023-10-30T13:59:11.476 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:11.478 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:59:11.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f14440664a0 msgr2=0x7f1444068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f14440664a0 0x7f1444068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f145c0af000 tx=0x7f145c0aa000 comp rx=0 tx=0).stop 2023-10-30T13:59:11.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14600c4c40 msgr2=0x7f14600c50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:11.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14600c4c40 0x7f14600c50c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1460071870 tx=0x7f145800ec10 comp rx=0 tx=0).stop 2023-10-30T13:59:11.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 shutdown_connections 2023-10-30T13:59:11.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f14440664a0 0x7f1444068960 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14600c4c40 0x7f14600c50c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f14601eef30 0x7f14601ef350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 --2- 172.21.15.31:0/1232208696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c003980 0x7f1460159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 >> 172.21.15.31:0/1232208696 conn(0x7f1460071fb0 msgr2=0x7f1460152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 shutdown_connections 2023-10-30T13:59:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:11.475+0000 7f1467b9d640 1 -- 172.21.15.31:0/1232208696 wait complete. 2023-10-30T13:59:11.501 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":777,"stamp":"2023-10-30T13:59:09.716573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185273216,"num_objects":10532,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10532,"num_whiteouts":0,"num_read":8937,"num_read_kb":1843152,"num_write":58135,"num_write_kb":2329402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":44255,"num_omap_keys":84,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61343,"ondisk_log_size":61343,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3092352,"kb_used_data":2371072,"kb_used_omap":12,"kb_used_meta":721203,"kb_avail":751882368,"statfs":{"total":773094113280,"available":769927544832,"internally_reserved":0,"allocated":2427977728,"data_stored":2371590088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":738512440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,36,0,0,22],"upper_bound":128},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8644735,"num_objects":1632,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1795,"num_write_kb":33612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":44255,"num_omap_keys":84,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004578"},"pg_stats":[{"pgid":"3.3f","version":"56'674","reported_seq":2893,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.519128+0000","last_change":"2023-10-30T13:45:28.601107+0000","last_active":"2023-10-30T13:59:03.519128+0000","last_peered":"2023-10-30T13:59:03.519128+0000","last_clean":"2023-10-30T13:59:03.519128+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:59:03.519128+0000","last_undegraded":"2023-10-30T13:59:03.519128+0000","last_fullsized":"2023-10-30T13:59:03.519128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:58.462635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098613959999999997,"stat_sum":{"num_bytes":30553715,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":124,"num_read_kb":8207,"num_write":638,"num_write_kb":39180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"56'657","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.487037+0000","last_change":"2023-10-30T13:45:27.935552+0000","last_active":"2023-10-30T13:59:03.487037+0000","last_peered":"2023-10-30T13:59:03.487037+0000","last_clean":"2023-10-30T13:59:03.487037+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:59:03.487037+0000","last_undegraded":"2023-10-30T13:59:03.487037+0000","last_fullsized":"2023-10-30T13:59:03.487037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001955118,"stat_sum":{"num_bytes":9994664,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":104,"num_read_kb":11,"num_write":617,"num_write_kb":23604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"56'684","reported_seq":2887,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.485730+0000","last_change":"2023-10-30T13:45:27.939453+0000","last_active":"2023-10-30T13:59:03.485730+0000","last_peered":"2023-10-30T13:59:03.485730+0000","last_clean":"2023-10-30T13:59:03.485730+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:59:03.485730+0000","last_undegraded":"2023-10-30T13:59:03.485730+0000","last_fullsized":"2023-10-30T13:59:03.485730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021120940000000001,"stat_sum":{"num_bytes":23390266,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":110,"num_read_kb":20497,"num_write":638,"num_write_kb":47001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"56'689","reported_seq":2915,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.497140+0000","last_change":"2023-10-30T13:45:28.487484+0000","last_active":"2023-10-30T13:59:03.497140+0000","last_peered":"2023-10-30T13:59:03.497140+0000","last_clean":"2023-10-30T13:59:03.497140+0000","last_became_active":"2023-10-30T13:35:48.779909+0000","last_became_peered":"2023-10-30T13:35:48.779909+0000","last_unstale":"2023-10-30T13:59:03.497140+0000","last_undegraded":"2023-10-30T13:59:03.497140+0000","last_fullsized":"2023-10-30T13:59:03.497140+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":689,"log_dups_size":0,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001317775,"stat_sum":{"num_bytes":10165225,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":133,"num_read_kb":8205,"num_write":656,"num_write_kb":40955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"56'701","reported_seq":2904,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.507846+0000","last_change":"2023-10-30T13:45:28.597729+0000","last_active":"2023-10-30T13:59:03.507846+0000","last_peered":"2023-10-30T13:59:03.507846+0000","last_clean":"2023-10-30T13:59:03.507846+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:59:03.507846+0000","last_undegraded":"2023-10-30T13:59:03.507846+0000","last_fullsized":"2023-10-30T13:59:03.507846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:29.242932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007614844,"stat_sum":{"num_bytes":19907850,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":110,"num_read_kb":8208,"num_write":664,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"56'666","reported_seq":2863,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.525123+0000","last_change":"2023-10-30T13:45:27.941070+0000","last_active":"2023-10-30T13:59:03.525123+0000","last_peered":"2023-10-30T13:59:03.525123+0000","last_clean":"2023-10-30T13:59:03.525123+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:59:03.525123+0000","last_undegraded":"2023-10-30T13:59:03.525123+0000","last_fullsized":"2023-10-30T13:59:03.525123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:40.443395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030849649999999998,"stat_sum":{"num_bytes":19227960,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":104,"num_read_kb":7536,"num_write":633,"num_write_kb":20026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"56'695","reported_seq":2879,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.509759+0000","last_change":"2023-10-30T13:45:28.597665+0000","last_active":"2023-10-30T13:59:03.509759+0000","last_peered":"2023-10-30T13:59:03.509759+0000","last_clean":"2023-10-30T13:59:03.509759+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:59:03.509759+0000","last_undegraded":"2023-10-30T13:59:03.509759+0000","last_fullsized":"2023-10-30T13:59:03.509759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:44.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082986029999999999,"stat_sum":{"num_bytes":10009773,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":91,"num_read_kb":11,"num_write":659,"num_write_kb":29217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"56'670","reported_seq":2896,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.521685+0000","last_change":"2023-10-30T13:45:28.780454+0000","last_active":"2023-10-30T13:59:03.521685+0000","last_peered":"2023-10-30T13:59:03.521685+0000","last_clean":"2023-10-30T13:59:03.521685+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:59:03.521685+0000","last_undegraded":"2023-10-30T13:59:03.521685+0000","last_fullsized":"2023-10-30T13:59:03.521685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021728070000000001,"stat_sum":{"num_bytes":10221735,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":132,"num_read_kb":1147,"num_write":641,"num_write_kb":24778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"56'666","reported_seq":2882,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.530531+0000","last_change":"2023-10-30T13:45:28.601219+0000","last_active":"2023-10-30T13:59:03.530531+0000","last_peered":"2023-10-30T13:59:03.530531+0000","last_clean":"2023-10-30T13:59:03.530531+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:59:03.530531+0000","last_undegraded":"2023-10-30T13:59:03.530531+0000","last_fullsized":"2023-10-30T13:59:03.530531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:53.268188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010045174,"stat_sum":{"num_bytes":10718551,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":124,"num_read_kb":4115,"num_write":615,"num_write_kb":24187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"56'617","reported_seq":2823,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.515899+0000","last_change":"2023-10-30T13:45:28.050787+0000","last_active":"2023-10-30T13:59:03.515899+0000","last_peered":"2023-10-30T13:59:03.515899+0000","last_clean":"2023-10-30T13:59:03.515899+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:59:03.515899+0000","last_undegraded":"2023-10-30T13:59:03.515899+0000","last_fullsized":"2023-10-30T13:59:03.515899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:02.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047277760000000004,"stat_sum":{"num_bytes":20230336,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":113,"num_read_kb":28,"num_write":585,"num_write_kb":25967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"56'612","reported_seq":2802,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.475584+0000","last_change":"2023-10-30T13:45:28.050006+0000","last_active":"2023-10-30T13:59:03.475584+0000","last_peered":"2023-10-30T13:59:03.475584+0000","last_clean":"2023-10-30T13:59:03.475584+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:59:03.475584+0000","last_undegraded":"2023-10-30T13:59:03.475584+0000","last_fullsized":"2023-10-30T13:59:03.475584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:30.724902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00429868,"stat_sum":{"num_bytes":2384023,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":97,"num_read_kb":17,"num_write":586,"num_write_kb":9055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"56'632","reported_seq":2841,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.519639+0000","last_change":"2023-10-30T13:45:27.946083+0000","last_active":"2023-10-30T13:59:03.519639+0000","last_peered":"2023-10-30T13:59:03.519639+0000","last_clean":"2023-10-30T13:59:03.519639+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:59:03.519639+0000","last_undegraded":"2023-10-30T13:59:03.519639+0000","last_fullsized":"2023-10-30T13:59:03.519639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:19.253547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066533010000000004,"stat_sum":{"num_bytes":18341132,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":8208,"num_write":596,"num_write_kb":32418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"56'648","reported_seq":2855,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.493278+0000","last_change":"2023-10-30T13:45:28.780475+0000","last_active":"2023-10-30T13:59:03.493278+0000","last_peered":"2023-10-30T13:59:03.493278+0000","last_clean":"2023-10-30T13:59:03.493278+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:59:03.493278+0000","last_undegraded":"2023-10-30T13:59:03.493278+0000","last_fullsized":"2023-10-30T13:59:03.493278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002338237,"stat_sum":{"num_bytes":14177749,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":114,"num_read_kb":12311,"num_write":601,"num_write_kb":37081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"56'736","reported_seq":2950,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.487153+0000","last_change":"2023-10-30T13:45:27.940425+0000","last_active":"2023-10-30T13:59:03.487153+0000","last_peered":"2023-10-30T13:59:03.487153+0000","last_clean":"2023-10-30T13:59:03.487153+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:59:03.487153+0000","last_undegraded":"2023-10-30T13:59:03.487153+0000","last_fullsized":"2023-10-30T13:59:03.487153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029248669999999998,"stat_sum":{"num_bytes":22804734,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":119,"num_read_kb":24041,"num_write":691,"num_write_kb":57342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"56'716","reported_seq":2923,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.497802+0000","last_change":"2023-10-30T13:45:27.934641+0000","last_active":"2023-10-30T13:59:03.497802+0000","last_peered":"2023-10-30T13:59:03.497802+0000","last_clean":"2023-10-30T13:59:03.497802+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:59:03.497802+0000","last_undegraded":"2023-10-30T13:59:03.497802+0000","last_fullsized":"2023-10-30T13:59:03.497802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021050410000000002,"stat_sum":{"num_bytes":18271578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":112,"num_read_kb":7334,"num_write":676,"num_write_kb":39770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"56'732","reported_seq":2949,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.524764+0000","last_change":"2023-10-30T13:45:27.943054+0000","last_active":"2023-10-30T13:59:03.524764+0000","last_peered":"2023-10-30T13:59:03.524764+0000","last_clean":"2023-10-30T13:59:03.524764+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:59:03.524764+0000","last_undegraded":"2023-10-30T13:59:03.524764+0000","last_fullsized":"2023-10-30T13:59:03.524764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:29.930187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052579080000000004,"stat_sum":{"num_bytes":35259912,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":123,"num_read_kb":12306,"num_write":693,"num_write_kb":43349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"56'710","reported_seq":2906,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.528575+0000","last_change":"2023-10-30T13:45:27.950182+0000","last_active":"2023-10-30T13:59:03.528575+0000","last_peered":"2023-10-30T13:59:03.528575+0000","last_clean":"2023-10-30T13:59:03.528575+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:59:03.528575+0000","last_undegraded":"2023-10-30T13:59:03.528575+0000","last_fullsized":"2023-10-30T13:59:03.528575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:09.257269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052923609999999998,"stat_sum":{"num_bytes":14461920,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":103,"num_read_kb":4113,"num_write":659,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"56'661","reported_seq":2874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.492542+0000","last_change":"2023-10-30T13:45:28.487735+0000","last_active":"2023-10-30T13:59:03.492542+0000","last_peered":"2023-10-30T13:59:03.492542+0000","last_clean":"2023-10-30T13:59:03.492542+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:59:03.492542+0000","last_undegraded":"2023-10-30T13:59:03.492542+0000","last_fullsized":"2023-10-30T13:59:03.492542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001369274,"stat_sum":{"num_bytes":15414431,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":118,"num_read_kb":4123,"num_write":623,"num_write_kb":31369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"56'728","reported_seq":2955,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.507685+0000","last_change":"2023-10-30T13:45:28.600018+0000","last_active":"2023-10-30T13:59:03.507685+0000","last_peered":"2023-10-30T13:59:03.507685+0000","last_clean":"2023-10-30T13:59:03.507685+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:59:03.507685+0000","last_undegraded":"2023-10-30T13:59:03.507685+0000","last_fullsized":"2023-10-30T13:59:03.507685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:15.777688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092525119999999992,"stat_sum":{"num_bytes":8879597,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":134,"num_read_kb":8209,"num_write":693,"num_write_kb":31072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"56'667","reported_seq":2860,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.519082+0000","last_change":"2023-10-30T13:45:28.051528+0000","last_active":"2023-10-30T13:59:03.519082+0000","last_peered":"2023-10-30T13:59:03.519082+0000","last_clean":"2023-10-30T13:59:03.519082+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:59:03.519082+0000","last_undegraded":"2023-10-30T13:59:03.519082+0000","last_fullsized":"2023-10-30T13:59:03.519082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:24.699943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00502631,"stat_sum":{"num_bytes":6211385,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":100,"num_read_kb":8209,"num_write":620,"num_write_kb":19966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"56'692","reported_seq":2898,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.508157+0000","last_change":"2023-10-30T13:45:27.940012+0000","last_active":"2023-10-30T13:59:03.508157+0000","last_peered":"2023-10-30T13:59:03.508157+0000","last_clean":"2023-10-30T13:59:03.508157+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:59:03.508157+0000","last_undegraded":"2023-10-30T13:59:03.508157+0000","last_fullsized":"2023-10-30T13:59:03.508157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026024469999999999,"stat_sum":{"num_bytes":18097476,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":111,"num_read_kb":21,"num_write":651,"num_write_kb":32402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"56'705","reported_seq":2904,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.524590+0000","last_change":"2023-10-30T13:45:27.947213+0000","last_active":"2023-10-30T13:59:03.524590+0000","last_peered":"2023-10-30T13:59:03.524590+0000","last_clean":"2023-10-30T13:59:03.524590+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:59:03.524590+0000","last_undegraded":"2023-10-30T13:59:03.524590+0000","last_fullsized":"2023-10-30T13:59:03.524590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:24.312446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054353270000000002,"stat_sum":{"num_bytes":10272429,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":105,"num_read_kb":4117,"num_write":665,"num_write_kb":16082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"56'701","reported_seq":2909,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.510809+0000","last_change":"2023-10-30T13:45:28.781012+0000","last_active":"2023-10-30T13:59:03.510809+0000","last_peered":"2023-10-30T13:59:03.510809+0000","last_clean":"2023-10-30T13:59:03.510809+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:59:03.510809+0000","last_undegraded":"2023-10-30T13:59:03.510809+0000","last_fullsized":"2023-10-30T13:59:03.510809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025617359999999998,"stat_sum":{"num_bytes":14333972,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":114,"num_read_kb":8213,"num_write":665,"num_write_kb":36470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"56'709","reported_seq":2930,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.516528+0000","last_change":"2023-10-30T13:45:27.942487+0000","last_active":"2023-10-30T13:59:03.516528+0000","last_peered":"2023-10-30T13:59:03.516528+0000","last_clean":"2023-10-30T13:59:03.516528+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:59:03.516528+0000","last_undegraded":"2023-10-30T13:59:03.516528+0000","last_fullsized":"2023-10-30T13:59:03.516528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:39.369789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004034279,"stat_sum":{"num_bytes":1595822,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":126,"num_read_kb":16,"num_write":676,"num_write_kb":12551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"56'674","reported_seq":2887,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.511918+0000","last_change":"2023-10-30T13:45:28.782023+0000","last_active":"2023-10-30T13:59:03.511918+0000","last_peered":"2023-10-30T13:59:03.511918+0000","last_clean":"2023-10-30T13:59:03.511918+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:59:03.511918+0000","last_undegraded":"2023-10-30T13:59:03.511918+0000","last_fullsized":"2023-10-30T13:59:03.511918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030593479999999999,"stat_sum":{"num_bytes":7858557,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":117,"num_read_kb":4108,"num_write":634,"num_write_kb":18163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"56'722","reported_seq":2928,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.529523+0000","last_change":"2023-10-30T13:45:28.051170+0000","last_active":"2023-10-30T13:59:03.529523+0000","last_peered":"2023-10-30T13:59:03.529523+0000","last_clean":"2023-10-30T13:59:03.529523+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:59:03.529523+0000","last_undegraded":"2023-10-30T13:59:03.529523+0000","last_fullsized":"2023-10-30T13:59:03.529523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":722,"log_dups_size":0,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:05.790369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049754609999999996,"stat_sum":{"num_bytes":9799460,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":111,"num_read_kb":24,"num_write":679,"num_write_kb":16350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"56'668","reported_seq":2874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.494238+0000","last_change":"2023-10-30T13:45:27.935505+0000","last_active":"2023-10-30T13:59:03.494238+0000","last_peered":"2023-10-30T13:59:03.494238+0000","last_clean":"2023-10-30T13:59:03.494238+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:59:03.494238+0000","last_undegraded":"2023-10-30T13:59:03.494238+0000","last_fullsized":"2023-10-30T13:59:03.494238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028468439999999999,"stat_sum":{"num_bytes":33375622,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":110,"num_read_kb":16400,"num_write":628,"num_write_kb":35724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"56'656","reported_seq":2863,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.514834+0000","last_change":"2023-10-30T13:45:28.781733+0000","last_active":"2023-10-30T13:59:03.514834+0000","last_peered":"2023-10-30T13:59:03.514834+0000","last_clean":"2023-10-30T13:59:03.514834+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:59:03.514834+0000","last_undegraded":"2023-10-30T13:59:03.514834+0000","last_fullsized":"2023-10-30T13:59:03.514834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002839121,"stat_sum":{"num_bytes":9811427,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":112,"num_read_kb":10701,"num_write":625,"num_write_kb":15312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"56'687","reported_seq":2892,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.505977+0000","last_change":"2023-10-30T13:45:27.941228+0000","last_active":"2023-10-30T13:59:03.505977+0000","last_peered":"2023-10-30T13:59:03.505977+0000","last_clean":"2023-10-30T13:59:03.505977+0000","last_became_active":"2023-10-30T13:35:48.847825+0000","last_became_peered":"2023-10-30T13:35:48.847825+0000","last_unstale":"2023-10-30T13:59:03.505977+0000","last_undegraded":"2023-10-30T13:59:03.505977+0000","last_fullsized":"2023-10-30T13:59:03.505977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028647770000000002,"stat_sum":{"num_bytes":7231756,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":102,"num_read_kb":4848,"num_write":653,"num_write_kb":26441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"56'692","reported_seq":2907,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.517579+0000","last_change":"2023-10-30T13:45:27.948917+0000","last_active":"2023-10-30T13:59:03.517579+0000","last_peered":"2023-10-30T13:59:03.517579+0000","last_clean":"2023-10-30T13:59:03.517579+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:59:03.517579+0000","last_undegraded":"2023-10-30T13:59:03.517579+0000","last_fullsized":"2023-10-30T13:59:03.517579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:51.840613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060200970000000003,"stat_sum":{"num_bytes":22333767,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":120,"num_read_kb":4116,"num_write":656,"num_write_kb":43887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"56'782","reported_seq":3017,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.523035+0000","last_change":"2023-10-30T13:45:28.487621+0000","last_active":"2023-10-30T13:59:03.523035+0000","last_peered":"2023-10-30T13:59:03.523035+0000","last_clean":"2023-10-30T13:59:03.523035+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:59:03.523035+0000","last_undegraded":"2023-10-30T13:59:03.523035+0000","last_fullsized":"2023-10-30T13:59:03.523035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001263171,"stat_sum":{"num_bytes":10546872,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":139,"num_read_kb":4118,"num_write":747,"num_write_kb":34715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"56'712","reported_seq":2917,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.487198+0000","last_change":"2023-10-30T13:45:28.048779+0000","last_active":"2023-10-30T13:59:03.487198+0000","last_peered":"2023-10-30T13:59:03.487198+0000","last_clean":"2023-10-30T13:59:03.487198+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:59:03.487198+0000","last_undegraded":"2023-10-30T13:59:03.487198+0000","last_fullsized":"2023-10-30T13:59:03.487198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:18.375625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036881090000000002,"stat_sum":{"num_bytes":14815628,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":110,"num_read_kb":20,"num_write":677,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"56'680","reported_seq":2881,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.489514+0000","last_change":"2023-10-30T13:45:27.944400+0000","last_active":"2023-10-30T13:59:03.489514+0000","last_peered":"2023-10-30T13:59:03.489514+0000","last_clean":"2023-10-30T13:59:03.489514+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:59:03.489514+0000","last_undegraded":"2023-10-30T13:59:03.489514+0000","last_fullsized":"2023-10-30T13:59:03.489514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:12.964999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065387930000000002,"stat_sum":{"num_bytes":14536361,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":108,"num_read_kb":8117,"num_write":641,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"56'695","reported_seq":765,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382814+0000","last_change":"2023-10-30T13:35:07.286333+0000","last_active":"2023-10-30T13:58:51.382814+0000","last_peered":"2023-10-30T13:58:51.382814+0000","last_clean":"2023-10-30T13:58:51.382814+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:58:51.382814+0000","last_undegraded":"2023-10-30T13:58:51.382814+0000","last_fullsized":"2023-10-30T13:58:51.382814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45057,"num_write":652,"num_write_kb":16749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"56'672","reported_seq":2893,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.517431+0000","last_change":"2023-10-30T13:45:27.941002+0000","last_active":"2023-10-30T13:59:03.517431+0000","last_peered":"2023-10-30T13:59:03.517431+0000","last_clean":"2023-10-30T13:59:03.517431+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:59:03.517431+0000","last_undegraded":"2023-10-30T13:59:03.517431+0000","last_fullsized":"2023-10-30T13:59:03.517431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031306289999999998,"stat_sum":{"num_bytes":22886210,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":127,"num_read_kb":6230,"num_write":644,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"56'743","reported_seq":874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.377388+0000","last_change":"2023-10-30T13:35:07.303785+0000","last_active":"2023-10-30T13:58:51.377388+0000","last_peered":"2023-10-30T13:58:51.377388+0000","last_clean":"2023-10-30T13:58:51.377388+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:58:51.377388+0000","last_undegraded":"2023-10-30T13:58:51.377388+0000","last_fullsized":"2023-10-30T13:58:51.377388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:49.800608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21047876,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":114393,"num_write":693,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"56'146","reported_seq":263,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.093135+0000","last_change":"2023-10-30T13:59:09.093208+0000","last_active":"2023-10-30T13:59:09.093135+0000","last_peered":"2023-10-30T13:59:09.093135+0000","last_clean":"2023-10-30T13:59:09.093135+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:59:09.093135+0000","last_undegraded":"2023-10-30T13:59:09.093135+0000","last_fullsized":"2023-10-30T13:59:09.093135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'146","last_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_deep_scrub":"56'146","last_deep_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.093046+0000","objects_scrubbed":7,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:36.674466+0000","scrub_duration":0.022130225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297526,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":20619,"num_write":102,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33717,"num_omap_keys":64,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"56'792","reported_seq":3029,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.495992+0000","last_change":"2023-10-30T13:45:28.781311+0000","last_active":"2023-10-30T13:59:03.495992+0000","last_peered":"2023-10-30T13:59:03.495992+0000","last_clean":"2023-10-30T13:59:03.495992+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:59:03.495992+0000","last_undegraded":"2023-10-30T13:59:03.495992+0000","last_fullsized":"2023-10-30T13:59:03.495992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027758349999999999,"stat_sum":{"num_bytes":1695799,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":144,"num_read_kb":4120,"num_write":754,"num_write_kb":40525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"56'596","reported_seq":678,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:06.387053+0000","last_change":"2023-10-30T13:35:07.269723+0000","last_active":"2023-10-30T13:59:06.387053+0000","last_peered":"2023-10-30T13:59:06.387053+0000","last_clean":"2023-10-30T13:59:06.387053+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:59:06.387053+0000","last_undegraded":"2023-10-30T13:59:06.387053+0000","last_fullsized":"2023-10-30T13:59:06.387053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:30.890632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":96432,"num_write":542,"num_write_kb":20769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"56'650","reported_seq":2858,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.495353+0000","last_change":"2023-10-30T13:45:27.934297+0000","last_active":"2023-10-30T13:59:03.495353+0000","last_peered":"2023-10-30T13:59:03.495353+0000","last_clean":"2023-10-30T13:59:03.495353+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:59:03.495353+0000","last_undegraded":"2023-10-30T13:59:03.495353+0000","last_fullsized":"2023-10-30T13:59:03.495353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017556029999999999,"stat_sum":{"num_bytes":7038572,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":115,"num_read_kb":8211,"num_write":611,"num_write_kb":33441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"56'127","reported_seq":186,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.382513+0000","last_change":"2023-10-30T13:35:07.303295+0000","last_active":"2023-10-30T13:58:51.382513+0000","last_peered":"2023-10-30T13:58:51.382513+0000","last_clean":"2023-10-30T13:58:51.382513+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:58:51.382513+0000","last_undegraded":"2023-10-30T13:58:51.382513+0000","last_fullsized":"2023-10-30T13:58:51.382513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:15.894036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":91,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"56'697","reported_seq":2889,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.508175+0000","last_change":"2023-10-30T13:45:28.487297+0000","last_active":"2023-10-30T13:59:03.508175+0000","last_peered":"2023-10-30T13:59:03.508175+0000","last_clean":"2023-10-30T13:59:03.508175+0000","last_became_active":"2023-10-30T13:35:49.007835+0000","last_became_peered":"2023-10-30T13:35:49.007835+0000","last_unstale":"2023-10-30T13:59:03.508175+0000","last_undegraded":"2023-10-30T13:59:03.508175+0000","last_fullsized":"2023-10-30T13:59:03.508175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011632000000000001,"stat_sum":{"num_bytes":15868329,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":101,"num_read_kb":5707,"num_write":652,"num_write_kb":27262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"56'385","reported_seq":458,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.290710+0000","last_change":"2023-10-30T13:35:07.286839+0000","last_active":"2023-10-30T13:59:03.290710+0000","last_peered":"2023-10-30T13:59:03.290710+0000","last_clean":"2023-10-30T13:59:03.290710+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:59:03.290710+0000","last_undegraded":"2023-10-30T13:59:03.290710+0000","last_fullsized":"2023-10-30T13:59:03.290710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:15.178023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":61443,"num_write":345,"num_write_kb":20724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"56'656","reported_seq":2878,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.523910+0000","last_change":"2023-10-30T13:45:27.941812+0000","last_active":"2023-10-30T13:59:03.523910+0000","last_peered":"2023-10-30T13:59:03.523910+0000","last_clean":"2023-10-30T13:59:03.523910+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:59:03.523910+0000","last_undegraded":"2023-10-30T13:59:03.523910+0000","last_fullsized":"2023-10-30T13:59:03.523910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028607060000000002,"stat_sum":{"num_bytes":18966683,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":128,"num_read_kb":11963,"num_write":623,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"56'713","reported_seq":799,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.423461+0000","last_change":"2023-10-30T13:35:07.296146+0000","last_active":"2023-10-30T13:58:56.423461+0000","last_peered":"2023-10-30T13:58:56.423461+0000","last_clean":"2023-10-30T13:58:56.423461+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:58:56.423461+0000","last_undegraded":"2023-10-30T13:58:56.423461+0000","last_fullsized":"2023-10-30T13:58:56.423461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":49185,"num_write":677,"num_write_kb":16860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"56'667","reported_seq":2860,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.515274+0000","last_change":"2023-10-30T13:45:27.947116+0000","last_active":"2023-10-30T13:59:03.515274+0000","last_peered":"2023-10-30T13:59:03.515274+0000","last_clean":"2023-10-30T13:59:03.515274+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:59:03.515274+0000","last_undegraded":"2023-10-30T13:59:03.515274+0000","last_fullsized":"2023-10-30T13:59:03.515274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:44.472162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070741379999999998,"stat_sum":{"num_bytes":9996128,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":99,"num_read_kb":8209,"num_write":632,"num_write_kb":23925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"56'1162","reported_seq":1266,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:06.468095+0000","last_change":"2023-10-30T13:35:07.286499+0000","last_active":"2023-10-30T13:59:06.468095+0000","last_peered":"2023-10-30T13:59:06.468095+0000","last_clean":"2023-10-30T13:59:06.468095+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:59:06.468095+0000","last_undegraded":"2023-10-30T13:59:06.468095+0000","last_fullsized":"2023-10-30T13:59:06.468095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1162,"log_dups_size":0,"ondisk_log_size":1162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:20.749892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15639923,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":86059,"num_write":1111,"num_write_kb":20034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"56'671","reported_seq":2868,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.506440+0000","last_change":"2023-10-30T13:45:28.596691+0000","last_active":"2023-10-30T13:59:03.506440+0000","last_peered":"2023-10-30T13:59:03.506440+0000","last_clean":"2023-10-30T13:59:03.506440+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:59:03.506440+0000","last_undegraded":"2023-10-30T13:59:03.506440+0000","last_fullsized":"2023-10-30T13:59:03.506440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:49:41.351825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070810969999999997,"stat_sum":{"num_bytes":18214186,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":104,"num_read_kb":12299,"num_write":638,"num_write_kb":31808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"56'431","reported_seq":529,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:06.375636+0000","last_change":"2023-10-30T13:35:07.276852+0000","last_active":"2023-10-30T13:59:06.375636+0000","last_peered":"2023-10-30T13:59:06.375636+0000","last_clean":"2023-10-30T13:59:06.375636+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:59:06.375636+0000","last_undegraded":"2023-10-30T13:59:06.375636+0000","last_fullsized":"2023-10-30T13:59:06.375636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":77906,"num_write":384,"num_write_kb":12555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"56'701","reported_seq":2930,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.371396+0000","last_change":"2023-10-30T13:45:28.487276+0000","last_active":"2023-10-30T13:59:08.371396+0000","last_peered":"2023-10-30T13:59:08.371396+0000","last_clean":"2023-10-30T13:59:08.371396+0000","last_became_active":"2023-10-30T13:35:49.010460+0000","last_became_peered":"2023-10-30T13:35:49.010460+0000","last_unstale":"2023-10-30T13:59:08.371396+0000","last_undegraded":"2023-10-30T13:59:08.371396+0000","last_fullsized":"2023-10-30T13:59:08.371396+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001187553,"stat_sum":{"num_bytes":13816511,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":138,"num_read_kb":8210,"num_write":662,"num_write_kb":56040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"56'177","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.380435+0000","last_change":"2023-10-30T13:35:07.288133+0000","last_active":"2023-10-30T13:58:51.380435+0000","last_peered":"2023-10-30T13:58:51.380435+0000","last_clean":"2023-10-30T13:58:51.380435+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:58:51.380435+0000","last_undegraded":"2023-10-30T13:58:51.380435+0000","last_fullsized":"2023-10-30T13:58:51.380435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:27.582304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":632,"num_write":147,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"56'713","reported_seq":2932,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.519975+0000","last_change":"2023-10-30T13:45:28.599507+0000","last_active":"2023-10-30T13:59:03.519975+0000","last_peered":"2023-10-30T13:59:03.519975+0000","last_clean":"2023-10-30T13:59:03.519975+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:59:03.519975+0000","last_undegraded":"2023-10-30T13:59:03.519975+0000","last_fullsized":"2023-10-30T13:59:03.519975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:11.218437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092364379999999996,"stat_sum":{"num_bytes":16180540,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":126,"num_read_kb":8225,"num_write":670,"num_write_kb":34788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"56'171","reported_seq":234,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.363936+0000","last_change":"2023-10-30T13:35:07.302488+0000","last_active":"2023-10-30T13:58:51.363936+0000","last_peered":"2023-10-30T13:58:51.363936+0000","last_clean":"2023-10-30T13:58:51.363936+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:58:51.363936+0000","last_undegraded":"2023-10-30T13:58:51.363936+0000","last_fullsized":"2023-10-30T13:58:51.363936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:59.688281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4103,"num_write":119,"num_write_kb":4220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"56'659","reported_seq":2852,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.486380+0000","last_change":"2023-10-30T13:45:28.049648+0000","last_active":"2023-10-30T13:59:03.486380+0000","last_peered":"2023-10-30T13:59:03.486380+0000","last_clean":"2023-10-30T13:59:03.486380+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:59:03.486380+0000","last_undegraded":"2023-10-30T13:59:03.486380+0000","last_fullsized":"2023-10-30T13:59:03.486380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:03.640232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045350950000000003,"stat_sum":{"num_bytes":14031523,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":99,"num_read_kb":8208,"num_write":613,"num_write_kb":14857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"56'613","reported_seq":723,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.570273+0000","last_change":"2023-10-30T13:35:07.277685+0000","last_active":"2023-10-30T13:58:52.570273+0000","last_peered":"2023-10-30T13:58:52.570273+0000","last_clean":"2023-10-30T13:58:52.570273+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:58:52.570273+0000","last_undegraded":"2023-10-30T13:58:52.570273+0000","last_fullsized":"2023-10-30T13:58:52.570273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":130,"num_read_kb":41105,"num_write":605,"num_write_kb":12685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"56'737","reported_seq":2947,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.511310+0000","last_change":"2023-10-30T13:45:28.598560+0000","last_active":"2023-10-30T13:59:03.511310+0000","last_peered":"2023-10-30T13:59:03.511310+0000","last_clean":"2023-10-30T13:59:03.511310+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:59:03.511310+0000","last_undegraded":"2023-10-30T13:59:03.511310+0000","last_fullsized":"2023-10-30T13:59:03.511310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:25.254829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089036940000000002,"stat_sum":{"num_bytes":6058105,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":119,"num_read_kb":8210,"num_write":694,"num_write_kb":35073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'351","reported_seq":425,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.537629+0000","last_change":"2023-10-30T13:35:07.280106+0000","last_active":"2023-10-30T13:58:32.537629+0000","last_peered":"2023-10-30T13:58:32.537629+0000","last_clean":"2023-10-30T13:58:32.537629+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:58:32.537629+0000","last_undegraded":"2023-10-30T13:58:32.537629+0000","last_fullsized":"2023-10-30T13:58:32.537629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:55.237168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":28724,"num_write":357,"num_write_kb":8477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"56'655","reported_seq":2847,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.511426+0000","last_change":"2023-10-30T13:45:28.600359+0000","last_active":"2023-10-30T13:59:03.511426+0000","last_peered":"2023-10-30T13:59:03.511426+0000","last_clean":"2023-10-30T13:59:03.511426+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:59:03.511426+0000","last_undegraded":"2023-10-30T13:59:03.511426+0000","last_fullsized":"2023-10-30T13:59:03.511426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:40.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094781629999999995,"stat_sum":{"num_bytes":6414224,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":99,"num_read_kb":8205,"num_write":617,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"56'641","reported_seq":719,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:06.361145+0000","last_change":"2023-10-30T13:35:07.287148+0000","last_active":"2023-10-30T13:59:06.361145+0000","last_peered":"2023-10-30T13:59:06.361145+0000","last_clean":"2023-10-30T13:59:06.361145+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:59:06.361145+0000","last_undegraded":"2023-10-30T13:59:06.361145+0000","last_fullsized":"2023-10-30T13:59:06.361145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":33457,"num_write":586,"num_write_kb":12711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"56'676","reported_seq":2880,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.517182+0000","last_change":"2023-10-30T13:45:28.601986+0000","last_active":"2023-10-30T13:59:03.517182+0000","last_peered":"2023-10-30T13:59:03.517182+0000","last_clean":"2023-10-30T13:59:03.517182+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:59:03.517182+0000","last_undegraded":"2023-10-30T13:59:03.517182+0000","last_fullsized":"2023-10-30T13:59:03.517182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:59.052456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010320058,"stat_sum":{"num_bytes":13797516,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":111,"num_read_kb":8210,"num_write":631,"num_write_kb":35965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"56'401","reported_seq":476,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:06.393003+0000","last_change":"2023-10-30T13:35:07.279634+0000","last_active":"2023-10-30T13:59:06.393003+0000","last_peered":"2023-10-30T13:59:06.393003+0000","last_clean":"2023-10-30T13:59:06.393003+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:59:06.393003+0000","last_undegraded":"2023-10-30T13:59:06.393003+0000","last_fullsized":"2023-10-30T13:59:06.393003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388678,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":36872,"num_write":370,"num_write_kb":12516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"56'747","reported_seq":2976,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.524830+0000","last_change":"2023-10-30T13:45:27.948762+0000","last_active":"2023-10-30T13:59:03.524830+0000","last_peered":"2023-10-30T13:59:03.524830+0000","last_clean":"2023-10-30T13:59:03.524830+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:59:03.524830+0000","last_undegraded":"2023-10-30T13:59:03.524830+0000","last_fullsized":"2023-10-30T13:59:03.524830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.722813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066640720000000001,"stat_sum":{"num_bytes":18395347,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":135,"num_read_kb":4114,"num_write":711,"num_write_kb":24326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"56'222","reported_seq":291,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.554129+0000","last_change":"2023-10-30T13:35:07.282357+0000","last_active":"2023-10-30T13:58:32.554129+0000","last_peered":"2023-10-30T13:58:32.554129+0000","last_clean":"2023-10-30T13:58:32.554129+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:58:32.554129+0000","last_undegraded":"2023-10-30T13:58:32.554129+0000","last_fullsized":"2023-10-30T13:58:32.554129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":20494,"num_write":198,"num_write_kb":4363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"56'677","reported_seq":2876,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.514039+0000","last_change":"2023-10-30T13:45:28.598725+0000","last_active":"2023-10-30T13:59:03.514039+0000","last_peered":"2023-10-30T13:59:03.514039+0000","last_clean":"2023-10-30T13:59:03.514039+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:59:03.514039+0000","last_undegraded":"2023-10-30T13:59:03.514039+0000","last_fullsized":"2023-10-30T13:59:03.514039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:29.217863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084474579999999997,"stat_sum":{"num_bytes":18100965,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":106,"num_read_kb":8207,"num_write":635,"num_write_kb":40451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:57:42.172646+0000","last_change":"2023-10-30T13:34:55.138791+0000","last_active":"2023-10-30T13:57:42.172646+0000","last_peered":"2023-10-30T13:57:42.172646+0000","last_clean":"2023-10-30T13:57:42.172646+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:57:42.172646+0000","last_undegraded":"2023-10-30T13:57:42.172646+0000","last_fullsized":"2023-10-30T13:57:42.172646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:34:53.554510+0000","last_clean_scrub_stamp":"2023-10-30T13:34:53.554510+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:43.854792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'346","reported_seq":413,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.460048+0000","last_change":"2023-10-30T13:35:07.279410+0000","last_active":"2023-10-30T13:58:56.460048+0000","last_peered":"2023-10-30T13:58:56.460048+0000","last_clean":"2023-10-30T13:58:56.460048+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:58:56.460048+0000","last_undegraded":"2023-10-30T13:58:56.460048+0000","last_fullsized":"2023-10-30T13:58:56.460048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:04.800007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388674,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":20491,"num_write":340,"num_write_kb":8443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'681","reported_seq":2884,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.497940+0000","last_change":"2023-10-30T13:45:28.780750+0000","last_active":"2023-10-30T13:59:03.497940+0000","last_peered":"2023-10-30T13:59:03.497940+0000","last_clean":"2023-10-30T13:59:03.497940+0000","last_became_active":"2023-10-30T13:35:48.847905+0000","last_became_peered":"2023-10-30T13:35:48.847905+0000","last_unstale":"2023-10-30T13:59:03.497940+0000","last_undegraded":"2023-10-30T13:59:03.497940+0000","last_fullsized":"2023-10-30T13:59:03.497940+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002389721,"stat_sum":{"num_bytes":10749534,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":100,"num_read_kb":11,"num_write":647,"num_write_kb":20249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"56'353","reported_seq":440,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.425438+0000","last_change":"2023-10-30T13:35:07.301174+0000","last_active":"2023-10-30T13:58:56.425438+0000","last_peered":"2023-10-30T13:58:56.425438+0000","last_clean":"2023-10-30T13:58:56.425438+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:58:56.425438+0000","last_undegraded":"2023-10-30T13:58:56.425438+0000","last_fullsized":"2023-10-30T13:58:56.425438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:44.364659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":102412,"num_write":307,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"56'701","reported_seq":2914,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.498706+0000","last_change":"2023-10-30T13:45:27.939160+0000","last_active":"2023-10-30T13:59:03.498706+0000","last_peered":"2023-10-30T13:59:03.498706+0000","last_clean":"2023-10-30T13:59:03.498706+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:59:03.498706+0000","last_undegraded":"2023-10-30T13:59:03.498706+0000","last_fullsized":"2023-10-30T13:59:03.498706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018610090000000001,"stat_sum":{"num_bytes":24271212,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":117,"num_read_kb":10,"num_write":671,"num_write_kb":33782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"56'1498","reported_seq":2023,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:52.567808+0000","last_change":"2023-10-30T13:35:07.282042+0000","last_active":"2023-10-30T13:58:52.567808+0000","last_peered":"2023-10-30T13:58:52.567808+0000","last_clean":"2023-10-30T13:58:52.567808+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:58:52.567808+0000","last_undegraded":"2023-10-30T13:58:52.567808+0000","last_fullsized":"2023-10-30T13:58:52.567808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959716,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":475,"num_read_kb":12479,"num_write":1460,"num_write_kb":17319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'662","reported_seq":2874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.484748+0000","last_change":"2023-10-30T13:45:27.934841+0000","last_active":"2023-10-30T13:59:03.484748+0000","last_peered":"2023-10-30T13:59:03.484748+0000","last_clean":"2023-10-30T13:59:03.484748+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:59:03.484748+0000","last_undegraded":"2023-10-30T13:59:03.484748+0000","last_fullsized":"2023-10-30T13:59:03.484748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022228600000000001,"stat_sum":{"num_bytes":9925305,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":116,"num_read_kb":4117,"num_write":612,"num_write_kb":24793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'630","reported_seq":703,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:06.395365+0000","last_change":"2023-10-30T13:35:07.281884+0000","last_active":"2023-10-30T13:59:06.395365+0000","last_peered":"2023-10-30T13:59:06.395365+0000","last_clean":"2023-10-30T13:59:06.395365+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:59:06.395365+0000","last_undegraded":"2023-10-30T13:59:06.395365+0000","last_fullsized":"2023-10-30T13:59:06.395365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:00.597191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":61440,"num_write":567,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'717","reported_seq":2913,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.499195+0000","last_change":"2023-10-30T13:45:27.945262+0000","last_active":"2023-10-30T13:59:03.499195+0000","last_peered":"2023-10-30T13:59:03.499195+0000","last_clean":"2023-10-30T13:59:03.499195+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:59:03.499195+0000","last_undegraded":"2023-10-30T13:59:03.499195+0000","last_fullsized":"2023-10-30T13:59:03.499195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:31.086225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066434720000000001,"stat_sum":{"num_bytes":26554596,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":101,"num_read_kb":3528,"num_write":689,"num_write_kb":37408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"56'729","reported_seq":803,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.710386+0000","last_change":"2023-10-30T13:59:08.710437+0000","last_active":"2023-10-30T13:59:08.710386+0000","last_peered":"2023-10-30T13:59:08.710386+0000","last_clean":"2023-10-30T13:59:08.710386+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:59:08.710386+0000","last_undegraded":"2023-10-30T13:59:08.710386+0000","last_fullsized":"2023-10-30T13:59:08.710386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'729","last_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_deep_scrub":"56'729","last_deep_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_clean_scrub_stamp":"2023-10-30T13:59:08.710307+0000","objects_scrubbed":3,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:48.485825+0000","scrub_duration":0.018720046000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":18914,"num_write":694,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10538,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"56'698","reported_seq":2901,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.519445+0000","last_change":"2023-10-30T13:45:27.949717+0000","last_active":"2023-10-30T13:59:03.519445+0000","last_peered":"2023-10-30T13:59:03.519445+0000","last_clean":"2023-10-30T13:59:03.519445+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:59:03.519445+0000","last_undegraded":"2023-10-30T13:59:03.519445+0000","last_fullsized":"2023-10-30T13:59:03.519445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:55.918379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064085849999999996,"stat_sum":{"num_bytes":10071990,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":4109,"num_write":659,"num_write_kb":40765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"56'756","reported_seq":826,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.368827+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:58:51.368827+0000","last_peered":"2023-10-30T13:58:51.368827+0000","last_clean":"2023-10-30T13:58:51.368827+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:58:51.368827+0000","last_undegraded":"2023-10-30T13:58:51.368827+0000","last_fullsized":"2023-10-30T13:58:51.368827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:15.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":707,"num_write_kb":16664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"56'675","reported_seq":2879,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.491071+0000","last_change":"2023-10-30T13:45:27.934883+0000","last_active":"2023-10-30T13:59:03.491071+0000","last_peered":"2023-10-30T13:59:03.491071+0000","last_clean":"2023-10-30T13:59:03.491071+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:59:03.491071+0000","last_undegraded":"2023-10-30T13:59:03.491071+0000","last_fullsized":"2023-10-30T13:59:03.491071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001735594,"stat_sum":{"num_bytes":6415761,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":108,"num_read_kb":4114,"num_write":644,"num_write_kb":12898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"56'279","reported_seq":358,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:06.444736+0000","last_change":"2023-10-30T13:35:07.294333+0000","last_active":"2023-10-30T13:59:06.444736+0000","last_peered":"2023-10-30T13:59:06.444736+0000","last_clean":"2023-10-30T13:59:06.444736+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:59:06.444736+0000","last_undegraded":"2023-10-30T13:59:06.444736+0000","last_fullsized":"2023-10-30T13:59:06.444736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":698,"num_write":517,"num_write_kb":2842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"56'618","reported_seq":2820,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.512401+0000","last_change":"2023-10-30T13:45:28.781685+0000","last_active":"2023-10-30T13:59:03.512401+0000","last_peered":"2023-10-30T13:59:03.512401+0000","last_clean":"2023-10-30T13:59:03.512401+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:59:03.512401+0000","last_undegraded":"2023-10-30T13:59:03.512401+0000","last_fullsized":"2023-10-30T13:59:03.512401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002858771,"stat_sum":{"num_bytes":14417139,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":106,"num_read_kb":4110,"num_write":585,"num_write_kb":24357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"56'119","reported_seq":196,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.385695+0000","last_change":"2023-10-30T13:35:07.302782+0000","last_active":"2023-10-30T13:58:51.385695+0000","last_peered":"2023-10-30T13:58:51.385695+0000","last_clean":"2023-10-30T13:58:51.385695+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:58:51.385695+0000","last_undegraded":"2023-10-30T13:58:51.385695+0000","last_fullsized":"2023-10-30T13:58:51.385695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:54.888544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":20522,"num_write":112,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"56'688","reported_seq":2896,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.490712+0000","last_change":"2023-10-30T13:45:27.941313+0000","last_active":"2023-10-30T13:59:03.490712+0000","last_peered":"2023-10-30T13:59:03.490712+0000","last_clean":"2023-10-30T13:59:03.490712+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:59:03.490712+0000","last_undegraded":"2023-10-30T13:59:03.490712+0000","last_fullsized":"2023-10-30T13:59:03.490712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027315170000000001,"stat_sum":{"num_bytes":18657914,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":112,"num_read_kb":8214,"num_write":646,"num_write_kb":32163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"56'145","reported_seq":214,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:59.377440+0000","last_change":"2023-10-30T13:35:07.299792+0000","last_active":"2023-10-30T13:58:59.377440+0000","last_peered":"2023-10-30T13:58:59.377440+0000","last_clean":"2023-10-30T13:58:59.377440+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:58:59.377440+0000","last_undegraded":"2023-10-30T13:58:59.377440+0000","last_fullsized":"2023-10-30T13:58:59.377440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:02:05.920076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":30625,"num_write":106,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"56'679","reported_seq":2872,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.488024+0000","last_change":"2023-10-30T13:45:27.946293+0000","last_active":"2023-10-30T13:59:03.488024+0000","last_peered":"2023-10-30T13:59:03.488024+0000","last_clean":"2023-10-30T13:59:03.488024+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:59:03.488024+0000","last_undegraded":"2023-10-30T13:59:03.488024+0000","last_fullsized":"2023-10-30T13:59:03.488024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:04.199638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075826330000000001,"stat_sum":{"num_bytes":5755271,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":98,"num_read_kb":7,"num_write":639,"num_write_kb":16186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'941","reported_seq":1008,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:59.304406+0000","last_change":"2023-10-30T13:35:07.285460+0000","last_active":"2023-10-30T13:58:59.304406+0000","last_peered":"2023-10-30T13:58:59.304406+0000","last_clean":"2023-10-30T13:58:59.304406+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:58:59.304406+0000","last_undegraded":"2023-10-30T13:58:59.304406+0000","last_fullsized":"2023-10-30T13:58:59.304406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":20487,"num_write":909,"num_write_kb":8631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"56'727","reported_seq":2935,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.495804+0000","last_change":"2023-10-30T13:45:28.782068+0000","last_active":"2023-10-30T13:59:03.495804+0000","last_peered":"2023-10-30T13:59:03.495804+0000","last_clean":"2023-10-30T13:59:03.495804+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:59:03.495804+0000","last_undegraded":"2023-10-30T13:59:03.495804+0000","last_fullsized":"2023-10-30T13:59:03.495804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029726169999999999,"stat_sum":{"num_bytes":19889578,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":112,"num_read_kb":12306,"num_write":688,"num_write_kb":31224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"56'477","reported_seq":551,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:04.304887+0000","last_change":"2023-10-30T13:35:07.285169+0000","last_active":"2023-10-30T13:59:04.304887+0000","last_peered":"2023-10-30T13:59:04.304887+0000","last_clean":"2023-10-30T13:59:04.304887+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:59:04.304887+0000","last_undegraded":"2023-10-30T13:59:04.304887+0000","last_fullsized":"2023-10-30T13:59:04.304887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":53262,"num_write":506,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"56'691","reported_seq":2897,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.493504+0000","last_change":"2023-10-30T13:45:28.781107+0000","last_active":"2023-10-30T13:59:03.493504+0000","last_peered":"2023-10-30T13:59:03.493504+0000","last_clean":"2023-10-30T13:59:03.493504+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:59:03.493504+0000","last_undegraded":"2023-10-30T13:59:03.493504+0000","last_fullsized":"2023-10-30T13:59:03.493504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002600925,"stat_sum":{"num_bytes":9733947,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":111,"num_read_kb":4112,"num_write":647,"num_write_kb":24811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"56'1285","reported_seq":1372,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:32.561985+0000","last_change":"2023-10-30T13:35:07.301683+0000","last_active":"2023-10-30T13:58:32.561985+0000","last_peered":"2023-10-30T13:58:32.561985+0000","last_clean":"2023-10-30T13:58:32.561985+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:58:32.561985+0000","last_undegraded":"2023-10-30T13:58:32.561985+0000","last_fullsized":"2023-10-30T13:58:32.561985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.707120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":81960,"num_write":1252,"num_write_kb":21130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"56'715","reported_seq":2918,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.499856+0000","last_change":"2023-10-30T13:45:28.050416+0000","last_active":"2023-10-30T13:59:03.499856+0000","last_peered":"2023-10-30T13:59:03.499856+0000","last_clean":"2023-10-30T13:59:03.499856+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:59:03.499856+0000","last_undegraded":"2023-10-30T13:59:03.499856+0000","last_fullsized":"2023-10-30T13:59:03.499856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:34.848723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051869250000000002,"stat_sum":{"num_bytes":18626842,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":5015,"num_write":667,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"56'657","reported_seq":2873,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.523059+0000","last_change":"2023-10-30T13:45:27.949465+0000","last_active":"2023-10-30T13:59:03.523059+0000","last_peered":"2023-10-30T13:59:03.523059+0000","last_clean":"2023-10-30T13:59:03.523059+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:59:03.523059+0000","last_undegraded":"2023-10-30T13:59:03.523059+0000","last_fullsized":"2023-10-30T13:59:03.523059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:57.261270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061609869999999997,"stat_sum":{"num_bytes":14245271,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":123,"num_read_kb":12313,"num_write":620,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"56'231","reported_seq":303,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.389390+0000","last_change":"2023-10-30T13:35:07.300290+0000","last_active":"2023-10-30T13:58:51.389390+0000","last_peered":"2023-10-30T13:58:51.389390+0000","last_clean":"2023-10-30T13:58:51.389390+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:58:51.389390+0000","last_undegraded":"2023-10-30T13:58:51.389390+0000","last_fullsized":"2023-10-30T13:58:51.389390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:04.049136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":24606,"num_write":206,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"56'713","reported_seq":2918,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.528269+0000","last_change":"2023-10-30T13:45:27.948077+0000","last_active":"2023-10-30T13:59:03.528269+0000","last_peered":"2023-10-30T13:59:03.528269+0000","last_clean":"2023-10-30T13:59:03.528269+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:59:03.528269+0000","last_undegraded":"2023-10-30T13:59:03.528269+0000","last_fullsized":"2023-10-30T13:59:03.528269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:48.941885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061457359999999997,"stat_sum":{"num_bytes":11313833,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":112,"num_read_kb":16,"num_write":670,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"56'950","reported_seq":1044,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:56.465087+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:58:56.465087+0000","last_peered":"2023-10-30T13:58:56.465087+0000","last_clean":"2023-10-30T13:58:56.465087+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:58:56.465087+0000","last_undegraded":"2023-10-30T13:58:56.465087+0000","last_fullsized":"2023-10-30T13:58:56.465087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:50:14.034257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":103644,"num_write":908,"num_write_kb":16901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"56'678","reported_seq":2895,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.525139+0000","last_change":"2023-10-30T13:45:27.941557+0000","last_active":"2023-10-30T13:59:03.525139+0000","last_peered":"2023-10-30T13:59:03.525139+0000","last_clean":"2023-10-30T13:59:03.525139+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:59:03.525139+0000","last_undegraded":"2023-10-30T13:59:03.525139+0000","last_fullsized":"2023-10-30T13:59:03.525139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029730300000000002,"stat_sum":{"num_bytes":19886092,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":121,"num_read_kb":4113,"num_write":647,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"56'586","reported_seq":667,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.912960+0000","last_change":"2023-10-30T13:35:07.286860+0000","last_active":"2023-10-30T13:59:08.912960+0000","last_peered":"2023-10-30T13:59:08.912960+0000","last_clean":"2023-10-30T13:59:08.912960+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:59:08.912960+0000","last_undegraded":"2023-10-30T13:59:08.912960+0000","last_fullsized":"2023-10-30T13:59:08.912960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":72355,"num_write":540,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"56'660","reported_seq":2855,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.913025+0000","last_change":"2023-10-30T13:45:28.781420+0000","last_active":"2023-10-30T13:59:08.913025+0000","last_peered":"2023-10-30T13:59:08.913025+0000","last_clean":"2023-10-30T13:59:08.913025+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:59:08.913025+0000","last_undegraded":"2023-10-30T13:59:08.913025+0000","last_fullsized":"2023-10-30T13:59:08.913025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026777319999999999,"stat_sum":{"num_bytes":7974576,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":98,"num_read_kb":4107,"num_write":628,"num_write_kb":17606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"56'356","reported_seq":452,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.383226+0000","last_change":"2023-10-30T13:35:07.293698+0000","last_active":"2023-10-30T13:58:51.383226+0000","last_peered":"2023-10-30T13:58:51.383226+0000","last_clean":"2023-10-30T13:58:51.383226+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:58:51.383226+0000","last_undegraded":"2023-10-30T13:58:51.383226+0000","last_fullsized":"2023-10-30T13:58:51.383226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4418509,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":57427,"num_write":339,"num_write_kb":8706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":925185184,"num_objects":10324,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":7268,"num_read_kb":400185,"num_write":41527,"num_write_kb":1950152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1902518272,"data_stored":1848830272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43971,"ondisk_log_size":43971,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":259497664,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":1553,"num_read_kb":1442867,"num_write":16449,"num_write_kb":376834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":44255,"num_omap_keys":84,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":519159808,"data_stored":518995328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17255,"ondisk_log_size":17255,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509736,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441460,"kb_used_data":333932,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93930380,"statfs":{"total":96636764160,"available":96184709120,"internally_reserved":0,"allocated":341946368,"data_stored":333745462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":110098891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85199999999999998,"15min":0.81200000000000006},"min":{"1min":0.433,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.1830000000000001,"5min":0.96299999999999997,"15min":0.90600000000000003},"min":{"1min":0.53500000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.3499999999999996,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.86199999999999999,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.0289999999999999,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.02,"5min":0.95499999999999996,"15min":0.92300000000000004},"min":{"1min":0.57399999999999995,"5min":0.45200000000000001,"15min":0.33400000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.625},"last":1.0489999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0389999999999999,"15min":0.97299999999999998},"min":{"1min":0.626,"5min":0.51300000000000001,"15min":0.32000000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.91700000000000004,"15min":0.94499999999999995},"min":{"1min":0.63300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":4.1120000000000001},"last":0.88900000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.75600000000000001,"5min":0.55300000000000005,"15min":0.45900000000000002},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":4.2069999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":4.4189999999999996,"5min":1.645,"15min":1.204},"min":{"1min":0.83299999999999996,"5min":0.47199999999999998,"15min":0.44400000000000001},"max":{"1min":59.844999999999999,"5min":59.844999999999999,"15min":59.844999999999999},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1080000000000001,"15min":1.089},"min":{"1min":0.85899999999999999,"5min":0.623,"15min":0.38100000000000001},"max":{"1min":1.853,"5min":1.853,"15min":3.1059999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.3220000000000001,"15min":1.2769999999999999},"min":{"1min":1.137,"5min":0.754,"15min":0.42399999999999999},"max":{"1min":1.907,"5min":1.907,"15min":3.198},"last":1.1930000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0049999999999999,"15min":1.093},"min":{"1min":0.65600000000000003,"5min":0.53100000000000003,"15min":0.40300000000000002},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":4.2759999999999998},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.228,"5min":1.258,"15min":1.238},"min":{"1min":0.80000000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5329999999999999,"5min":1.8380000000000001,"15min":3.4380000000000002},"last":1.1499999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92200000000000004,"15min":1.091},"min":{"1min":0.55400000000000005,"5min":0.48999999999999999,"15min":0.48399999999999999},"max":{"1min":1.3109999999999999,"5min":1.429,"15min":3.9630000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.458,"5min":1.131,"15min":1.216},"min":{"1min":0.84399999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":7.7060000000000004,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":1.018}]}]},{"osd":3,"up_from":15,"seq":64424509735,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416492,"kb_used_data":317284,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93955348,"statfs":{"total":96636764160,"available":96210276352,"internally_reserved":0,"allocated":324898816,"data_stored":316663032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":101579208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":9.3320000000000007},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0369999999999999,"15min":0.72399999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.26100000000000001},"max":{"1min":0.71599999999999997,"5min":38.082999999999998,"15min":38.082999999999998},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.70199999999999996,"15min":0.60699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":11.599,"15min":11.599},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.70899999999999996,"15min":0.66100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":15.715999999999999,"15min":15.715999999999999},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.72699999999999998,"15min":0.66700000000000004},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.28999999999999998},"max":{"1min":0.83899999999999997,"5min":11.646000000000001,"15min":12.351000000000001},"last":0.626},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.755,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.83399999999999996,"5min":15.641999999999999,"15min":15.641999999999999},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.016,"15min":0.92000000000000004},"min":{"1min":0.59999999999999998,"5min":0.55400000000000005,"15min":0.51600000000000001},"max":{"1min":0.98899999999999999,"5min":15.759,"15min":15.759},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.298,"15min":1.0640000000000001},"min":{"1min":0.58499999999999996,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":38.137999999999998,"15min":38.137999999999998},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.4139999999999999,"15min":1.097},"min":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.52600000000000002},"max":{"1min":1.036,"5min":40.808,"15min":40.808},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.0309999999999999,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.55800000000000005,"15min":0.49299999999999999},"max":{"1min":1.0660000000000001,"5min":16.498000000000001,"15min":19.835999999999999},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.96499999999999997,"15min":0.91800000000000004},"min":{"1min":0.621,"5min":0.56799999999999995,"15min":0.53500000000000003},"max":{"1min":1.2370000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.998,"15min":0.96099999999999997},"min":{"1min":0.68500000000000005,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":1.091,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.84499999999999997}]},{"osd":7,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1020000000000001,"15min":0.95799999999999996},"min":{"1min":0.72499999999999998,"5min":0.60599999999999998,"15min":0.59999999999999998},"max":{"1min":1.2549999999999999,"5min":16.082999999999998,"15min":16.082999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.88,"5min":1.5609999999999999,"15min":1.111},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.58199999999999996},"max":{"1min":1.228,"5min":52.902999999999999,"15min":52.902999999999999},"last":0.99099999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232160,"kb_used_data":187096,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94139680,"statfs":{"total":96636764160,"available":96399032320,"internally_reserved":0,"allocated":191586304,"data_stored":186614303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.45700000000000002},"min":{"1min":0.32100000000000001,"5min":0.251,"15min":0.246},"max":{"1min":0.83199999999999996,"5min":3.407,"15min":3.8969999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.371,"5min":0.35299999999999998,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":5.6980000000000004,"15min":5.6980000000000004},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51000000000000001,"15min":0.60099999999999998},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68100000000000005,"5min":5.5750000000000002,"15min":36.109000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58899999999999997,"5min":5.4569999999999999,"15min":8.1890000000000001},"last":0.498}]},{"osd":3,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.63500000000000001,"15min":0.68999999999999995},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.29199999999999998},"max":{"1min":0.73999999999999999,"5min":5.532,"15min":35.890999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66800000000000004,"15min":0.69799999999999995},"min":{"1min":0.48199999999999998,"5min":0.42899999999999999,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":5.2830000000000004,"15min":22.846},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84099999999999997},"min":{"1min":0.67000000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0549999999999999,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.496,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":5.6059999999999999,"15min":7.8220000000000001},"last":0.94599999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.93999999999999995,"5min":5.4260000000000002,"15min":9.9109999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.86499999999999999},"min":{"1min":0.70799999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":5.4870000000000001,"15min":5.4870000000000001},"last":1.3360000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:58:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":1.2030000000000001,"15min":0.95999999999999996},"min":{"1min":0.64100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":1.379},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0960000000000001,"15min":0.97599999999999998},"min":{"1min":0.68300000000000005,"5min":0.60099999999999998,"15min":0.56599999999999995},"max":{"1min":15.975,"5min":15.975,"15min":15.975},"last":1.6220000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.75700000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.169,"5min":5.3899999999999997,"15min":5.3899999999999997},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.91900000000000004,"15min":0.91600000000000004},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53700000000000003},"max":{"1min":1.0800000000000001,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":1.097}]}]},{"osd":1,"up_from":15,"seq":64424509734,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353060,"kb_used_data":245596,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94018780,"statfs":{"total":96636764160,"available":96275230720,"internally_reserved":0,"allocated":251490304,"data_stored":246176700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.437},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.245},"max":{"1min":0.67200000000000004,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":0.315},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55100000000000005,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":0.66700000000000004,"5min":2.5920000000000001,"15min":22.149000000000001},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.315},"max":{"1min":0.749,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.75900000000000001},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":2.8130000000000002,"15min":54.530000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.67800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34100000000000003},"max":{"1min":0.83799999999999997,"5min":2.879,"15min":25.457000000000001},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78300000000000003,"15min":0.81299999999999994},"min":{"1min":0.53900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94699999999999995,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.79500000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":1.054,"5min":2.101,"15min":2.101},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83699999999999997,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.505},"max":{"1min":1.012,"5min":2.7810000000000001,"15min":11.164999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83599999999999997},"min":{"1min":0.624,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":0.94399999999999995,"5min":2.8460000000000001,"15min":2.8820000000000001},"last":0.65800000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.83699999999999997},"min":{"1min":0.57299999999999995,"5min":0.55600000000000005,"15min":0.497},"max":{"1min":0.95399999999999996,"5min":2.903,"15min":5.9180000000000001},"last":0.754},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.85299999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":1.9930000000000001,"15min":2.2629999999999999},"last":0.64100000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:58:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.002,"15min":0.96899999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":12.221,"5min":12.221,"15min":12.221},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.373,"5min":1.0209999999999999,"15min":0.94499999999999995},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":12.153,"5min":12.153,"15min":12.153},"last":0.85199999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542440,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340048,"kb_used_data":246472,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94031792,"statfs":{"total":96636764160,"available":96288555008,"internally_reserved":0,"allocated":252387328,"data_stored":247227008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":95812039},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.1240000000000001,"15min":1.042},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.41999999999999998},"max":{"1min":1.0389999999999999,"5min":19.741,"15min":19.741},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.131,"15min":0.99099999999999999},"min":{"1min":0.69599999999999995,"5min":0.55800000000000005,"15min":0.34899999999999998},"max":{"1min":1.3600000000000001,"5min":13.863,"15min":13.863},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93700000000000006,"15min":0.86799999999999999},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.46100000000000002},"max":{"1min":1.0669999999999999,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.1819999999999999,"15min":1.0289999999999999},"min":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.46800000000000003},"max":{"1min":1.2929999999999999,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":0.68500000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96199999999999997,"15min":0.97699999999999998},"min":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.335,"5min":3.5329999999999999,"15min":3.7709999999999999},"last":0.873},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.095,"15min":1.083},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.4059999999999999,"5min":3.407,"15min":4.6840000000000002},"last":1.2809999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.97599999999999998,"15min":1.014},"min":{"1min":0.67100000000000004,"5min":0.52800000000000002,"15min":0.46700000000000003},"max":{"1min":1.359,"5min":3.7999999999999998,"15min":4.7629999999999999},"last":1.448},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.171,"15min":1.0840000000000001},"min":{"1min":0.92900000000000005,"5min":0.59299999999999997,"15min":0.54600000000000004},"max":{"1min":1.4179999999999999,"5min":3.968,"15min":3.968},"last":1.5429999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2290000000000001,"15min":1.208},"min":{"1min":0.79600000000000004,"5min":0.55800000000000005,"15min":0.51000000000000001},"max":{"1min":1.4710000000000001,"5min":3.8900000000000001,"15min":16.204000000000001},"last":1.3979999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1519999999999999,"15min":1.2649999999999999},"min":{"1min":0.63700000000000001,"5min":0.622,"15min":0.42699999999999999},"max":{"1min":1.399,"5min":19.398,"15min":43.198},"last":1.617}]},{"osd":5,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.98499999999999999,"15min":1.0489999999999999},"min":{"1min":0.70899999999999996,"5min":0.47699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98399999999999999,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3129999999999999,"15min":1.3009999999999999},"min":{"1min":1.1879999999999999,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":1.522,"5min":3.7269999999999999,"15min":3.802},"last":1.6659999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.833,"15min":1.5880000000000001},"min":{"1min":0.83499999999999996,"5min":0.58899999999999997,"15min":0.55400000000000005},"max":{"1min":1.2709999999999999,"5min":70.335999999999999,"15min":73.228999999999999},"last":1.24},{"interface":"front","average":{"1min":1.079,"5min":1.675,"15min":1.637},"min":{"1min":0.82699999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.381,"5min":42.164999999999999,"15min":73.289000000000001},"last":1.5009999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542438,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429260,"kb_used_data":335620,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93942580,"statfs":{"total":96636764160,"available":96197201920,"internally_reserved":0,"allocated":343674880,"data_stored":335552063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,36,0,0,22],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.89900000000000002,"15min":0.879},"min":{"1min":0.54600000000000004,"5min":0.46999999999999997,"15min":0.40400000000000003},"max":{"1min":3.9159999999999999,"5min":3.9159999999999999,"15min":5.0599999999999996},"last":1.7350000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.83099999999999996,"15min":0.872},"min":{"1min":0.52900000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0569999999999999,"5min":1.5620000000000001,"15min":3.4849999999999999},"last":5.2240000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.92600000000000005,"15min":0.871},"min":{"1min":0.55300000000000005,"5min":0.40600000000000003,"15min":0.33600000000000002},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":4.6520000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.85099999999999998,"15min":0.90000000000000002},"min":{"1min":0.50700000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":4.5300000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95899999999999996,"15min":0.95599999999999996},"min":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.51100000000000001},"max":{"1min":1.462,"5min":1.897,"15min":5.3710000000000004},"last":5.109},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1279999999999999,"15min":1.056},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.53200000000000003},"max":{"1min":1.369,"5min":2.081,"15min":2.887},"last":5.5709999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.095,"15min":1.0620000000000001},"min":{"1min":0.82999999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":5.476},"last":5.2709999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.97599999999999998,"15min":1.0349999999999999},"min":{"1min":0.67200000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1779999999999999,"5min":1.52,"15min":3.8500000000000001},"last":4.6950000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95599999999999996,"15min":0.98699999999999999},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.38400000000000001},"max":{"1min":1.3360000000000001,"5min":1.847,"15min":4.1710000000000003},"last":5.0229999999999997},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.66900000000000004,"5min":0.49199999999999999,"15min":0.41799999999999998},"max":{"1min":1.3009999999999999,"5min":1.7270000000000001,"15min":4.2519999999999998},"last":4.843}]},{"osd":6,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1899999999999999,"15min":1.1739999999999999},"min":{"1min":0.747,"5min":0.65000000000000002,"15min":0.621},"max":{"1min":1.546,"5min":2.177,"15min":5.6029999999999998},"last":5.3090000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1459999999999999,"15min":1.1739999999999999},"min":{"1min":0.69499999999999995,"5min":0.56100000000000005,"15min":0.41899999999999998},"max":{"1min":1.403,"5min":1.839,"15min":4.077},"last":5.1920000000000002}]},{"osd":7,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.147,"15min":1.23},"min":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.45100000000000001},"max":{"1min":1.4650000000000001,"5min":1.7949999999999999,"15min":9.9390000000000001},"last":10.884},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.2430000000000001,"15min":1.2889999999999999},"min":{"1min":0.97399999999999998,"5min":0.78000000000000003,"15min":0.48499999999999999},"max":{"1min":1.415,"5min":1.907,"15min":4.3399999999999999},"last":5.335}]}]},{"osd":4,"up_from":14,"seq":60129542438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418192,"kb_used_data":337032,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93953648,"statfs":{"total":96636764160,"available":96208535552,"internally_reserved":0,"allocated":345120768,"data_stored":336789947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.91100000000000003,"15min":0.873},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.167,"5min":8.7680000000000007,"15min":8.7680000000000007},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.89000000000000001,"15min":0.86099999999999999},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.125,"5min":6.5990000000000002,"15min":6.5990000000000002},"last":0.86699999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82899999999999996,"15min":0.89700000000000002},"min":{"1min":0.51600000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.236,"5min":1.375,"15min":5.8010000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.85899999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.262,"5min":1.6970000000000001,"15min":3.0499999999999998},"last":1.022}]},{"osd":2,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88500000000000001,"15min":0.91600000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.212,"5min":1.331,"15min":3.323},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.98599999999999999,"15min":0.97099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.252,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":1.0840000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.1100000000000001,"15min":1.0209999999999999},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.151,"5min":8.0570000000000004,"15min":8.0570000000000004},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.056,"15min":1.0009999999999999},"min":{"1min":0.65900000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.381,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.94399999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.018,"15min":1.0600000000000001},"min":{"1min":0.82699999999999996,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.325,"5min":4.0030000000000001,"15min":8.4550000000000001},"last":1.054},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.175,"15min":1.171},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.57699999999999996},"max":{"1min":1.3959999999999999,"5min":4.0609999999999999,"15min":8.5399999999999991},"last":1.1100000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:58:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1859999999999999,"15min":1.083},"min":{"1min":0.76300000000000001,"5min":0.503,"15min":0.45100000000000001},"max":{"1min":1.4870000000000001,"5min":16.989999999999998,"15min":16.989999999999998},"last":1.206},{"interface":"front","average":{"1min":1.196,"5min":1.173,"15min":1.149},"min":{"1min":0.86699999999999999,"5min":0.41299999999999998,"15min":0.36699999999999999},"max":{"1min":1.599,"5min":4.4530000000000003,"15min":4.4530000000000003},"last":1.2749999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:42 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1719999999999999,"15min":1.163},"min":{"1min":0.84199999999999997,"5min":0.45100000000000001,"15min":0.42199999999999999},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":3.4510000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.298,"15min":1.3049999999999999},"min":{"1min":1.0269999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.5470000000000002},"last":1.377}]}]},{"osd":0,"up_from":14,"seq":60129542440,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461680,"kb_used_data":368040,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93910160,"statfs":{"total":96636764160,"available":96164003840,"internally_reserved":0,"allocated":376872960,"data_stored":368821573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.98899999999999999,"15min":0.63900000000000001},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":0.63900000000000001,"5min":45.311,"15min":45.311},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.59599999999999997},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.27200000000000002},"max":{"1min":0.72599999999999998,"5min":2.2410000000000001,"15min":27.616},"last":0.436}]},{"osd":2,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.29099999999999998},"max":{"1min":0.82999999999999996,"5min":0.88100000000000001,"15min":2.5449999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.44,"5min":0.40899999999999997,"15min":0.35199999999999998},"max":{"1min":0.752,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.67500000000000004,"15min":0.56100000000000005},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":0.79300000000000004,"5min":12.368,"15min":12.368},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.75800000000000001,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.40899999999999997},"max":{"1min":0.83299999999999996,"5min":12.442,"15min":12.442},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84799999999999998,"15min":0.85999999999999999},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.41599999999999998},"max":{"1min":1,"5min":2.6949999999999998,"15min":9.4979999999999993},"last":0.753},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.871,"15min":0.86499999999999999},"min":{"1min":0.629,"5min":0.60199999999999998,"15min":0.46700000000000003},"max":{"1min":1.1220000000000001,"5min":2.6440000000000001,"15min":9.5429999999999993},"last":0.84699999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.88,"15min":0.878},"min":{"1min":0.72199999999999998,"5min":0.47099999999999997,"15min":0.317},"max":{"1min":1.087,"5min":1.163,"15min":11.026},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.375,"15min":1.0309999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.46999999999999997},"max":{"1min":1.1759999999999999,"5min":44.225999999999999,"15min":44.225999999999999},"last":0.78500000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88,"15min":0.873},"min":{"1min":0.73899999999999999,"5min":0.63,"15min":0.48799999999999999},"max":{"1min":1.0649999999999999,"5min":2.5169999999999999,"15min":2.9300000000000002},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.435},"max":{"1min":1.133,"5min":1.236,"15min":11.087999999999999},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.91400000000000003,"15min":0.879},"min":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.50900000000000001},"max":{"1min":1.0740000000000001,"5min":5.6849999999999996,"15min":5.6849999999999996},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.93200000000000005,"15min":0.93300000000000005},"min":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.57199999999999995},"max":{"1min":1.0840000000000001,"5min":2.6030000000000002,"15min":9.5890000000000004},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50573312,"data_stored":50555931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46346240,"data_stored":46320604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74674176,"data_stored":74645967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":71393280,"data_stored":71379650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67129344,"data_stored":67110274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":74559488,"data_stored":74539847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":67350528,"data_stored":67333247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325066752,"data_stored":317352305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183717888,"data_stored":178743923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144601088,"data_stored":139970730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":249585664,"data_stored":241694096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":273088512,"data_stored":265087328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":275906560,"data_stored":268118820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":177188864,"data_stored":172364192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":273362944,"data_stored":265498878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:59:11.510 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.510 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.510 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.511 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.511 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.512 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.512 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.512 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.513 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.513 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.513 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.514 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.514 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.515 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.515 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.515 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.516 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.516 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.516 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.517 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.517 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.518 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.518 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.518 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.519 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.519 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.519 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.520 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.520 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.521 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.521 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.521 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.522 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.522 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.522 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.523 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.523 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.524 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.524 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.524 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.525 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.525 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.525 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.526 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.526 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.527 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.527 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.527 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.528 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.528 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.528 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.529 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.529 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.529 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.530 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.530 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.531 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.531 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.531 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.532 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.532 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.532 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.533 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.533 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T13:34:53.554510+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.533 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.534 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.534 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.535 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.535 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.535 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.536 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.536 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.536 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.537 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.537 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.538 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.538 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.538 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.539 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.539 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.539 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.540 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.540 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.541 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.541 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.541 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.542 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.542 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.542 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.543 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.543 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.544 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.544 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.544 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.545 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:11.545 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:59:29.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:29.500 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:31.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:59:31.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.691+0000 7f2514a91640 1 Processor -- start 2023-10-30T13:59:31.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.691+0000 7f2514a91640 1 -- start start 2023-10-30T13:59:31.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.691+0000 7f2514a91640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f251015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.691+0000 7f2514a91640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f25100c4a10 0x7f25100c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f2514a91640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2510159f20 con 0x7f251015a3b0 2023-10-30T13:59:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f2514a91640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251015dec0 con 0x7f25100c4a10 2023-10-30T13:59:31.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f2514a91640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25101599a0 con 0x7f2510153c80 2023-10-30T13:59:31.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f25100c4a10 0x7f25100c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:31.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f25100c4a10 0x7f25100c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51306/0 (socket says 172.21.15.31:51306) 2023-10-30T13:59:31.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 -- 172.21.15.31:0/2527595600 learned_addr learned my addr 172.21.15.31:0/2527595600 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:31.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250e59c640 1 --2- 172.21.15.31:0/2527595600 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f251015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:31.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 -- 172.21.15.31:0/2527595600 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 msgr2=0x7f251015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:31.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 --2- 172.21.15.31:0/2527595600 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f251015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 -- 172.21.15.31:0/2527595600 >> v1:172.21.15.33:6789/0 conn(0x7f251015a3b0 legacy=0x7f25100ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:31.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 -- 172.21.15.31:0/2527595600 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100c47e0 con 0x7f25100c4a10 2023-10-30T13:59:31.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250dd9b640 1 --2- 172.21.15.31:0/2527595600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100c42a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2504009870 tx=0x7f250401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b79c0d19edf4bab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.695+0000 7f250d59a640 1 -- 172.21.15.31:0/2527595600 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25040a8a30 con 0x7f25100c4a10 2023-10-30T13:59:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f250d59a640 1 -- 172.21.15.31:0/2527595600 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25040a8b90 con 0x7f25100c4a10 2023-10-30T13:59:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f250d59a640 1 -- 172.21.15.31:0/2527595600 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25040b0440 con 0x7f25100c4a10 2023-10-30T13:59:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- 172.21.15.31:0/2527595600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 msgr2=0x7f25100c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 --2- 172.21.15.31:0/2527595600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100c42a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2504009870 tx=0x7f250401ed30 comp rx=0 tx=0).stop 2023-10-30T13:59:31.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- 172.21.15.31:0/2527595600 shutdown_connections 2023-10-30T13:59:31.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 --2- 172.21.15.31:0/2527595600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100c42a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 --2- 172.21.15.31:0/2527595600 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f251015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- 172.21.15.31:0/2527595600 >> 172.21.15.31:0/2527595600 conn(0x7f25100715c0 msgr2=0x7f25100719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:31.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- 172.21.15.31:0/2527595600 shutdown_connections 2023-10-30T13:59:31.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- 172.21.15.31:0/2527595600 wait complete. 2023-10-30T13:59:31.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 Processor -- start 2023-10-30T13:59:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- start start 2023-10-30T13:59:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f25100ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f251015a3b0 0x7f25100b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101ef230 con 0x7f25100c4a10 2023-10-30T13:59:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100b28d0 con 0x7f251015a3b0 2023-10-30T13:59:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f2514a91640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25101ef4f0 con 0x7f2510153c80 2023-10-30T13:59:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f250e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f250e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51310/0 (socket says 172.21.15.31:51310) 2023-10-30T13:59:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.699+0000 7f250e59c640 1 -- 172.21.15.31:0/441388817 learned_addr learned my addr 172.21.15.31:0/441388817 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250dd9b640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f25100ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250ed9d640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f251015a3b0 0x7f25100b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250e59c640 1 -- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 msgr2=0x7f25100ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250e59c640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f25100ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250e59c640 1 -- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f251015a3b0 msgr2=0x7f25100b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250e59c640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f251015a3b0 0x7f25100b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250e59c640 1 -- 172.21.15.31:0/441388817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25040094d0 con 0x7f25100c4a10 2023-10-30T13:59:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f250e59c640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100b2020 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f250000e930 tx=0x7f250000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f24feffd640 1 -- 172.21.15.31:0/441388817 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500009690 con 0x7f25100c4a10 2023-10-30T13:59:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f24feffd640 1 -- 172.21.15.31:0/441388817 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250000ca00 con 0x7f25100c4a10 2023-10-30T13:59:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f24feffd640 1 -- 172.21.15.31:0/441388817 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000cd70 con 0x7f25100c4a10 2023-10-30T13:59:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25101ef7d0 con 0x7f25100c4a10 2023-10-30T13:59:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25101efd90 con 0x7f25100c4a10 2023-10-30T13:59:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251004e170 con 0x7f25100c4a10 2023-10-30T13:59:31.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f24feffd640 1 -- 172.21.15.31:0/441388817 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2500013470 con 0x7f25100c4a10 2023-10-30T13:59:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f24feffd640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24f00664d0 0x7f24f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.703+0000 7f24feffd640 1 -- 172.21.15.31:0/441388817 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f2500084420 con 0x7f25100c4a10 2023-10-30T13:59:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.707+0000 7f250dd9b640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24f00664d0 0x7f24f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.707+0000 7f250dd9b640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24f00664d0 0x7f24f0068990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f25040b2000 tx=0x7f2504009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.711+0000 7f24feffd640 1 -- 172.21.15.31:0/441388817 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2500051660 con 0x7f25100c4a10 2023-10-30T13:59:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.879+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f251015d900 con 0x7f24f00664d0 2023-10-30T13:59:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.891+0000 7f24feffd640 1 -- 172.21.15.31:0/441388817 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292607 (secure 0 0 0) 0x7f25040b0540 con 0x7f24f00664d0 2023-10-30T13:59:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:59:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24f00664d0 msgr2=0x7f24f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24f00664d0 0x7f24f0068990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f25040b2000 tx=0x7f2504009040 comp rx=0 tx=0).stop 2023-10-30T13:59:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 msgr2=0x7f25100b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:31.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100b2020 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f250000e930 tx=0x7f250000edf0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 shutdown_connections 2023-10-30T13:59:31.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f24f00664d0 0x7f24f0068990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f251015a3b0 0x7f25100b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f2510153c80 0x7f25100ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 --2- 172.21.15.31:0/441388817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25100c4a10 0x7f25100b2020 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:31.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 >> 172.21.15.31:0/441388817 conn(0x7f25100715c0 msgr2=0x7f2510157970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:31.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 shutdown_connections 2023-10-30T13:59:31.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:31.895+0000 7f2514a91640 1 -- 172.21.15.31:0/441388817 wait complete. 2023-10-30T13:59:31.921 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":788,"stamp":"2023-10-30T13:59:31.724397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185890865,"num_objects":10532,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10532,"num_whiteouts":0,"num_read":8937,"num_read_kb":1843152,"num_write":58243,"num_write_kb":2330064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":756644,"num_omap_keys":1301,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64899,"ondisk_log_size":64899,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3132336,"kb_used_data":2372272,"kb_used_omap":12,"kb_used_meta":759987,"kb_avail":751842384,"statfs":{"total":773094113280,"available":769886601216,"internally_reserved":0,"allocated":2429206528,"data_stored":2372825386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":778227256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":195300,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":195638,"num_omap_keys":325,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004113"},"pg_stats":[{"pgid":"3.3f","version":"56'674","reported_seq":2894,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.530747+0000","last_change":"2023-10-30T13:45:28.601107+0000","last_active":"2023-10-30T13:59:03.530747+0000","last_peered":"2023-10-30T13:59:03.530747+0000","last_clean":"2023-10-30T13:59:03.530747+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:59:03.530747+0000","last_undegraded":"2023-10-30T13:59:03.530747+0000","last_fullsized":"2023-10-30T13:59:03.530747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098613959999999997,"stat_sum":{"num_bytes":30553715,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":124,"num_read_kb":8207,"num_write":638,"num_write_kb":39180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"56'657","reported_seq":2857,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.487037+0000","last_change":"2023-10-30T13:45:27.935552+0000","last_active":"2023-10-30T13:59:03.487037+0000","last_peered":"2023-10-30T13:59:03.487037+0000","last_clean":"2023-10-30T13:59:03.487037+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:59:03.487037+0000","last_undegraded":"2023-10-30T13:59:03.487037+0000","last_fullsized":"2023-10-30T13:59:03.487037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001955118,"stat_sum":{"num_bytes":9994664,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":104,"num_read_kb":11,"num_write":617,"num_write_kb":23604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"56'815","reported_seq":3032,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:22.708425+0000","last_change":"2023-10-30T13:59:22.708475+0000","last_active":"2023-10-30T13:59:22.708425+0000","last_peered":"2023-10-30T13:59:22.708425+0000","last_clean":"2023-10-30T13:59:22.708425+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:59:22.708425+0000","last_undegraded":"2023-10-30T13:59:22.708425+0000","last_fullsized":"2023-10-30T13:59:22.708425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'815","last_scrub_stamp":"2023-10-30T13:59:22.708353+0000","last_deep_scrub":"56'815","last_deep_scrub_stamp":"2023-10-30T13:59:22.708353+0000","last_clean_scrub_stamp":"2023-10-30T13:59:22.708353+0000","objects_scrubbed":162,"log_size":815,"log_dups_size":0,"ondisk_log_size":815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:43:36.533585+0000","scrub_duration":0.169844142,"objects_trimmed":0,"snaptrim_duration":0.0021120940000000001,"stat_sum":{"num_bytes":23390266,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":110,"num_read_kb":20497,"num_write":638,"num_write_kb":47001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"56'828","reported_seq":3069,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.387579+0000","last_change":"2023-10-30T13:59:24.387627+0000","last_active":"2023-10-30T13:59:24.387579+0000","last_peered":"2023-10-30T13:59:24.387579+0000","last_clean":"2023-10-30T13:59:24.387579+0000","last_became_active":"2023-10-30T13:35:48.779909+0000","last_became_peered":"2023-10-30T13:35:48.779909+0000","last_unstale":"2023-10-30T13:59:24.387579+0000","last_undegraded":"2023-10-30T13:59:24.387579+0000","last_fullsized":"2023-10-30T13:59:24.387579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'828","last_scrub_stamp":"2023-10-30T13:59:24.387502+0000","last_deep_scrub":"56'828","last_deep_scrub_stamp":"2023-10-30T13:59:24.387502+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.387502+0000","objects_scrubbed":159,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:33.212806+0000","scrub_duration":0.154922691,"objects_trimmed":0,"snaptrim_duration":0.001317775,"stat_sum":{"num_bytes":10165225,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":133,"num_read_kb":8205,"num_write":656,"num_write_kb":40955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"56'701","reported_seq":2905,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.510951+0000","last_change":"2023-10-30T13:45:28.597729+0000","last_active":"2023-10-30T13:59:03.510951+0000","last_peered":"2023-10-30T13:59:03.510951+0000","last_clean":"2023-10-30T13:59:03.510951+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:59:03.510951+0000","last_undegraded":"2023-10-30T13:59:03.510951+0000","last_fullsized":"2023-10-30T13:59:03.510951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007614844,"stat_sum":{"num_bytes":19907850,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":110,"num_read_kb":8208,"num_write":664,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"56'666","reported_seq":2864,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.525952+0000","last_change":"2023-10-30T13:45:27.941070+0000","last_active":"2023-10-30T13:59:03.525952+0000","last_peered":"2023-10-30T13:59:03.525952+0000","last_clean":"2023-10-30T13:59:03.525952+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:59:03.525952+0000","last_undegraded":"2023-10-30T13:59:03.525952+0000","last_fullsized":"2023-10-30T13:59:03.525952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030849649999999998,"stat_sum":{"num_bytes":19227960,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":104,"num_read_kb":7536,"num_write":633,"num_write_kb":20026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"56'695","reported_seq":2880,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.510666+0000","last_change":"2023-10-30T13:45:28.597665+0000","last_active":"2023-10-30T13:59:03.510666+0000","last_peered":"2023-10-30T13:59:03.510666+0000","last_clean":"2023-10-30T13:59:03.510666+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:59:03.510666+0000","last_undegraded":"2023-10-30T13:59:03.510666+0000","last_fullsized":"2023-10-30T13:59:03.510666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082986029999999999,"stat_sum":{"num_bytes":10009773,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":91,"num_read_kb":11,"num_write":659,"num_write_kb":29217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"56'670","reported_seq":2896,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.521685+0000","last_change":"2023-10-30T13:45:28.780454+0000","last_active":"2023-10-30T13:59:03.521685+0000","last_peered":"2023-10-30T13:59:03.521685+0000","last_clean":"2023-10-30T13:59:03.521685+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:59:03.521685+0000","last_undegraded":"2023-10-30T13:59:03.521685+0000","last_fullsized":"2023-10-30T13:59:03.521685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021728070000000001,"stat_sum":{"num_bytes":10221735,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":132,"num_read_kb":1147,"num_write":641,"num_write_kb":24778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"56'666","reported_seq":2884,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.089833+0000","last_change":"2023-10-30T13:45:28.601219+0000","last_active":"2023-10-30T13:59:11.089833+0000","last_peered":"2023-10-30T13:59:11.089833+0000","last_clean":"2023-10-30T13:59:11.089833+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:59:11.089833+0000","last_undegraded":"2023-10-30T13:59:11.089833+0000","last_fullsized":"2023-10-30T13:59:11.089833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010045174,"stat_sum":{"num_bytes":10718551,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":124,"num_read_kb":4115,"num_write":615,"num_write_kb":24187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"56'617","reported_seq":2825,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.547852+0000","last_change":"2023-10-30T13:45:28.050787+0000","last_active":"2023-10-30T13:59:10.547852+0000","last_peered":"2023-10-30T13:59:10.547852+0000","last_clean":"2023-10-30T13:59:10.547852+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:59:10.547852+0000","last_undegraded":"2023-10-30T13:59:10.547852+0000","last_fullsized":"2023-10-30T13:59:10.547852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047277760000000004,"stat_sum":{"num_bytes":20230336,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":113,"num_read_kb":28,"num_write":585,"num_write_kb":25967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"56'612","reported_seq":2804,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.547849+0000","last_change":"2023-10-30T13:45:28.050006+0000","last_active":"2023-10-30T13:59:10.547849+0000","last_peered":"2023-10-30T13:59:10.547849+0000","last_clean":"2023-10-30T13:59:10.547849+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:59:10.547849+0000","last_undegraded":"2023-10-30T13:59:10.547849+0000","last_fullsized":"2023-10-30T13:59:10.547849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00429868,"stat_sum":{"num_bytes":2384023,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":97,"num_read_kb":17,"num_write":586,"num_write_kb":9055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"56'789","reported_seq":3015,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.827760+0000","last_change":"2023-10-30T13:59:19.827838+0000","last_active":"2023-10-30T13:59:19.827760+0000","last_peered":"2023-10-30T13:59:19.827760+0000","last_clean":"2023-10-30T13:59:19.827760+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:59:19.827760+0000","last_undegraded":"2023-10-30T13:59:19.827760+0000","last_fullsized":"2023-10-30T13:59:19.827760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'789","last_scrub_stamp":"2023-10-30T13:59:19.827650+0000","last_deep_scrub":"56'789","last_deep_scrub_stamp":"2023-10-30T13:59:19.827650+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.827650+0000","objects_scrubbed":175,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:07.295597+0000","scrub_duration":0.27225982599999998,"objects_trimmed":0,"snaptrim_duration":0.0066533010000000004,"stat_sum":{"num_bytes":18341132,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":8208,"num_write":596,"num_write_kb":32418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"56'648","reported_seq":2856,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.913193+0000","last_change":"2023-10-30T13:45:28.780475+0000","last_active":"2023-10-30T13:59:08.913193+0000","last_peered":"2023-10-30T13:59:08.913193+0000","last_clean":"2023-10-30T13:59:08.913193+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:59:08.913193+0000","last_undegraded":"2023-10-30T13:59:08.913193+0000","last_fullsized":"2023-10-30T13:59:08.913193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002338237,"stat_sum":{"num_bytes":14177749,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":114,"num_read_kb":12311,"num_write":601,"num_write_kb":37081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"56'871","reported_seq":3099,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.750623+0000","last_change":"2023-10-30T13:59:24.750679+0000","last_active":"2023-10-30T13:59:24.750623+0000","last_peered":"2023-10-30T13:59:24.750623+0000","last_clean":"2023-10-30T13:59:24.750623+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:59:24.750623+0000","last_undegraded":"2023-10-30T13:59:24.750623+0000","last_fullsized":"2023-10-30T13:59:24.750623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'871","last_scrub_stamp":"2023-10-30T13:59:24.750548+0000","last_deep_scrub":"56'871","last_deep_scrub_stamp":"2023-10-30T13:59:24.750548+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.750548+0000","objects_scrubbed":167,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:44.353295+0000","scrub_duration":0.19578169600000001,"objects_trimmed":0,"snaptrim_duration":0.0029248669999999998,"stat_sum":{"num_bytes":22804734,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":119,"num_read_kb":24041,"num_write":691,"num_write_kb":57342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"56'716","reported_seq":2923,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.497802+0000","last_change":"2023-10-30T13:45:27.934641+0000","last_active":"2023-10-30T13:59:03.497802+0000","last_peered":"2023-10-30T13:59:03.497802+0000","last_clean":"2023-10-30T13:59:03.497802+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:59:03.497802+0000","last_undegraded":"2023-10-30T13:59:03.497802+0000","last_fullsized":"2023-10-30T13:59:03.497802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021050410000000002,"stat_sum":{"num_bytes":18271578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":112,"num_read_kb":7334,"num_write":676,"num_write_kb":39770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"56'732","reported_seq":2950,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.524842+0000","last_change":"2023-10-30T13:45:27.943054+0000","last_active":"2023-10-30T13:59:03.524842+0000","last_peered":"2023-10-30T13:59:03.524842+0000","last_clean":"2023-10-30T13:59:03.524842+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:59:03.524842+0000","last_undegraded":"2023-10-30T13:59:03.524842+0000","last_fullsized":"2023-10-30T13:59:03.524842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052579080000000004,"stat_sum":{"num_bytes":35259912,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":123,"num_read_kb":12306,"num_write":693,"num_write_kb":43349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"56'710","reported_seq":2907,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.529864+0000","last_change":"2023-10-30T13:45:27.950182+0000","last_active":"2023-10-30T13:59:03.529864+0000","last_peered":"2023-10-30T13:59:03.529864+0000","last_clean":"2023-10-30T13:59:03.529864+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:59:03.529864+0000","last_undegraded":"2023-10-30T13:59:03.529864+0000","last_fullsized":"2023-10-30T13:59:03.529864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052923609999999998,"stat_sum":{"num_bytes":14461920,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":103,"num_read_kb":4113,"num_write":659,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"56'763","reported_seq":2989,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.418430+0000","last_change":"2023-10-30T13:59:17.418479+0000","last_active":"2023-10-30T13:59:17.418430+0000","last_peered":"2023-10-30T13:59:17.418430+0000","last_clean":"2023-10-30T13:59:17.418430+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:59:17.418430+0000","last_undegraded":"2023-10-30T13:59:17.418430+0000","last_fullsized":"2023-10-30T13:59:17.418430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'763","last_scrub_stamp":"2023-10-30T13:59:17.418359+0000","last_deep_scrub":"56'763","last_deep_scrub_stamp":"2023-10-30T13:59:17.418359+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.418359+0000","objects_scrubbed":142,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:00.183579+0000","scrub_duration":0.20971843000000001,"objects_trimmed":0,"snaptrim_duration":0.001369274,"stat_sum":{"num_bytes":15414431,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":118,"num_read_kb":4123,"num_write":623,"num_write_kb":31369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"56'728","reported_seq":2956,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.507882+0000","last_change":"2023-10-30T13:45:28.600018+0000","last_active":"2023-10-30T13:59:03.507882+0000","last_peered":"2023-10-30T13:59:03.507882+0000","last_clean":"2023-10-30T13:59:03.507882+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:59:03.507882+0000","last_undegraded":"2023-10-30T13:59:03.507882+0000","last_fullsized":"2023-10-30T13:59:03.507882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092525119999999992,"stat_sum":{"num_bytes":8879597,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":134,"num_read_kb":8209,"num_write":693,"num_write_kb":31072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"56'667","reported_seq":2861,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.519225+0000","last_change":"2023-10-30T13:45:28.051528+0000","last_active":"2023-10-30T13:59:03.519225+0000","last_peered":"2023-10-30T13:59:03.519225+0000","last_clean":"2023-10-30T13:59:03.519225+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:59:03.519225+0000","last_undegraded":"2023-10-30T13:59:03.519225+0000","last_fullsized":"2023-10-30T13:59:03.519225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00502631,"stat_sum":{"num_bytes":6211385,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":100,"num_read_kb":8209,"num_write":620,"num_write_kb":19966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"56'692","reported_seq":2898,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.508157+0000","last_change":"2023-10-30T13:45:27.940012+0000","last_active":"2023-10-30T13:59:03.508157+0000","last_peered":"2023-10-30T13:59:03.508157+0000","last_clean":"2023-10-30T13:59:03.508157+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:59:03.508157+0000","last_undegraded":"2023-10-30T13:59:03.508157+0000","last_fullsized":"2023-10-30T13:59:03.508157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026024469999999999,"stat_sum":{"num_bytes":18097476,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":111,"num_read_kb":21,"num_write":651,"num_write_kb":32402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"56'705","reported_seq":2905,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.526011+0000","last_change":"2023-10-30T13:45:27.947213+0000","last_active":"2023-10-30T13:59:03.526011+0000","last_peered":"2023-10-30T13:59:03.526011+0000","last_clean":"2023-10-30T13:59:03.526011+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:59:03.526011+0000","last_undegraded":"2023-10-30T13:59:03.526011+0000","last_fullsized":"2023-10-30T13:59:03.526011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054353270000000002,"stat_sum":{"num_bytes":10272429,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":105,"num_read_kb":4117,"num_write":665,"num_write_kb":16082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"56'701","reported_seq":2909,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.510809+0000","last_change":"2023-10-30T13:45:28.781012+0000","last_active":"2023-10-30T13:59:03.510809+0000","last_peered":"2023-10-30T13:59:03.510809+0000","last_clean":"2023-10-30T13:59:03.510809+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:59:03.510809+0000","last_undegraded":"2023-10-30T13:59:03.510809+0000","last_fullsized":"2023-10-30T13:59:03.510809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025617359999999998,"stat_sum":{"num_bytes":14333972,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":114,"num_read_kb":8213,"num_write":665,"num_write_kb":36470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"56'850","reported_seq":3086,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.738390+0000","last_change":"2023-10-30T13:59:24.738493+0000","last_active":"2023-10-30T13:59:24.738390+0000","last_peered":"2023-10-30T13:59:24.738390+0000","last_clean":"2023-10-30T13:59:24.738390+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:59:24.738390+0000","last_undegraded":"2023-10-30T13:59:24.738390+0000","last_fullsized":"2023-10-30T13:59:24.738390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'850","last_scrub_stamp":"2023-10-30T13:59:24.738213+0000","last_deep_scrub":"56'850","last_deep_scrub_stamp":"2023-10-30T13:59:24.738213+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.738213+0000","objects_scrubbed":157,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:41:27.710826+0000","scrub_duration":0.30834826700000001,"objects_trimmed":0,"snaptrim_duration":0.004034279,"stat_sum":{"num_bytes":1595822,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":126,"num_read_kb":16,"num_write":676,"num_write_kb":12551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"56'674","reported_seq":2887,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.511918+0000","last_change":"2023-10-30T13:45:28.782023+0000","last_active":"2023-10-30T13:59:03.511918+0000","last_peered":"2023-10-30T13:59:03.511918+0000","last_clean":"2023-10-30T13:59:03.511918+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:59:03.511918+0000","last_undegraded":"2023-10-30T13:59:03.511918+0000","last_fullsized":"2023-10-30T13:59:03.511918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030593479999999999,"stat_sum":{"num_bytes":7858557,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":117,"num_read_kb":4108,"num_write":634,"num_write_kb":18163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"56'722","reported_seq":2929,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.529915+0000","last_change":"2023-10-30T13:45:28.051170+0000","last_active":"2023-10-30T13:59:03.529915+0000","last_peered":"2023-10-30T13:59:03.529915+0000","last_clean":"2023-10-30T13:59:03.529915+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:59:03.529915+0000","last_undegraded":"2023-10-30T13:59:03.529915+0000","last_fullsized":"2023-10-30T13:59:03.529915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":722,"log_dups_size":0,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049754609999999996,"stat_sum":{"num_bytes":9799460,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":111,"num_read_kb":24,"num_write":679,"num_write_kb":16350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"56'668","reported_seq":2874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.494238+0000","last_change":"2023-10-30T13:45:27.935505+0000","last_active":"2023-10-30T13:59:03.494238+0000","last_peered":"2023-10-30T13:59:03.494238+0000","last_clean":"2023-10-30T13:59:03.494238+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:59:03.494238+0000","last_undegraded":"2023-10-30T13:59:03.494238+0000","last_fullsized":"2023-10-30T13:59:03.494238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028468439999999999,"stat_sum":{"num_bytes":33375622,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":110,"num_read_kb":16400,"num_write":628,"num_write_kb":35724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"56'791","reported_seq":3012,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.370003+0000","last_change":"2023-10-30T13:59:25.370057+0000","last_active":"2023-10-30T13:59:25.370003+0000","last_peered":"2023-10-30T13:59:25.370003+0000","last_clean":"2023-10-30T13:59:25.370003+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:59:25.370003+0000","last_undegraded":"2023-10-30T13:59:25.370003+0000","last_fullsized":"2023-10-30T13:59:25.370003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'791","last_scrub_stamp":"2023-10-30T13:59:25.369932+0000","last_deep_scrub":"56'791","last_deep_scrub_stamp":"2023-10-30T13:59:25.369932+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.369932+0000","objects_scrubbed":174,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:57.726447+0000","scrub_duration":0.25855898900000002,"objects_trimmed":0,"snaptrim_duration":0.002839121,"stat_sum":{"num_bytes":9811427,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":112,"num_read_kb":10701,"num_write":625,"num_write_kb":15312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"56'687","reported_seq":2892,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.505977+0000","last_change":"2023-10-30T13:45:27.941228+0000","last_active":"2023-10-30T13:59:03.505977+0000","last_peered":"2023-10-30T13:59:03.505977+0000","last_clean":"2023-10-30T13:59:03.505977+0000","last_became_active":"2023-10-30T13:35:48.847825+0000","last_became_peered":"2023-10-30T13:35:48.847825+0000","last_unstale":"2023-10-30T13:59:03.505977+0000","last_undegraded":"2023-10-30T13:59:03.505977+0000","last_fullsized":"2023-10-30T13:59:03.505977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028647770000000002,"stat_sum":{"num_bytes":7231756,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":102,"num_read_kb":4848,"num_write":653,"num_write_kb":26441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"56'692","reported_seq":2908,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.525543+0000","last_change":"2023-10-30T13:45:27.948917+0000","last_active":"2023-10-30T13:59:03.525543+0000","last_peered":"2023-10-30T13:59:03.525543+0000","last_clean":"2023-10-30T13:59:03.525543+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:59:03.525543+0000","last_undegraded":"2023-10-30T13:59:03.525543+0000","last_fullsized":"2023-10-30T13:59:03.525543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060200970000000003,"stat_sum":{"num_bytes":22333767,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":120,"num_read_kb":4116,"num_write":656,"num_write_kb":43887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"56'925","reported_seq":3175,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.513765+0000","last_change":"2023-10-30T13:59:16.513815+0000","last_active":"2023-10-30T13:59:16.513765+0000","last_peered":"2023-10-30T13:59:16.513765+0000","last_clean":"2023-10-30T13:59:16.513765+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:59:16.513765+0000","last_undegraded":"2023-10-30T13:59:16.513765+0000","last_fullsized":"2023-10-30T13:59:16.513765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'925","last_scrub_stamp":"2023-10-30T13:59:16.513655+0000","last_deep_scrub":"56'925","last_deep_scrub_stamp":"2023-10-30T13:59:16.513655+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.513655+0000","objects_scrubbed":186,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:03.990730+0000","scrub_duration":0.27611799599999998,"objects_trimmed":0,"snaptrim_duration":0.001263171,"stat_sum":{"num_bytes":10546872,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":139,"num_read_kb":4118,"num_write":747,"num_write_kb":34715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"56'712","reported_seq":2918,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.488456+0000","last_change":"2023-10-30T13:45:28.048779+0000","last_active":"2023-10-30T13:59:03.488456+0000","last_peered":"2023-10-30T13:59:03.488456+0000","last_clean":"2023-10-30T13:59:03.488456+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:59:03.488456+0000","last_undegraded":"2023-10-30T13:59:03.488456+0000","last_fullsized":"2023-10-30T13:59:03.488456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036881090000000002,"stat_sum":{"num_bytes":14815628,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":110,"num_read_kb":20,"num_write":677,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"56'797","reported_seq":3013,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.874226+0000","last_change":"2023-10-30T13:59:15.874311+0000","last_active":"2023-10-30T13:59:15.874226+0000","last_peered":"2023-10-30T13:59:15.874226+0000","last_clean":"2023-10-30T13:59:15.874226+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:59:15.874226+0000","last_undegraded":"2023-10-30T13:59:15.874226+0000","last_fullsized":"2023-10-30T13:59:15.874226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'797","last_scrub_stamp":"2023-10-30T13:59:15.874087+0000","last_deep_scrub":"56'797","last_deep_scrub_stamp":"2023-10-30T13:59:15.874087+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.874087+0000","objects_scrubbed":163,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:30.985181+0000","scrub_duration":0.29104613299999998,"objects_trimmed":0,"snaptrim_duration":0.0065387930000000002,"stat_sum":{"num_bytes":14536361,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":108,"num_read_kb":8117,"num_write":641,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"56'698","reported_seq":776,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.085592+0000","last_change":"2023-10-30T13:59:12.085635+0000","last_active":"2023-10-30T13:59:12.085592+0000","last_peered":"2023-10-30T13:59:12.085592+0000","last_clean":"2023-10-30T13:59:12.085592+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:59:12.085592+0000","last_undegraded":"2023-10-30T13:59:12.085592+0000","last_fullsized":"2023-10-30T13:59:12.085592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'698","last_scrub_stamp":"2023-10-30T13:59:12.085518+0000","last_deep_scrub":"56'698","last_deep_scrub_stamp":"2023-10-30T13:59:12.085518+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.085518+0000","objects_scrubbed":4,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:55:05.427263+0000","scrub_duration":0.022533350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45057,"num_write":652,"num_write_kb":16749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":500,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"56'816","reported_seq":3051,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.871830+0000","last_change":"2023-10-30T13:59:20.871886+0000","last_active":"2023-10-30T13:59:20.871830+0000","last_peered":"2023-10-30T13:59:20.871830+0000","last_clean":"2023-10-30T13:59:20.871830+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:59:20.871830+0000","last_undegraded":"2023-10-30T13:59:20.871830+0000","last_fullsized":"2023-10-30T13:59:20.871830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'816","last_scrub_stamp":"2023-10-30T13:59:20.871754+0000","last_deep_scrub":"56'816","last_deep_scrub_stamp":"2023-10-30T13:59:20.871754+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.871754+0000","objects_scrubbed":166,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:54.494538+0000","scrub_duration":0.28592286,"objects_trimmed":0,"snaptrim_duration":0.0031306289999999998,"stat_sum":{"num_bytes":22886210,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":127,"num_read_kb":6230,"num_write":644,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"56'754","reported_seq":894,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:26.898884+0000","last_change":"2023-10-30T13:59:26.899016+0000","last_active":"2023-10-30T13:59:26.898884+0000","last_peered":"2023-10-30T13:59:26.898884+0000","last_clean":"2023-10-30T13:59:26.898884+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:59:26.898884+0000","last_undegraded":"2023-10-30T13:59:26.898884+0000","last_fullsized":"2023-10-30T13:59:26.898884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'754","last_scrub_stamp":"2023-10-30T13:59:26.898658+0000","last_deep_scrub":"56'754","last_deep_scrub_stamp":"2023-10-30T13:59:26.898658+0000","last_clean_scrub_stamp":"2023-10-30T13:59:26.898658+0000","objects_scrubbed":11,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:38.802448+0000","scrub_duration":0.095300314999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21047876,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":114393,"num_write":693,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21258,"num_omap_keys":40,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"56'146","reported_seq":264,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.093208+0000","last_change":"2023-10-30T13:59:09.093208+0000","last_active":"2023-10-30T13:59:09.093208+0000","last_peered":"2023-10-30T13:59:09.093208+0000","last_clean":"2023-10-30T13:59:09.093208+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:59:09.093208+0000","last_undegraded":"2023-10-30T13:59:09.093208+0000","last_fullsized":"2023-10-30T13:59:09.093208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'146","last_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_deep_scrub":"56'146","last_deep_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.093046+0000","objects_scrubbed":7,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:36.674466+0000","scrub_duration":0.022130225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297526,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":20619,"num_write":102,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33717,"num_omap_keys":64,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"56'792","reported_seq":3030,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.913063+0000","last_change":"2023-10-30T13:45:28.781311+0000","last_active":"2023-10-30T13:59:08.913063+0000","last_peered":"2023-10-30T13:59:08.913063+0000","last_clean":"2023-10-30T13:59:08.913063+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:59:08.913063+0000","last_undegraded":"2023-10-30T13:59:08.913063+0000","last_fullsized":"2023-10-30T13:59:08.913063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027758349999999999,"stat_sum":{"num_bytes":1695799,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":144,"num_read_kb":4120,"num_write":754,"num_write_kb":40525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"56'600","reported_seq":692,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.512895+0000","last_change":"2023-10-30T13:59:10.512895+0000","last_active":"2023-10-30T13:59:10.512895+0000","last_peered":"2023-10-30T13:59:10.512895+0000","last_clean":"2023-10-30T13:59:10.512895+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:59:10.512895+0000","last_undegraded":"2023-10-30T13:59:10.512895+0000","last_fullsized":"2023-10-30T13:59:10.512895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'600","last_scrub_stamp":"2023-10-30T13:59:10.512557+0000","last_deep_scrub":"56'600","last_deep_scrub_stamp":"2023-10-30T13:59:10.512557+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.512557+0000","objects_scrubbed":4,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:37:46.271383+0000","scrub_duration":0.060732762000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":96432,"num_write":542,"num_write_kb":20769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7630,"num_omap_keys":14,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"56'774","reported_seq":2997,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.878582+0000","last_change":"2023-10-30T13:59:17.878582+0000","last_active":"2023-10-30T13:59:17.878582+0000","last_peered":"2023-10-30T13:59:17.878582+0000","last_clean":"2023-10-30T13:59:17.878582+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:59:17.878582+0000","last_undegraded":"2023-10-30T13:59:17.878582+0000","last_fullsized":"2023-10-30T13:59:17.878582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'774","last_scrub_stamp":"2023-10-30T13:59:17.878418+0000","last_deep_scrub":"56'774","last_deep_scrub_stamp":"2023-10-30T13:59:17.878418+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.878418+0000","objects_scrubbed":155,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:30.998088+0000","scrub_duration":0.157018885,"objects_trimmed":0,"snaptrim_duration":0.0017556029999999999,"stat_sum":{"num_bytes":7038572,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":115,"num_read_kb":8211,"num_write":611,"num_write_kb":33441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"56'131","reported_seq":200,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.802120+0000","last_change":"2023-10-30T13:59:20.802254+0000","last_active":"2023-10-30T13:59:20.802120+0000","last_peered":"2023-10-30T13:59:20.802120+0000","last_clean":"2023-10-30T13:59:20.802120+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:59:20.802120+0000","last_undegraded":"2023-10-30T13:59:20.802120+0000","last_fullsized":"2023-10-30T13:59:20.802120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'131","last_scrub_stamp":"2023-10-30T13:59:20.801920+0000","last_deep_scrub":"56'131","last_deep_scrub_stamp":"2023-10-30T13:59:20.801920+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.801920+0000","objects_scrubbed":7,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:04.676924+0000","scrub_duration":0.017330934999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":91,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20610,"num_omap_keys":38,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"56'827","reported_seq":3033,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:27.694964+0000","last_change":"2023-10-30T13:59:27.695039+0000","last_active":"2023-10-30T13:59:27.694964+0000","last_peered":"2023-10-30T13:59:27.694964+0000","last_clean":"2023-10-30T13:59:27.694964+0000","last_became_active":"2023-10-30T13:35:49.007835+0000","last_became_peered":"2023-10-30T13:35:49.007835+0000","last_unstale":"2023-10-30T13:59:27.694964+0000","last_undegraded":"2023-10-30T13:59:27.694964+0000","last_fullsized":"2023-10-30T13:59:27.694964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'827","last_scrub_stamp":"2023-10-30T13:59:27.694847+0000","last_deep_scrub":"56'827","last_deep_scrub_stamp":"2023-10-30T13:59:27.694847+0000","last_clean_scrub_stamp":"2023-10-30T13:59:27.694847+0000","objects_scrubbed":154,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:47.134642+0000","scrub_duration":0.465225628,"objects_trimmed":0,"snaptrim_duration":0.0011632000000000001,"stat_sum":{"num_bytes":15868329,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":101,"num_read_kb":5707,"num_write":652,"num_write_kb":27262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"56'390","reported_seq":473,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.497506+0000","last_change":"2023-10-30T13:59:13.497506+0000","last_active":"2023-10-30T13:59:13.497506+0000","last_peered":"2023-10-30T13:59:13.497506+0000","last_clean":"2023-10-30T13:59:13.497506+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:59:13.497506+0000","last_undegraded":"2023-10-30T13:59:13.497506+0000","last_fullsized":"2023-10-30T13:59:13.497506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'390","last_scrub_stamp":"2023-10-30T13:59:13.497014+0000","last_deep_scrub":"56'390","last_deep_scrub_stamp":"2023-10-30T13:59:13.497014+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.497014+0000","objects_scrubbed":6,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:50.064647+0000","scrub_duration":0.074294748999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":61443,"num_write":345,"num_write_kb":20724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"56'809","reported_seq":3047,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.828504+0000","last_change":"2023-10-30T13:59:19.828578+0000","last_active":"2023-10-30T13:59:19.828504+0000","last_peered":"2023-10-30T13:59:19.828504+0000","last_clean":"2023-10-30T13:59:19.828504+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:59:19.828504+0000","last_undegraded":"2023-10-30T13:59:19.828504+0000","last_fullsized":"2023-10-30T13:59:19.828504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'809","last_scrub_stamp":"2023-10-30T13:59:19.828412+0000","last_deep_scrub":"56'809","last_deep_scrub_stamp":"2023-10-30T13:59:19.828412+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.828412+0000","objects_scrubbed":177,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:57.553096+0000","scrub_duration":0.224065866,"objects_trimmed":0,"snaptrim_duration":0.0028607060000000002,"stat_sum":{"num_bytes":18966683,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":128,"num_read_kb":11963,"num_write":623,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"56'719","reported_seq":814,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.839263+0000","last_change":"2023-10-30T13:59:10.839263+0000","last_active":"2023-10-30T13:59:10.839263+0000","last_peered":"2023-10-30T13:59:10.839263+0000","last_clean":"2023-10-30T13:59:10.839263+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:59:10.839263+0000","last_undegraded":"2023-10-30T13:59:10.839263+0000","last_fullsized":"2023-10-30T13:59:10.839263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'719","last_scrub_stamp":"2023-10-30T13:59:10.839113+0000","last_deep_scrub":"56'719","last_deep_scrub_stamp":"2023-10-30T13:59:10.839113+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.839113+0000","objects_scrubbed":8,"log_size":719,"log_dups_size":0,"ondisk_log_size":719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:45.790385+0000","scrub_duration":0.068460164000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":49185,"num_write":677,"num_write_kb":16860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21363,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"56'788","reported_seq":2996,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:14.912002+0000","last_change":"2023-10-30T13:59:14.912126+0000","last_active":"2023-10-30T13:59:14.912002+0000","last_peered":"2023-10-30T13:59:14.912002+0000","last_clean":"2023-10-30T13:59:14.912002+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:59:14.912002+0000","last_undegraded":"2023-10-30T13:59:14.912002+0000","last_fullsized":"2023-10-30T13:59:14.912002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'788","last_scrub_stamp":"2023-10-30T13:59:14.911840+0000","last_deep_scrub":"56'788","last_deep_scrub_stamp":"2023-10-30T13:59:14.911840+0000","last_clean_scrub_stamp":"2023-10-30T13:59:14.911840+0000","objects_scrubbed":165,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:58.599422+0000","scrub_duration":0.31948987000000001,"objects_trimmed":0,"snaptrim_duration":0.0070741379999999998,"stat_sum":{"num_bytes":9996128,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":99,"num_read_kb":8209,"num_write":632,"num_write_kb":23925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"56'1277","reported_seq":1390,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:26.352207+0000","last_change":"2023-10-30T13:59:23.855056+0000","last_active":"2023-10-30T13:59:26.352207+0000","last_peered":"2023-10-30T13:59:26.352207+0000","last_clean":"2023-10-30T13:59:26.352207+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:59:26.352207+0000","last_undegraded":"2023-10-30T13:59:26.352207+0000","last_fullsized":"2023-10-30T13:59:26.352207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1274","last_scrub_stamp":"2023-10-30T13:59:23.854712+0000","last_deep_scrub":"56'1274","last_deep_scrub_stamp":"2023-10-30T13:59:23.854712+0000","last_clean_scrub_stamp":"2023-10-30T13:59:23.854712+0000","objects_scrubbed":10,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:21.240711+0000","scrub_duration":0.074187731000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16257572,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":86059,"num_write":1219,"num_write_kb":20696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29629,"num_omap_keys":53,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"56'671","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.089956+0000","last_change":"2023-10-30T13:45:28.596691+0000","last_active":"2023-10-30T13:59:11.089956+0000","last_peered":"2023-10-30T13:59:11.089956+0000","last_clean":"2023-10-30T13:59:11.089956+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:59:11.089956+0000","last_undegraded":"2023-10-30T13:59:11.089956+0000","last_fullsized":"2023-10-30T13:59:11.089956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070810969999999997,"stat_sum":{"num_bytes":18214186,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":104,"num_read_kb":12299,"num_write":638,"num_write_kb":31808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"56'436","reported_seq":542,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.574205+0000","last_change":"2023-10-30T13:59:08.574261+0000","last_active":"2023-10-30T13:59:08.574205+0000","last_peered":"2023-10-30T13:59:08.574205+0000","last_clean":"2023-10-30T13:59:08.574205+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:59:08.574205+0000","last_undegraded":"2023-10-30T13:59:08.574205+0000","last_fullsized":"2023-10-30T13:59:08.574205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'436","last_scrub_stamp":"2023-10-30T13:59:08.574118+0000","last_deep_scrub":"56'436","last_deep_scrub_stamp":"2023-10-30T13:59:08.574118+0000","last_clean_scrub_stamp":"2023-10-30T13:59:08.574118+0000","objects_scrubbed":8,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:26.795616+0000","scrub_duration":0.024734227000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":77906,"num_write":384,"num_write_kb":12555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":59046,"num_omap_keys":109,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"56'843","reported_seq":3086,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:22.473412+0000","last_change":"2023-10-30T13:59:22.473466+0000","last_active":"2023-10-30T13:59:22.473412+0000","last_peered":"2023-10-30T13:59:22.473412+0000","last_clean":"2023-10-30T13:59:22.473412+0000","last_became_active":"2023-10-30T13:35:49.010460+0000","last_became_peered":"2023-10-30T13:35:49.010460+0000","last_unstale":"2023-10-30T13:59:22.473412+0000","last_undegraded":"2023-10-30T13:59:22.473412+0000","last_fullsized":"2023-10-30T13:59:22.473412+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'843","last_scrub_stamp":"2023-10-30T13:59:22.473336+0000","last_deep_scrub":"56'843","last_deep_scrub_stamp":"2023-10-30T13:59:22.473336+0000","last_clean_scrub_stamp":"2023-10-30T13:59:22.473336+0000","objects_scrubbed":164,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:50:11.354602+0000","scrub_duration":0.279252891,"objects_trimmed":0,"snaptrim_duration":0.001187553,"stat_sum":{"num_bytes":13816511,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":138,"num_read_kb":8210,"num_write":662,"num_write_kb":56040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"56'179","reported_seq":248,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.414864+0000","last_change":"2023-10-30T13:59:19.414972+0000","last_active":"2023-10-30T13:59:19.414864+0000","last_peered":"2023-10-30T13:59:19.414864+0000","last_clean":"2023-10-30T13:59:19.414864+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:59:19.414864+0000","last_undegraded":"2023-10-30T13:59:19.414864+0000","last_fullsized":"2023-10-30T13:59:19.414864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'179","last_scrub_stamp":"2023-10-30T13:59:19.414635+0000","last_deep_scrub":"56'179","last_deep_scrub_stamp":"2023-10-30T13:59:19.414635+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.414635+0000","objects_scrubbed":3,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.618899+0000","scrub_duration":0.0097075200000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":632,"num_write":147,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13012,"num_omap_keys":24,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"56'713","reported_seq":2933,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.520124+0000","last_change":"2023-10-30T13:45:28.599507+0000","last_active":"2023-10-30T13:59:03.520124+0000","last_peered":"2023-10-30T13:59:03.520124+0000","last_clean":"2023-10-30T13:59:03.520124+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:59:03.520124+0000","last_undegraded":"2023-10-30T13:59:03.520124+0000","last_fullsized":"2023-10-30T13:59:03.520124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092364379999999996,"stat_sum":{"num_bytes":16180540,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":126,"num_read_kb":8225,"num_write":670,"num_write_kb":34788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"56'176","reported_seq":248,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.774022+0000","last_change":"2023-10-30T13:59:25.774136+0000","last_active":"2023-10-30T13:59:25.774022+0000","last_peered":"2023-10-30T13:59:25.774022+0000","last_clean":"2023-10-30T13:59:25.774022+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:59:25.774022+0000","last_undegraded":"2023-10-30T13:59:25.774022+0000","last_fullsized":"2023-10-30T13:59:25.774022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'176","last_scrub_stamp":"2023-10-30T13:59:25.773814+0000","last_deep_scrub":"56'176","last_deep_scrub_stamp":"2023-10-30T13:59:25.773814+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.773814+0000","objects_scrubbed":5,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:01.029899+0000","scrub_duration":0.016157685000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4103,"num_write":119,"num_write_kb":4220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31622,"num_omap_keys":58,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"56'659","reported_seq":2853,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.486548+0000","last_change":"2023-10-30T13:45:28.049648+0000","last_active":"2023-10-30T13:59:03.486548+0000","last_peered":"2023-10-30T13:59:03.486548+0000","last_clean":"2023-10-30T13:59:03.486548+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:59:03.486548+0000","last_undegraded":"2023-10-30T13:59:03.486548+0000","last_fullsized":"2023-10-30T13:59:03.486548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045350950000000003,"stat_sum":{"num_bytes":14031523,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":99,"num_read_kb":8208,"num_write":613,"num_write_kb":14857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"56'625","reported_seq":743,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.373340+0000","last_change":"2023-10-30T13:59:09.373401+0000","last_active":"2023-10-30T13:59:09.373340+0000","last_peered":"2023-10-30T13:59:09.373340+0000","last_clean":"2023-10-30T13:59:09.373340+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:59:09.373340+0000","last_undegraded":"2023-10-30T13:59:09.373340+0000","last_fullsized":"2023-10-30T13:59:09.373340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'625","last_scrub_stamp":"2023-10-30T13:59:09.373262+0000","last_deep_scrub":"56'625","last_deep_scrub_stamp":"2023-10-30T13:59:09.373262+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.373262+0000","objects_scrubbed":12,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:19.085134+0000","scrub_duration":0.038380236999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":130,"num_read_kb":41105,"num_write":605,"num_write_kb":12685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":87540,"num_omap_keys":149,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"56'854","reported_seq":3079,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.886946+0000","last_change":"2023-10-30T13:59:13.886999+0000","last_active":"2023-10-30T13:59:13.886946+0000","last_peered":"2023-10-30T13:59:13.886946+0000","last_clean":"2023-10-30T13:59:13.886946+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:59:13.886946+0000","last_undegraded":"2023-10-30T13:59:13.886946+0000","last_fullsized":"2023-10-30T13:59:13.886946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'854","last_scrub_stamp":"2023-10-30T13:59:13.886861+0000","last_deep_scrub":"56'854","last_deep_scrub_stamp":"2023-10-30T13:59:13.886861+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.886861+0000","objects_scrubbed":157,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:39.441296+0000","scrub_duration":0.25638365299999999,"objects_trimmed":0,"snaptrim_duration":0.0089036940000000002,"stat_sum":{"num_bytes":6058105,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":119,"num_read_kb":8210,"num_write":694,"num_write_kb":35073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'356","reported_seq":439,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.554847+0000","last_change":"2023-10-30T13:59:12.555089+0000","last_active":"2023-10-30T13:59:12.554847+0000","last_peered":"2023-10-30T13:59:12.554847+0000","last_clean":"2023-10-30T13:59:12.554847+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:59:12.554847+0000","last_undegraded":"2023-10-30T13:59:12.554847+0000","last_fullsized":"2023-10-30T13:59:12.554847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'356","last_scrub_stamp":"2023-10-30T13:59:12.554618+0000","last_deep_scrub":"56'356","last_deep_scrub_stamp":"2023-10-30T13:59:12.554618+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.554618+0000","objects_scrubbed":6,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:45.923483+0000","scrub_duration":0.048085234999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":28724,"num_write":357,"num_write_kb":8477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20972,"num_omap_keys":39,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"56'782","reported_seq":2990,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.912849+0000","last_change":"2023-10-30T13:59:12.912905+0000","last_active":"2023-10-30T13:59:12.912849+0000","last_peered":"2023-10-30T13:59:12.912849+0000","last_clean":"2023-10-30T13:59:12.912849+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:59:12.912849+0000","last_undegraded":"2023-10-30T13:59:12.912849+0000","last_fullsized":"2023-10-30T13:59:12.912849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'782","last_scrub_stamp":"2023-10-30T13:59:12.912738+0000","last_deep_scrub":"56'782","last_deep_scrub_stamp":"2023-10-30T13:59:12.912738+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.912738+0000","objects_scrubbed":175,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:22.069756+0000","scrub_duration":0.27457796699999998,"objects_trimmed":0,"snaptrim_duration":0.0094781629999999995,"stat_sum":{"num_bytes":6414224,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":99,"num_read_kb":8205,"num_write":617,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"56'644","reported_seq":730,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.281554+0000","last_change":"2023-10-30T13:59:12.281633+0000","last_active":"2023-10-30T13:59:12.281554+0000","last_peered":"2023-10-30T13:59:12.281554+0000","last_clean":"2023-10-30T13:59:12.281554+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:59:12.281554+0000","last_undegraded":"2023-10-30T13:59:12.281554+0000","last_fullsized":"2023-10-30T13:59:12.281554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'644","last_scrub_stamp":"2023-10-30T13:59:12.281461+0000","last_deep_scrub":"56'644","last_deep_scrub_stamp":"2023-10-30T13:59:12.281461+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.281461+0000","objects_scrubbed":6,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:09.939939+0000","scrub_duration":0.025851885000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":33457,"num_write":586,"num_write_kb":12711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33662,"num_omap_keys":63,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"56'676","reported_seq":2881,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.530680+0000","last_change":"2023-10-30T13:45:28.601986+0000","last_active":"2023-10-30T13:59:03.530680+0000","last_peered":"2023-10-30T13:59:03.530680+0000","last_clean":"2023-10-30T13:59:03.530680+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:59:03.530680+0000","last_undegraded":"2023-10-30T13:59:03.530680+0000","last_fullsized":"2023-10-30T13:59:03.530680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010320058,"stat_sum":{"num_bytes":13797516,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":111,"num_read_kb":8210,"num_write":631,"num_write_kb":35965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"56'405","reported_seq":488,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.759775+0000","last_change":"2023-10-30T13:59:09.759865+0000","last_active":"2023-10-30T13:59:09.759775+0000","last_peered":"2023-10-30T13:59:09.759775+0000","last_clean":"2023-10-30T13:59:09.759775+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:59:09.759775+0000","last_undegraded":"2023-10-30T13:59:09.759775+0000","last_fullsized":"2023-10-30T13:59:09.759775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'405","last_scrub_stamp":"2023-10-30T13:59:09.759704+0000","last_deep_scrub":"56'405","last_deep_scrub_stamp":"2023-10-30T13:59:09.759704+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.759704+0000","objects_scrubbed":4,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:26:19.730801+0000","scrub_duration":0.036239831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388678,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":36872,"num_write":370,"num_write_kb":12516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11466,"num_omap_keys":21,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"56'747","reported_seq":2977,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.525310+0000","last_change":"2023-10-30T13:45:27.948762+0000","last_active":"2023-10-30T13:59:03.525310+0000","last_peered":"2023-10-30T13:59:03.525310+0000","last_clean":"2023-10-30T13:59:03.525310+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:59:03.525310+0000","last_undegraded":"2023-10-30T13:59:03.525310+0000","last_fullsized":"2023-10-30T13:59:03.525310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066640720000000001,"stat_sum":{"num_bytes":18395347,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":135,"num_read_kb":4114,"num_write":711,"num_write_kb":24326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"56'227","reported_seq":304,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.292401+0000","last_change":"2023-10-30T13:59:11.292450+0000","last_active":"2023-10-30T13:59:11.292401+0000","last_peered":"2023-10-30T13:59:11.292401+0000","last_clean":"2023-10-30T13:59:11.292401+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:59:11.292401+0000","last_undegraded":"2023-10-30T13:59:11.292401+0000","last_fullsized":"2023-10-30T13:59:11.292401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'227","last_scrub_stamp":"2023-10-30T13:59:11.292333+0000","last_deep_scrub":"56'227","last_deep_scrub_stamp":"2023-10-30T13:59:11.292333+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.292333+0000","objects_scrubbed":5,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:09.441346+0000","scrub_duration":0.024139252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":20494,"num_write":198,"num_write_kb":4363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10920,"num_omap_keys":20,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"56'677","reported_seq":2877,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.514165+0000","last_change":"2023-10-30T13:45:28.598725+0000","last_active":"2023-10-30T13:59:03.514165+0000","last_peered":"2023-10-30T13:59:03.514165+0000","last_clean":"2023-10-30T13:59:03.514165+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:59:03.514165+0000","last_undegraded":"2023-10-30T13:59:03.514165+0000","last_fullsized":"2023-10-30T13:59:03.514165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084474579999999997,"stat_sum":{"num_bytes":18100965,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":106,"num_read_kb":8207,"num_write":635,"num_write_kb":40451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"56'119","reported_seq":223,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:14.603525+0000","last_change":"2023-10-30T13:59:14.603715+0000","last_active":"2023-10-30T13:59:14.603525+0000","last_peered":"2023-10-30T13:59:14.603525+0000","last_clean":"2023-10-30T13:59:14.603525+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:59:14.603525+0000","last_undegraded":"2023-10-30T13:59:14.603525+0000","last_fullsized":"2023-10-30T13:59:14.603525+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'119","last_scrub_stamp":"2023-10-30T13:59:14.603316+0000","last_deep_scrub":"56'119","last_deep_scrub_stamp":"2023-10-30T13:59:14.603316+0000","last_clean_scrub_stamp":"2023-10-30T13:59:14.603316+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:48.801390+0000","scrub_duration":0.012158702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'353","reported_seq":429,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.516018+0000","last_change":"2023-10-30T13:59:11.516196+0000","last_active":"2023-10-30T13:59:11.516018+0000","last_peered":"2023-10-30T13:59:11.516018+0000","last_clean":"2023-10-30T13:59:11.516018+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:59:11.516018+0000","last_undegraded":"2023-10-30T13:59:11.516018+0000","last_fullsized":"2023-10-30T13:59:11.516018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'353","last_scrub_stamp":"2023-10-30T13:59:11.515780+0000","last_deep_scrub":"56'353","last_deep_scrub_stamp":"2023-10-30T13:59:11.515780+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.515780+0000","objects_scrubbed":8,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:30.246583+0000","scrub_duration":0.054846111000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388674,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":20491,"num_write":340,"num_write_kb":8443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31208,"num_omap_keys":58,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'799","reported_seq":3017,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.131857+0000","last_change":"2023-10-30T13:59:17.131904+0000","last_active":"2023-10-30T13:59:17.131857+0000","last_peered":"2023-10-30T13:59:17.131857+0000","last_clean":"2023-10-30T13:59:17.131857+0000","last_became_active":"2023-10-30T13:35:48.847905+0000","last_became_peered":"2023-10-30T13:35:48.847905+0000","last_unstale":"2023-10-30T13:59:17.131857+0000","last_undegraded":"2023-10-30T13:59:17.131857+0000","last_fullsized":"2023-10-30T13:59:17.131857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'799","last_scrub_stamp":"2023-10-30T13:59:17.131773+0000","last_deep_scrub":"56'799","last_deep_scrub_stamp":"2023-10-30T13:59:17.131773+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.131773+0000","objects_scrubbed":182,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:20.503652+0000","scrub_duration":0.14335236700000001,"objects_trimmed":0,"snaptrim_duration":0.002389721,"stat_sum":{"num_bytes":10749534,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":100,"num_read_kb":11,"num_write":647,"num_write_kb":20249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"56'357","reported_seq":454,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:21.575803+0000","last_change":"2023-10-30T13:59:21.575920+0000","last_active":"2023-10-30T13:59:21.575803+0000","last_peered":"2023-10-30T13:59:21.575803+0000","last_clean":"2023-10-30T13:59:21.575803+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:59:21.575803+0000","last_undegraded":"2023-10-30T13:59:21.575803+0000","last_fullsized":"2023-10-30T13:59:21.575803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'357","last_scrub_stamp":"2023-10-30T13:59:21.575603+0000","last_deep_scrub":"56'357","last_deep_scrub_stamp":"2023-10-30T13:59:21.575603+0000","last_clean_scrub_stamp":"2023-10-30T13:59:21.575603+0000","objects_scrubbed":5,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:43.316435+0000","scrub_duration":0.021550125999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":102412,"num_write":307,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16214,"num_omap_keys":31,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"56'824","reported_seq":3051,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.910863+0000","last_change":"2023-10-30T13:59:16.910909+0000","last_active":"2023-10-30T13:59:16.910863+0000","last_peered":"2023-10-30T13:59:16.910863+0000","last_clean":"2023-10-30T13:59:16.910863+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:59:16.910863+0000","last_undegraded":"2023-10-30T13:59:16.910863+0000","last_fullsized":"2023-10-30T13:59:16.910863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'824","last_scrub_stamp":"2023-10-30T13:59:16.910789+0000","last_deep_scrub":"56'824","last_deep_scrub_stamp":"2023-10-30T13:59:16.910789+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.910789+0000","objects_scrubbed":156,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:06.670797+0000","scrub_duration":0.25255747699999997,"objects_trimmed":0,"snaptrim_duration":0.0018610090000000001,"stat_sum":{"num_bytes":24271212,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":117,"num_read_kb":10,"num_write":671,"num_write_kb":33782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"56'1505","reported_seq":2038,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.342054+0000","last_change":"2023-10-30T13:59:10.342107+0000","last_active":"2023-10-30T13:59:10.342054+0000","last_peered":"2023-10-30T13:59:10.342054+0000","last_clean":"2023-10-30T13:59:10.342054+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:59:10.342054+0000","last_undegraded":"2023-10-30T13:59:10.342054+0000","last_fullsized":"2023-10-30T13:59:10.342054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1505","last_scrub_stamp":"2023-10-30T13:59:10.341982+0000","last_deep_scrub":"56'1505","last_deep_scrub_stamp":"2023-10-30T13:59:10.341982+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.341982+0000","objects_scrubbed":9,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:03.471145+0000","scrub_duration":0.046294123999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959716,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":475,"num_read_kb":12479,"num_write":1460,"num_write_kb":17319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8598,"num_omap_keys":16,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'803","reported_seq":3030,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.925329+0000","last_change":"2023-10-30T13:59:20.925381+0000","last_active":"2023-10-30T13:59:20.925329+0000","last_peered":"2023-10-30T13:59:20.925329+0000","last_clean":"2023-10-30T13:59:20.925329+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:59:20.925329+0000","last_undegraded":"2023-10-30T13:59:20.925329+0000","last_fullsized":"2023-10-30T13:59:20.925329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'803","last_scrub_stamp":"2023-10-30T13:59:20.925256+0000","last_deep_scrub":"56'803","last_deep_scrub_stamp":"2023-10-30T13:59:20.925256+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.925256+0000","objects_scrubbed":166,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:15.483272+0000","scrub_duration":0.17049674500000001,"objects_trimmed":0,"snaptrim_duration":0.0022228600000000001,"stat_sum":{"num_bytes":9925305,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":116,"num_read_kb":4117,"num_write":612,"num_write_kb":24793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'634","reported_seq":716,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.440153+0000","last_change":"2023-10-30T13:59:16.440238+0000","last_active":"2023-10-30T13:59:16.440153+0000","last_peered":"2023-10-30T13:59:16.440153+0000","last_clean":"2023-10-30T13:59:16.440153+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:59:16.440153+0000","last_undegraded":"2023-10-30T13:59:16.440153+0000","last_fullsized":"2023-10-30T13:59:16.440153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'634","last_scrub_stamp":"2023-10-30T13:59:16.440060+0000","last_deep_scrub":"56'634","last_deep_scrub_stamp":"2023-10-30T13:59:16.440060+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.440060+0000","objects_scrubbed":4,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:53.542410+0000","scrub_duration":0.021825480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":61440,"num_write":567,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":28266,"num_omap_keys":53,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'840","reported_seq":3051,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.860098+0000","last_change":"2023-10-30T13:59:16.860153+0000","last_active":"2023-10-30T13:59:16.860098+0000","last_peered":"2023-10-30T13:59:16.860098+0000","last_clean":"2023-10-30T13:59:16.860098+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:59:16.860098+0000","last_undegraded":"2023-10-30T13:59:16.860098+0000","last_fullsized":"2023-10-30T13:59:16.860098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'840","last_scrub_stamp":"2023-10-30T13:59:16.860020+0000","last_deep_scrub":"56'840","last_deep_scrub_stamp":"2023-10-30T13:59:16.860020+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.860020+0000","objects_scrubbed":157,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:19.322252+0000","scrub_duration":0.29827281300000003,"objects_trimmed":0,"snaptrim_duration":0.0066434720000000001,"stat_sum":{"num_bytes":26554596,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":101,"num_read_kb":3528,"num_write":689,"num_write_kb":37408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"56'729","reported_seq":803,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.710386+0000","last_change":"2023-10-30T13:59:08.710437+0000","last_active":"2023-10-30T13:59:08.710386+0000","last_peered":"2023-10-30T13:59:08.710386+0000","last_clean":"2023-10-30T13:59:08.710386+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:59:08.710386+0000","last_undegraded":"2023-10-30T13:59:08.710386+0000","last_fullsized":"2023-10-30T13:59:08.710386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'729","last_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_deep_scrub":"56'729","last_deep_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_clean_scrub_stamp":"2023-10-30T13:59:08.710307+0000","objects_scrubbed":3,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:48.485825+0000","scrub_duration":0.018720046000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":18914,"num_write":694,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10538,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"56'698","reported_seq":2902,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.523271+0000","last_change":"2023-10-30T13:45:27.949717+0000","last_active":"2023-10-30T13:59:03.523271+0000","last_peered":"2023-10-30T13:59:03.523271+0000","last_clean":"2023-10-30T13:59:03.523271+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:59:03.523271+0000","last_undegraded":"2023-10-30T13:59:03.523271+0000","last_fullsized":"2023-10-30T13:59:03.523271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064085849999999996,"stat_sum":{"num_bytes":10071990,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":4109,"num_write":659,"num_write_kb":40765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"56'756","reported_seq":827,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:51.384059+0000","last_change":"2023-10-30T13:35:07.300023+0000","last_active":"2023-10-30T13:58:51.384059+0000","last_peered":"2023-10-30T13:58:51.384059+0000","last_clean":"2023-10-30T13:58:51.384059+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:58:51.384059+0000","last_undegraded":"2023-10-30T13:58:51.384059+0000","last_fullsized":"2023-10-30T13:58:51.384059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":707,"num_write_kb":16664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"56'810","reported_seq":3028,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:27.812278+0000","last_change":"2023-10-30T13:59:27.812324+0000","last_active":"2023-10-30T13:59:27.812278+0000","last_peered":"2023-10-30T13:59:27.812278+0000","last_clean":"2023-10-30T13:59:27.812278+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:59:27.812278+0000","last_undegraded":"2023-10-30T13:59:27.812278+0000","last_fullsized":"2023-10-30T13:59:27.812278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'810","last_scrub_stamp":"2023-10-30T13:59:27.812198+0000","last_deep_scrub":"56'810","last_deep_scrub_stamp":"2023-10-30T13:59:27.812198+0000","last_clean_scrub_stamp":"2023-10-30T13:59:27.812198+0000","objects_scrubbed":155,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:19.443546+0000","scrub_duration":0.136984935,"objects_trimmed":0,"snaptrim_duration":0.001735594,"stat_sum":{"num_bytes":6415761,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":108,"num_read_kb":4114,"num_write":644,"num_write_kb":12898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"56'283","reported_seq":370,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.889445+0000","last_change":"2023-10-30T13:59:11.889543+0000","last_active":"2023-10-30T13:59:11.889445+0000","last_peered":"2023-10-30T13:59:11.889445+0000","last_clean":"2023-10-30T13:59:11.889445+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:59:11.889445+0000","last_undegraded":"2023-10-30T13:59:11.889445+0000","last_fullsized":"2023-10-30T13:59:11.889445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'283","last_scrub_stamp":"2023-10-30T13:59:11.889356+0000","last_deep_scrub":"56'283","last_deep_scrub_stamp":"2023-10-30T13:59:11.889356+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.889356+0000","objects_scrubbed":6,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:04.195980+0000","scrub_duration":0.13641584700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":698,"num_write":517,"num_write_kb":2842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":36848,"num_omap_keys":69,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"56'618","reported_seq":2820,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.512401+0000","last_change":"2023-10-30T13:45:28.781685+0000","last_active":"2023-10-30T13:59:03.512401+0000","last_peered":"2023-10-30T13:59:03.512401+0000","last_clean":"2023-10-30T13:59:03.512401+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:59:03.512401+0000","last_undegraded":"2023-10-30T13:59:03.512401+0000","last_fullsized":"2023-10-30T13:59:03.512401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002858771,"stat_sum":{"num_bytes":14417139,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":106,"num_read_kb":4110,"num_write":585,"num_write_kb":24357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"56'124","reported_seq":210,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.849710+0000","last_change":"2023-10-30T13:59:17.849829+0000","last_active":"2023-10-30T13:59:17.849710+0000","last_peered":"2023-10-30T13:59:17.849710+0000","last_clean":"2023-10-30T13:59:17.849710+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:59:17.849710+0000","last_undegraded":"2023-10-30T13:59:17.849710+0000","last_fullsized":"2023-10-30T13:59:17.849710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'124","last_scrub_stamp":"2023-10-30T13:59:17.849515+0000","last_deep_scrub":"56'124","last_deep_scrub_stamp":"2023-10-30T13:59:17.849515+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.849515+0000","objects_scrubbed":7,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:54.554158+0000","scrub_duration":0.025438601000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":20522,"num_write":112,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30984,"num_omap_keys":57,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"56'824","reported_seq":3046,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.809876+0000","last_change":"2023-10-30T13:59:17.809939+0000","last_active":"2023-10-30T13:59:17.809876+0000","last_peered":"2023-10-30T13:59:17.809876+0000","last_clean":"2023-10-30T13:59:17.809876+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:59:17.809876+0000","last_undegraded":"2023-10-30T13:59:17.809876+0000","last_fullsized":"2023-10-30T13:59:17.809876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'824","last_scrub_stamp":"2023-10-30T13:59:17.809790+0000","last_deep_scrub":"56'824","last_deep_scrub_stamp":"2023-10-30T13:59:17.809790+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.809790+0000","objects_scrubbed":162,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:00.364334+0000","scrub_duration":0.16950805699999999,"objects_trimmed":0,"snaptrim_duration":0.0027315170000000001,"stat_sum":{"num_bytes":18657914,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":112,"num_read_kb":8214,"num_write":646,"num_write_kb":32163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"56'148","reported_seq":226,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.625271+0000","last_change":"2023-10-30T13:59:17.625403+0000","last_active":"2023-10-30T13:59:17.625271+0000","last_peered":"2023-10-30T13:59:17.625271+0000","last_clean":"2023-10-30T13:59:17.625271+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:59:17.625271+0000","last_undegraded":"2023-10-30T13:59:17.625271+0000","last_fullsized":"2023-10-30T13:59:17.625271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'148","last_scrub_stamp":"2023-10-30T13:59:17.625064+0000","last_deep_scrub":"56'148","last_deep_scrub_stamp":"2023-10-30T13:59:17.625064+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.625064+0000","objects_scrubbed":3,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:01.340603+0000","scrub_duration":0.011205155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":30625,"num_write":106,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25608,"num_omap_keys":47,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"56'788","reported_seq":2995,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.814579+0000","last_change":"2023-10-30T13:59:13.814677+0000","last_active":"2023-10-30T13:59:13.814579+0000","last_peered":"2023-10-30T13:59:13.814579+0000","last_clean":"2023-10-30T13:59:13.814579+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:59:13.814579+0000","last_undegraded":"2023-10-30T13:59:13.814579+0000","last_fullsized":"2023-10-30T13:59:13.814579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'788","last_scrub_stamp":"2023-10-30T13:59:13.814451+0000","last_deep_scrub":"56'788","last_deep_scrub_stamp":"2023-10-30T13:59:13.814451+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.814451+0000","objects_scrubbed":145,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:05.475962+0000","scrub_duration":0.26681934600000001,"objects_trimmed":0,"snaptrim_duration":0.0075826330000000001,"stat_sum":{"num_bytes":5755271,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":98,"num_read_kb":7,"num_write":639,"num_write_kb":16186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'946","reported_seq":1021,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.799532+0000","last_change":"2023-10-30T13:59:16.799588+0000","last_active":"2023-10-30T13:59:16.799532+0000","last_peered":"2023-10-30T13:59:16.799532+0000","last_clean":"2023-10-30T13:59:16.799532+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:59:16.799532+0000","last_undegraded":"2023-10-30T13:59:16.799532+0000","last_fullsized":"2023-10-30T13:59:16.799532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'946","last_scrub_stamp":"2023-10-30T13:59:16.799459+0000","last_deep_scrub":"56'946","last_deep_scrub_stamp":"2023-10-30T13:59:16.799459+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.799459+0000","objects_scrubbed":6,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:25.410921+0000","scrub_duration":0.051098538999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":20487,"num_write":909,"num_write_kb":8631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16326,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"56'727","reported_seq":2935,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.495804+0000","last_change":"2023-10-30T13:45:28.782068+0000","last_active":"2023-10-30T13:59:03.495804+0000","last_peered":"2023-10-30T13:59:03.495804+0000","last_clean":"2023-10-30T13:59:03.495804+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:59:03.495804+0000","last_undegraded":"2023-10-30T13:59:03.495804+0000","last_fullsized":"2023-10-30T13:59:03.495804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029726169999999999,"stat_sum":{"num_bytes":19889578,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":112,"num_read_kb":12306,"num_write":688,"num_write_kb":31224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"56'485","reported_seq":567,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.778391+0000","last_change":"2023-10-30T13:59:15.778438+0000","last_active":"2023-10-30T13:59:15.778391+0000","last_peered":"2023-10-30T13:59:15.778391+0000","last_clean":"2023-10-30T13:59:15.778391+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:59:15.778391+0000","last_undegraded":"2023-10-30T13:59:15.778391+0000","last_fullsized":"2023-10-30T13:59:15.778391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'485","last_scrub_stamp":"2023-10-30T13:59:15.778326+0000","last_deep_scrub":"56'485","last_deep_scrub_stamp":"2023-10-30T13:59:15.778326+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.778326+0000","objects_scrubbed":8,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:34.958856+0000","scrub_duration":0.04450407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":53262,"num_write":506,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35586,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"56'691","reported_seq":2897,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.493504+0000","last_change":"2023-10-30T13:45:28.781107+0000","last_active":"2023-10-30T13:59:03.493504+0000","last_peered":"2023-10-30T13:59:03.493504+0000","last_clean":"2023-10-30T13:59:03.493504+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:59:03.493504+0000","last_undegraded":"2023-10-30T13:59:03.493504+0000","last_fullsized":"2023-10-30T13:59:03.493504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002600925,"stat_sum":{"num_bytes":9733947,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":111,"num_read_kb":4112,"num_write":647,"num_write_kb":24811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"56'1293","reported_seq":1389,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.627944+0000","last_change":"2023-10-30T13:59:25.628077+0000","last_active":"2023-10-30T13:59:25.627944+0000","last_peered":"2023-10-30T13:59:25.627944+0000","last_clean":"2023-10-30T13:59:25.627944+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:59:25.627944+0000","last_undegraded":"2023-10-30T13:59:25.627944+0000","last_fullsized":"2023-10-30T13:59:25.627944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1293","last_scrub_stamp":"2023-10-30T13:59:25.627729+0000","last_deep_scrub":"56'1293","last_deep_scrub_stamp":"2023-10-30T13:59:25.627729+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.627729+0000","objects_scrubbed":8,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:46:32.130653+0000","scrub_duration":0.082082413000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":81960,"num_write":1252,"num_write_kb":21130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27956,"num_omap_keys":18,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"56'715","reported_seq":2919,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.517958+0000","last_change":"2023-10-30T13:45:28.050416+0000","last_active":"2023-10-30T13:59:03.517958+0000","last_peered":"2023-10-30T13:59:03.517958+0000","last_clean":"2023-10-30T13:59:03.517958+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:59:03.517958+0000","last_undegraded":"2023-10-30T13:59:03.517958+0000","last_fullsized":"2023-10-30T13:59:03.517958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051869250000000002,"stat_sum":{"num_bytes":18626842,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":5015,"num_write":667,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"56'657","reported_seq":2874,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.553657+0000","last_change":"2023-10-30T13:45:27.949465+0000","last_active":"2023-10-30T13:59:03.553657+0000","last_peered":"2023-10-30T13:59:03.553657+0000","last_clean":"2023-10-30T13:59:03.553657+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:59:03.553657+0000","last_undegraded":"2023-10-30T13:59:03.553657+0000","last_fullsized":"2023-10-30T13:59:03.553657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061609869999999997,"stat_sum":{"num_bytes":14245271,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":123,"num_read_kb":12313,"num_write":620,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"56'235","reported_seq":316,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.601546+0000","last_change":"2023-10-30T13:59:24.601686+0000","last_active":"2023-10-30T13:59:24.601546+0000","last_peered":"2023-10-30T13:59:24.601546+0000","last_clean":"2023-10-30T13:59:24.601546+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:59:24.601546+0000","last_undegraded":"2023-10-30T13:59:24.601546+0000","last_fullsized":"2023-10-30T13:59:24.601546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'235","last_scrub_stamp":"2023-10-30T13:59:24.601285+0000","last_deep_scrub":"56'235","last_deep_scrub_stamp":"2023-10-30T13:59:24.601285+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.601285+0000","objects_scrubbed":4,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:28.578041+0000","scrub_duration":0.016429274000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":24606,"num_write":206,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35337,"num_omap_keys":63,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"56'713","reported_seq":2919,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.528417+0000","last_change":"2023-10-30T13:45:27.948077+0000","last_active":"2023-10-30T13:59:03.528417+0000","last_peered":"2023-10-30T13:59:03.528417+0000","last_clean":"2023-10-30T13:59:03.528417+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:59:03.528417+0000","last_undegraded":"2023-10-30T13:59:03.528417+0000","last_fullsized":"2023-10-30T13:59:03.528417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061457359999999997,"stat_sum":{"num_bytes":11313833,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":112,"num_read_kb":16,"num_write":670,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"56'950","reported_seq":1045,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:58:58.285824+0000","last_change":"2023-10-30T13:35:07.302305+0000","last_active":"2023-10-30T13:58:58.285824+0000","last_peered":"2023-10-30T13:58:58.285824+0000","last_clean":"2023-10-30T13:58:58.285824+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:58:58.285824+0000","last_undegraded":"2023-10-30T13:58:58.285824+0000","last_fullsized":"2023-10-30T13:58:58.285824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:06.235948+0000","last_clean_scrub_stamp":"2023-10-30T13:35:06.235948+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":103644,"num_write":908,"num_write_kb":16901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"56'829","reported_seq":3061,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:26.753705+0000","last_change":"2023-10-30T13:59:26.753762+0000","last_active":"2023-10-30T13:59:26.753705+0000","last_peered":"2023-10-30T13:59:26.753705+0000","last_clean":"2023-10-30T13:59:26.753705+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:59:26.753705+0000","last_undegraded":"2023-10-30T13:59:26.753705+0000","last_fullsized":"2023-10-30T13:59:26.753705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'829","last_scrub_stamp":"2023-10-30T13:59:26.753633+0000","last_deep_scrub":"56'829","last_deep_scrub_stamp":"2023-10-30T13:59:26.753633+0000","last_clean_scrub_stamp":"2023-10-30T13:59:26.753633+0000","objects_scrubbed":184,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:01:43.957619+0000","scrub_duration":0.27683662399999998,"objects_trimmed":0,"snaptrim_duration":0.0029730300000000002,"stat_sum":{"num_bytes":19886092,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":121,"num_read_kb":4113,"num_write":647,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"56'590","reported_seq":679,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.115956+0000","last_change":"2023-10-30T13:59:13.116019+0000","last_active":"2023-10-30T13:59:13.115956+0000","last_peered":"2023-10-30T13:59:13.115956+0000","last_clean":"2023-10-30T13:59:13.115956+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:59:13.115956+0000","last_undegraded":"2023-10-30T13:59:13.115956+0000","last_fullsized":"2023-10-30T13:59:13.115956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'590","last_scrub_stamp":"2023-10-30T13:59:13.115857+0000","last_deep_scrub":"56'590","last_deep_scrub_stamp":"2023-10-30T13:59:13.115857+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.115857+0000","objects_scrubbed":4,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:55.093851+0000","scrub_duration":0.036041838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":72355,"num_write":540,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14742,"num_omap_keys":27,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"56'660","reported_seq":2855,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.913025+0000","last_change":"2023-10-30T13:45:28.781420+0000","last_active":"2023-10-30T13:59:08.913025+0000","last_peered":"2023-10-30T13:59:08.913025+0000","last_clean":"2023-10-30T13:59:08.913025+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:59:08.913025+0000","last_undegraded":"2023-10-30T13:59:08.913025+0000","last_fullsized":"2023-10-30T13:59:08.913025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026777319999999999,"stat_sum":{"num_bytes":7974576,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":98,"num_read_kb":4107,"num_write":628,"num_write_kb":17606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"56'364","reported_seq":468,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.646097+0000","last_change":"2023-10-30T13:59:15.646200+0000","last_active":"2023-10-30T13:59:15.646097+0000","last_peered":"2023-10-30T13:59:15.646097+0000","last_clean":"2023-10-30T13:59:15.646097+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:59:15.646097+0000","last_undegraded":"2023-10-30T13:59:15.646097+0000","last_fullsized":"2023-10-30T13:59:15.646097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'364","last_scrub_stamp":"2023-10-30T13:59:15.645995+0000","last_deep_scrub":"56'364","last_deep_scrub_stamp":"2023-10-30T13:59:15.645995+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.645995+0000","objects_scrubbed":11,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:40.607663+0000","scrub_duration":0.031506254999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4418509,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":57427,"num_write":339,"num_write_kb":8706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35486,"num_omap_keys":15,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":925185184,"num_objects":10324,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":7268,"num_read_kb":400185,"num_write":41527,"num_write_kb":1950152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1902518272,"data_stored":1848830272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47265,"ondisk_log_size":47265,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":260115313,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":1553,"num_read_kb":1442867,"num_write":16557,"num_write_kb":377496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":756644,"num_omap_keys":1301,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520388608,"data_stored":520230626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17515,"ondisk_log_size":17515,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509740,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445620,"kb_used_data":333932,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93926220,"statfs":{"total":96636764160,"available":96180449280,"internally_reserved":0,"allocated":341946368,"data_stored":333745462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":114358731},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85199999999999998,"15min":0.81200000000000006},"min":{"1min":0.433,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.1830000000000001,"5min":0.96299999999999997,"15min":0.90600000000000003},"min":{"1min":0.53500000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.3499999999999996,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":0.79800000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.86199999999999999,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.0289999999999999,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":0.871},{"interface":"front","average":{"1min":1.02,"5min":0.95499999999999996,"15min":0.92300000000000004},"min":{"1min":0.57399999999999995,"5min":0.45200000000000001,"15min":0.33400000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.625},"last":1.151}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0389999999999999,"15min":0.97299999999999998},"min":{"1min":0.626,"5min":0.51300000000000001,"15min":0.32000000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.91700000000000004,"15min":0.94499999999999995},"min":{"1min":0.63300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":4.1120000000000001},"last":1.0660000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.75600000000000001,"5min":0.55300000000000005,"15min":0.45900000000000002},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":4.2069999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":4.4189999999999996,"5min":1.645,"15min":1.204},"min":{"1min":0.83299999999999996,"5min":0.47199999999999998,"15min":0.44400000000000001},"max":{"1min":59.844999999999999,"5min":59.844999999999999,"15min":59.844999999999999},"last":0.75600000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1080000000000001,"15min":1.089},"min":{"1min":0.85899999999999999,"5min":0.623,"15min":0.38100000000000001},"max":{"1min":1.853,"5min":1.853,"15min":3.1059999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.3220000000000001,"15min":1.2769999999999999},"min":{"1min":1.137,"5min":0.754,"15min":0.42399999999999999},"max":{"1min":1.907,"5min":1.907,"15min":3.198},"last":1.3899999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0049999999999999,"15min":1.093},"min":{"1min":0.65600000000000003,"5min":0.53100000000000003,"15min":0.40300000000000002},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":4.2759999999999998},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.258,"15min":1.238},"min":{"1min":0.80000000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5329999999999999,"5min":1.8380000000000001,"15min":3.4380000000000002},"last":1.47}]},{"osd":6,"last update":"Mon Oct 30 13:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92200000000000004,"15min":1.091},"min":{"1min":0.55400000000000005,"5min":0.48999999999999999,"15min":0.48399999999999999},"max":{"1min":1.3109999999999999,"5min":1.429,"15min":3.9630000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.458,"5min":1.131,"15min":1.216},"min":{"1min":0.84399999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":7.7060000000000004,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":1.1060000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509739,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421252,"kb_used_data":317884,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93950588,"statfs":{"total":96636764160,"available":96205402112,"internally_reserved":0,"allocated":325513216,"data_stored":317280681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":105839048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":9.3320000000000007},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0369999999999999,"15min":0.72399999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.26100000000000001},"max":{"1min":0.71599999999999997,"5min":38.082999999999998,"15min":38.082999999999998},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.70199999999999996,"15min":0.60699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":11.599,"15min":11.599},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.70899999999999996,"15min":0.66100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":15.715999999999999,"15min":15.715999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.72699999999999998,"15min":0.66700000000000004},"min":{"1min":0.437,"5min":0.35799999999999998,"15min":0.28999999999999998},"max":{"1min":0.83899999999999997,"5min":11.646000000000001,"15min":12.351000000000001},"last":0.441},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.755,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.83399999999999996,"5min":15.641999999999999,"15min":15.641999999999999},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.016,"15min":0.92000000000000004},"min":{"1min":0.59999999999999998,"5min":0.55400000000000005,"15min":0.51600000000000001},"max":{"1min":0.98899999999999999,"5min":15.759,"15min":15.759},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.298,"15min":1.0640000000000001},"min":{"1min":0.58499999999999996,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":38.137999999999998,"15min":38.137999999999998},"last":0.622}]},{"osd":5,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.4139999999999999,"15min":1.097},"min":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.52600000000000002},"max":{"1min":1.036,"5min":40.808,"15min":40.808},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.0309999999999999,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.55800000000000005,"15min":0.49299999999999999},"max":{"1min":1.0660000000000001,"5min":16.498000000000001,"15min":19.835999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.96499999999999997,"15min":0.91800000000000004},"min":{"1min":0.621,"5min":0.56799999999999995,"15min":0.53500000000000003},"max":{"1min":1.2370000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.998,"15min":0.96099999999999997},"min":{"1min":0.68500000000000005,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":1.091,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.748}]},{"osd":7,"last update":"Mon Oct 30 13:58:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1020000000000001,"15min":0.95799999999999996},"min":{"1min":0.72499999999999998,"5min":0.60599999999999998,"15min":0.59999999999999998},"max":{"1min":1.2549999999999999,"5min":16.082999999999998,"15min":16.082999999999998},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.88,"5min":1.5609999999999999,"15min":1.111},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.58199999999999996},"max":{"1min":1.228,"5min":52.902999999999999,"15min":52.902999999999999},"last":0.72999999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254304,"kb_used_data":187096,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94117536,"statfs":{"total":96636764160,"available":96376356864,"internally_reserved":0,"allocated":191586304,"data_stored":186614303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.45700000000000002},"min":{"1min":0.32100000000000001,"5min":0.251,"15min":0.246},"max":{"1min":0.83199999999999996,"5min":3.407,"15min":3.8969999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.371,"5min":0.35299999999999998,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":5.6980000000000004,"15min":5.6980000000000004},"last":0.66300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.51000000000000001,"15min":0.60099999999999998},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68100000000000005,"5min":5.5750000000000002,"15min":36.109000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58899999999999997,"5min":5.4569999999999999,"15min":8.1890000000000001},"last":0.72399999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.63500000000000001,"15min":0.68999999999999995},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.29199999999999998},"max":{"1min":0.73999999999999999,"5min":5.532,"15min":35.890999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66800000000000004,"15min":0.69799999999999995},"min":{"1min":0.48199999999999998,"5min":0.42899999999999999,"15min":0.371},"max":{"1min":0.78900000000000003,"5min":5.2830000000000004,"15min":22.846},"last":0.76700000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84099999999999997},"min":{"1min":0.67000000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0549999999999999,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.496,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":5.6059999999999999,"15min":7.8220000000000001},"last":0.83599999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.93999999999999995,"5min":5.4260000000000002,"15min":9.9109999999999996},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.86499999999999999},"min":{"1min":0.70799999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":5.4870000000000001,"15min":5.4870000000000001},"last":0.91500000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:58:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":1.2030000000000001,"15min":0.95999999999999996},"min":{"1min":0.64100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0960000000000001,"15min":0.97599999999999998},"min":{"1min":0.68300000000000005,"5min":0.60099999999999998,"15min":0.56599999999999995},"max":{"1min":15.975,"5min":15.975,"15min":15.975},"last":1.0089999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.75700000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.169,"5min":5.3899999999999997,"15min":5.3899999999999997},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.91900000000000004,"15min":0.91600000000000004},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53700000000000003},"max":{"1min":1.0800000000000001,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":1.0669999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509738,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357220,"kb_used_data":245596,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94014620,"statfs":{"total":96636764160,"available":96270970880,"internally_reserved":0,"allocated":251490304,"data_stored":246176700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114293190},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.437},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.245},"max":{"1min":0.67200000000000004,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":0.442},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55100000000000005,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":0.66700000000000004,"5min":2.5920000000000001,"15min":22.149000000000001},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.315},"max":{"1min":0.749,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.75900000000000001},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":2.8130000000000002,"15min":54.530000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.67800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34100000000000003},"max":{"1min":0.83799999999999997,"5min":2.879,"15min":25.457000000000001},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78300000000000003,"15min":0.81299999999999994},"min":{"1min":0.53900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94699999999999995,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.79500000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":1.054,"5min":2.101,"15min":2.101},"last":0.63}]},{"osd":5,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83699999999999997,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.505},"max":{"1min":1.012,"5min":2.7810000000000001,"15min":11.164999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83599999999999997},"min":{"1min":0.624,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":0.94399999999999995,"5min":2.8460000000000001,"15min":2.8820000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.83699999999999997},"min":{"1min":0.57299999999999995,"5min":0.55600000000000005,"15min":0.497},"max":{"1min":0.95399999999999996,"5min":2.903,"15min":5.9180000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.85299999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":1.9930000000000001,"15min":2.2629999999999999},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:58:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.002,"15min":0.96899999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":12.221,"5min":12.221,"15min":12.221},"last":1.085},{"interface":"front","average":{"1min":1.373,"5min":1.0209999999999999,"15min":0.94499999999999995},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":12.153,"5min":12.153,"15min":12.153},"last":1.1100000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542444,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344808,"kb_used_data":247072,"kb_used_omap":1,"kb_used_meta":97726,"kb_avail":94027032,"statfs":{"total":96636764160,"available":96283680768,"internally_reserved":0,"allocated":253001728,"data_stored":247844657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":100071879},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.1240000000000001,"15min":1.042},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.41999999999999998},"max":{"1min":1.0389999999999999,"5min":19.741,"15min":19.741},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.131,"15min":0.99099999999999999},"min":{"1min":0.69599999999999995,"5min":0.55800000000000005,"15min":0.34899999999999998},"max":{"1min":1.3600000000000001,"5min":13.863,"15min":13.863},"last":1.0449999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93700000000000006,"15min":0.86799999999999999},"min":{"1min":0.55600000000000005,"5min":0.496,"15min":0.46100000000000002},"max":{"1min":1.0669999999999999,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.079,"5min":1.1819999999999999,"15min":1.0289999999999999},"min":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.46800000000000003},"max":{"1min":1.2929999999999999,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":1.117}]},{"osd":2,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96199999999999997,"15min":0.97699999999999998},"min":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.335,"5min":3.5329999999999999,"15min":3.7709999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.095,"15min":1.083},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.4059999999999999,"5min":3.407,"15min":4.6840000000000002},"last":0.72499999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.97599999999999998,"15min":1.014},"min":{"1min":0.67100000000000004,"5min":0.52800000000000002,"15min":0.46700000000000003},"max":{"1min":1.359,"5min":3.7999999999999998,"15min":4.7629999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.171,"15min":1.0840000000000001},"min":{"1min":0.92900000000000005,"5min":0.59299999999999997,"15min":0.54600000000000004},"max":{"1min":1.4179999999999999,"5min":3.968,"15min":3.968},"last":1.1719999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2290000000000001,"15min":1.208},"min":{"1min":0.79600000000000004,"5min":0.55800000000000005,"15min":0.51000000000000001},"max":{"1min":1.4710000000000001,"5min":3.8900000000000001,"15min":16.204000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1519999999999999,"15min":1.2649999999999999},"min":{"1min":0.63700000000000001,"5min":0.622,"15min":0.42699999999999999},"max":{"1min":1.399,"5min":19.398,"15min":43.198},"last":1.27}]},{"osd":5,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.98499999999999999,"15min":1.0489999999999999},"min":{"1min":0.70899999999999996,"5min":0.47699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98399999999999999,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3129999999999999,"15min":1.3009999999999999},"min":{"1min":1.1879999999999999,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":1.522,"5min":3.7269999999999999,"15min":3.802},"last":1.2210000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:58:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.833,"15min":1.5880000000000001},"min":{"1min":0.83499999999999996,"5min":0.58899999999999997,"15min":0.55400000000000005},"max":{"1min":1.2709999999999999,"5min":70.335999999999999,"15min":73.228999999999999},"last":1.01},{"interface":"front","average":{"1min":1.079,"5min":1.675,"15min":1.637},"min":{"1min":0.82699999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.381,"5min":42.164999999999999,"15min":73.289000000000001},"last":0.875}]}]},{"osd":5,"up_from":14,"seq":60129542442,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416780,"kb_used_data":335620,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93955060,"statfs":{"total":96636764160,"available":96209981440,"internally_reserved":0,"allocated":343674880,"data_stored":335552063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.89900000000000002,"15min":0.879},"min":{"1min":0.54600000000000004,"5min":0.46999999999999997,"15min":0.40400000000000003},"max":{"1min":3.9159999999999999,"5min":3.9159999999999999,"15min":5.0599999999999996},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.83099999999999996,"15min":0.872},"min":{"1min":0.52900000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0569999999999999,"5min":1.5620000000000001,"15min":3.4849999999999999},"last":1.0009999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.92600000000000005,"15min":0.871},"min":{"1min":0.55300000000000005,"5min":0.40600000000000003,"15min":0.33600000000000002},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.85099999999999998,"15min":0.90000000000000002},"min":{"1min":0.50700000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.8740000000000001,"5min":3.8740000000000001,"15min":3.8740000000000001},"last":1.0549999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95899999999999996,"15min":0.95599999999999996},"min":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.51100000000000001},"max":{"1min":1.462,"5min":1.897,"15min":5.3710000000000004},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1279999999999999,"15min":1.056},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.53200000000000003},"max":{"1min":1.369,"5min":2.081,"15min":2.887},"last":1.3979999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.095,"15min":1.0620000000000001},"min":{"1min":0.82999999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":5.476},"last":1.4630000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.97599999999999998,"15min":1.0349999999999999},"min":{"1min":0.67200000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1779999999999999,"5min":1.52,"15min":3.8500000000000001},"last":1.034}]},{"osd":4,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95599999999999996,"15min":0.98699999999999999},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.38400000000000001},"max":{"1min":1.3360000000000001,"5min":1.847,"15min":4.1710000000000003},"last":1.244},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.66900000000000004,"5min":0.49199999999999999,"15min":0.41799999999999998},"max":{"1min":1.3009999999999999,"5min":1.7270000000000001,"15min":4.2519999999999998},"last":0.83799999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1899999999999999,"15min":1.1739999999999999},"min":{"1min":0.747,"5min":0.65000000000000002,"15min":0.621},"max":{"1min":1.546,"5min":2.177,"15min":5.6029999999999998},"last":1.147},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1459999999999999,"15min":1.1739999999999999},"min":{"1min":0.69499999999999995,"5min":0.56100000000000005,"15min":0.41899999999999998},"max":{"1min":1.403,"5min":1.839,"15min":4.077},"last":1.1990000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.147,"15min":1.23},"min":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.45100000000000001},"max":{"1min":1.4650000000000001,"5min":1.7949999999999999,"15min":9.9390000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.2430000000000001,"15min":1.2889999999999999},"min":{"1min":0.97399999999999998,"5min":0.78000000000000003,"15min":0.48499999999999999},"max":{"1min":1.415,"5min":1.907,"15min":4.3399999999999999},"last":1.357}]}]},{"osd":4,"up_from":14,"seq":60129542442,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426512,"kb_used_data":337032,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93945328,"statfs":{"total":96636764160,"available":96200015872,"internally_reserved":0,"allocated":345120768,"data_stored":336789947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.872},"min":{"1min":0.45000000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.3159999999999998,"5min":8.7680000000000007,"15min":8.7680000000000007},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.329,"5min":0.999,"15min":0.89600000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":10.52,"5min":10.52,"15min":10.52},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.82299999999999995,"15min":0.88900000000000001},"min":{"1min":0.51500000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":1.236,"15min":5.8010000000000002},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.86499999999999999,"15min":0.874},"min":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.73999999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.55400000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":3.323},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0329999999999999,"15min":0.98199999999999998},"min":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.0549999999999997,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":0.82199999999999995}]},{"osd":3,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.105,"15min":1.0209999999999999},"min":{"1min":0.69299999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.268,"5min":8.0570000000000004,"15min":8.0570000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.0529999999999999,"15min":0.995},"min":{"1min":0.66800000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1459999999999999,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.92300000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.018,"15min":1.0600000000000001},"min":{"1min":0.82699999999999996,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.325,"5min":4.0030000000000001,"15min":8.4550000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.175,"15min":1.171},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.57699999999999996},"max":{"1min":1.3959999999999999,"5min":4.0609999999999999,"15min":8.5399999999999991},"last":1.022}]},{"osd":6,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.2010000000000001,"15min":1.0900000000000001},"min":{"1min":0.56000000000000005,"5min":0.503,"15min":0.45100000000000001},"max":{"1min":1.3680000000000001,"5min":16.989999999999998,"15min":16.989999999999998},"last":1.125},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1890000000000001,"15min":1.1599999999999999},"min":{"1min":0.81599999999999995,"5min":0.41299999999999998,"15min":0.36699999999999999},"max":{"1min":1.4690000000000001,"5min":4.4530000000000003,"15min":4.4530000000000003},"last":1.2829999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:42 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1719999999999999,"15min":1.163},"min":{"1min":0.84199999999999997,"5min":0.45100000000000001,"15min":0.42199999999999999},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":3.4510000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.298,"15min":1.3049999999999999},"min":{"1min":1.0269999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.5470000000000002},"last":1.5309999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542444,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465840,"kb_used_data":368040,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93906000,"statfs":{"total":96636764160,"available":96159744000,"internally_reserved":0,"allocated":376872960,"data_stored":368821573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":100137414},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.98899999999999999,"15min":0.63900000000000001},"min":{"1min":0.32900000000000001,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":0.63900000000000001,"5min":45.311,"15min":45.311},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.59599999999999997},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.27200000000000002},"max":{"1min":0.72599999999999998,"5min":2.2410000000000001,"15min":27.616},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.29099999999999998},"max":{"1min":0.82999999999999996,"5min":0.88100000000000001,"15min":2.5449999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.44,"5min":0.40899999999999997,"15min":0.35199999999999998},"max":{"1min":0.752,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.67500000000000004,"15min":0.56100000000000005},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":0.79300000000000004,"5min":12.368,"15min":12.368},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.75800000000000001,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.40899999999999997},"max":{"1min":0.83299999999999996,"5min":12.442,"15min":12.442},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84799999999999998,"15min":0.85999999999999999},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.41599999999999998},"max":{"1min":1,"5min":2.6949999999999998,"15min":9.4979999999999993},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.871,"15min":0.86499999999999999},"min":{"1min":0.629,"5min":0.60199999999999998,"15min":0.46700000000000003},"max":{"1min":1.1220000000000001,"5min":2.6440000000000001,"15min":9.5429999999999993},"last":1.1699999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.88,"15min":0.878},"min":{"1min":0.72199999999999998,"5min":0.47099999999999997,"15min":0.317},"max":{"1min":1.087,"5min":1.163,"15min":11.026},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.375,"15min":1.0309999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.46999999999999997},"max":{"1min":1.1759999999999999,"5min":44.225999999999999,"15min":44.225999999999999},"last":0.95499999999999996}]},{"osd":6,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88,"15min":0.873},"min":{"1min":0.73899999999999999,"5min":0.63,"15min":0.48799999999999999},"max":{"1min":1.0649999999999999,"5min":2.5169999999999999,"15min":2.9300000000000002},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.435},"max":{"1min":1.133,"5min":1.236,"15min":11.087999999999999},"last":0.86899999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.91400000000000003,"15min":0.879},"min":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.50900000000000001},"max":{"1min":1.0740000000000001,"5min":5.6849999999999996,"15min":5.6849999999999996},"last":1.2529999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.93200000000000005,"15min":0.93300000000000005},"min":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.57199999999999995},"max":{"1min":1.0840000000000001,"5min":2.6030000000000002,"15min":9.5890000000000004},"last":1.073}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50573312,"data_stored":50555931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46346240,"data_stored":46320604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75263616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":71393280,"data_stored":71379650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67129344,"data_stored":67110274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":75173888,"data_stored":75157496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":67350528,"data_stored":67333247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325066752,"data_stored":317352305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183717888,"data_stored":178743923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144601088,"data_stored":139970730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":249585664,"data_stored":241694096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":273088512,"data_stored":265087328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":275906560,"data_stored":268118820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":177188864,"data_stored":172364192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":273362944,"data_stored":265498878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:59:31.927 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.928 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.928 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.929 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.929 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.929 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.930 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.930 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.930 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.931 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.931 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.931 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.932 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.932 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.932 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.933 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.933 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.933 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.934 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.934 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.935 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.935 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.935 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.936 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.936 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.936 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.937 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.937 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.938 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.938 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.938 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.939 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.939 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.940 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.940 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.941 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.941 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.941 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.942 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.942 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T13:35:06.235948+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=6, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.943 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:31.943 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:59:51.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:59:52.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.095+0000 7f1408005640 1 Processor -- start 2023-10-30T13:59:52.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.095+0000 7f1408005640 1 -- start start 2023-10-30T13:59:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.095+0000 7f1408005640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dbe0 0x7f140006e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.095+0000 7f1408005640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1400071f30 0x7f1400156880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.095+0000 7f1408005640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1400156dc0 con 0x7f140006dfc0 2023-10-30T13:59:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.095+0000 7f1408005640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140006ea70 con 0x7f1400071f30 2023-10-30T13:59:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.095+0000 7f1408005640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1400070c40 con 0x7f140006dbe0 2023-10-30T13:59:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1400071f30 0x7f1400156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f1400071f30 0x7f1400156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52022/0 (socket says 172.21.15.31:52022) 2023-10-30T13:59:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 -- 172.21.15.31:0/3194007668 learned_addr learned my addr 172.21.15.31:0/3194007668 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f1405da1640 1 --2- 172.21.15.31:0/3194007668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dbe0 0x7f140006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 -- 172.21.15.31:0/3194007668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dbe0 msgr2=0x7f140006e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:59:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 --2- 172.21.15.31:0/3194007668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dbe0 0x7f140006e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 -- 172.21.15.31:0/3194007668 >> v1:172.21.15.33:6789/0 conn(0x7f140006dfc0 legacy=0x7f1400157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 -- 172.21.15.31:0/3194007668 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1400070e90 con 0x7f1400071f30 2023-10-30T13:59:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f14065a2640 1 --2- 172.21.15.31:0/3194007668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f1400156880 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009e50 tx=0x7f13f801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9206f061c30603a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f1404d9f640 1 -- 172.21.15.31:0/3194007668 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80a5790 con 0x7f1400071f30 2023-10-30T13:59:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f1404d9f640 1 -- 172.21.15.31:0/3194007668 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f80a58f0 con 0x7f1400071f30 2023-10-30T13:59:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.099+0000 7f1404d9f640 1 -- 172.21.15.31:0/3194007668 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80a5c80 con 0x7f1400071f30 2023-10-30T13:59:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- 172.21.15.31:0/3194007668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 msgr2=0x7f1400156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 --2- 172.21.15.31:0/3194007668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f1400156880 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009e50 tx=0x7f13f801c950 comp rx=0 tx=0).stop 2023-10-30T13:59:52.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- 172.21.15.31:0/3194007668 shutdown_connections 2023-10-30T13:59:52.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 --2- 172.21.15.31:0/3194007668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f1400156880 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 --2- 172.21.15.31:0/3194007668 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dbe0 0x7f140006e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- 172.21.15.31:0/3194007668 >> 172.21.15.31:0/3194007668 conn(0x7f14000c7f60 msgr2=0x7f14000c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:52.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- 172.21.15.31:0/3194007668 shutdown_connections 2023-10-30T13:59:52.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- 172.21.15.31:0/3194007668 wait complete. 2023-10-30T13:59:52.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 Processor -- start 2023-10-30T13:59:52.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- start start 2023-10-30T13:59:52.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f140006dbe0 0x7f1400154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:52.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dfc0 0x7f14000c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f14001f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001495f0 con 0x7f1400071f30 2023-10-30T13:59:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001f1c50 con 0x7f140006dbe0 2023-10-30T13:59:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1408005640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1400153df0 con 0x7f140006dfc0 2023-10-30T13:59:52.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f14001f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:52.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f14001f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52024/0 (socket says 172.21.15.31:52024) 2023-10-30T13:59:52.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 -- 172.21.15.31:0/11344846 learned_addr learned my addr 172.21.15.31:0/11344846 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T13:59:52.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f1405da1640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f140006dbe0 0x7f1400154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:52.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14055a0640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dfc0 0x7f14000c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:52.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 -- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dfc0 msgr2=0x7f14000c7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:52.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dfc0 0x7f14000c7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 -- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f140006dbe0 msgr2=0x7f1400154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f140006dbe0 0x7f1400154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.103+0000 7f14065a2640 1 -- 172.21.15.31:0/11344846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8009b30 con 0x7f1400071f30 2023-10-30T13:59:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f14065a2640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f14001f20e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f13f801a580 tx=0x7f13f80196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f13f67fc640 1 -- 172.21.15.31:0/11344846 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80b8670 con 0x7f1400071f30 2023-10-30T13:59:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f13f67fc640 1 -- 172.21.15.31:0/11344846 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f80b87d0 con 0x7f1400071f30 2023-10-30T13:59:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1400149800 con 0x7f1400071f30 2023-10-30T13:59:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f13f67fc640 1 -- 172.21.15.31:0/11344846 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80b8b80 con 0x7f1400071f30 2023-10-30T13:59:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14001540d0 con 0x7f1400071f30 2023-10-30T13:59:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14001f1dc0 con 0x7f1400071f30 2023-10-30T13:59:52.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f13f67fc640 1 -- 172.21.15.31:0/11344846 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13f80b8ce0 con 0x7f1400071f30 2023-10-30T13:59:52.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f13f67fc640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f13e40664a0 0x7f13e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:59:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f13f67fc640 1 -- 172.21.15.31:0/11344846 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f13f811f810 con 0x7f1400071f30 2023-10-30T13:59:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.107+0000 7f1405da1640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f13e40664a0 0x7f13e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:59:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.111+0000 7f1405da1640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f13e40664a0 0x7f13e4068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009a50 tx=0x7f13fc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:59:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.115+0000 7f13f67fc640 1 -- 172.21.15.31:0/11344846 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13f80eca40 con 0x7f1400071f30 2023-10-30T13:59:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.267+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1400071a10 con 0x7f13e40664a0 2023-10-30T13:59:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f13f67fc640 1 -- 172.21.15.31:0/11344846 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294204 (secure 0 0 0) 0x7f13fc0119a0 con 0x7f13e40664a0 2023-10-30T13:59:52.295 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T13:59:52.297 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T13:59:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f13e40664a0 msgr2=0x7f13e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f13e40664a0 0x7f13e4068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009a50 tx=0x7f13fc008040 comp rx=0 tx=0).stop 2023-10-30T13:59:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 msgr2=0x7f14001f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:59:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f14001f20e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f13f801a580 tx=0x7f13f80196f0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 shutdown_connections 2023-10-30T13:59:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f13e40664a0 0x7f13e4068960 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1400071f30 0x7f14001f20e0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f140006dfc0 0x7f14000c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 --2- 172.21.15.31:0/11344846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f140006dbe0 0x7f1400154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:59:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 >> 172.21.15.31:0/11344846 conn(0x7f14000c7f60 msgr2=0x7f140014a750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:59:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 shutdown_connections 2023-10-30T13:59:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T13:59:52.291+0000 7f1408005640 1 -- 172.21.15.31:0/11344846 wait complete. 2023-10-30T13:59:52.315 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2023-10-30T13:59:51.732454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186181137,"num_objects":10532,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10532,"num_whiteouts":0,"num_read":8937,"num_read_kb":1843152,"num_write":58260,"num_write_kb":2330355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":782235,"num_omap_keys":1350,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69541,"ondisk_log_size":69541,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3124584,"kb_used_data":2372840,"kb_used_omap":12,"kb_used_meta":751667,"kb_avail":751850136,"statfs":{"total":773094113280,"available":769894539264,"internally_reserved":0,"allocated":2429788160,"data_stored":2373405930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":769707576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":185584,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005177"},"pg_stats":[{"pgid":"3.3f","version":"56'674","reported_seq":2894,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.530747+0000","last_change":"2023-10-30T13:45:28.601107+0000","last_active":"2023-10-30T13:59:03.530747+0000","last_peered":"2023-10-30T13:59:03.530747+0000","last_clean":"2023-10-30T13:59:03.530747+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:59:03.530747+0000","last_undegraded":"2023-10-30T13:59:03.530747+0000","last_fullsized":"2023-10-30T13:59:03.530747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098613959999999997,"stat_sum":{"num_bytes":30553715,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":124,"num_read_kb":8207,"num_write":638,"num_write_kb":39180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"56'818","reported_seq":3032,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:33.959432+0000","last_change":"2023-10-30T13:59:33.959488+0000","last_active":"2023-10-30T13:59:33.959432+0000","last_peered":"2023-10-30T13:59:33.959432+0000","last_clean":"2023-10-30T13:59:33.959432+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:59:33.959432+0000","last_undegraded":"2023-10-30T13:59:33.959432+0000","last_fullsized":"2023-10-30T13:59:33.959432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'818","last_scrub_stamp":"2023-10-30T13:59:33.959361+0000","last_deep_scrub":"56'818","last_deep_scrub_stamp":"2023-10-30T13:59:33.959361+0000","last_clean_scrub_stamp":"2023-10-30T13:59:33.959361+0000","objects_scrubbed":161,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:47.001349+0000","scrub_duration":0.182073817,"objects_trimmed":0,"snaptrim_duration":0.001955118,"stat_sum":{"num_bytes":9994664,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":104,"num_read_kb":11,"num_write":617,"num_write_kb":23604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"56'815","reported_seq":3032,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:22.708425+0000","last_change":"2023-10-30T13:59:22.708475+0000","last_active":"2023-10-30T13:59:22.708425+0000","last_peered":"2023-10-30T13:59:22.708425+0000","last_clean":"2023-10-30T13:59:22.708425+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:59:22.708425+0000","last_undegraded":"2023-10-30T13:59:22.708425+0000","last_fullsized":"2023-10-30T13:59:22.708425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'815","last_scrub_stamp":"2023-10-30T13:59:22.708353+0000","last_deep_scrub":"56'815","last_deep_scrub_stamp":"2023-10-30T13:59:22.708353+0000","last_clean_scrub_stamp":"2023-10-30T13:59:22.708353+0000","objects_scrubbed":162,"log_size":815,"log_dups_size":0,"ondisk_log_size":815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:43:36.533585+0000","scrub_duration":0.169844142,"objects_trimmed":0,"snaptrim_duration":0.0021120940000000001,"stat_sum":{"num_bytes":23390266,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":110,"num_read_kb":20497,"num_write":638,"num_write_kb":47001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"56'828","reported_seq":3069,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.387579+0000","last_change":"2023-10-30T13:59:24.387627+0000","last_active":"2023-10-30T13:59:24.387579+0000","last_peered":"2023-10-30T13:59:24.387579+0000","last_clean":"2023-10-30T13:59:24.387579+0000","last_became_active":"2023-10-30T13:35:48.779909+0000","last_became_peered":"2023-10-30T13:35:48.779909+0000","last_unstale":"2023-10-30T13:59:24.387579+0000","last_undegraded":"2023-10-30T13:59:24.387579+0000","last_fullsized":"2023-10-30T13:59:24.387579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'828","last_scrub_stamp":"2023-10-30T13:59:24.387502+0000","last_deep_scrub":"56'828","last_deep_scrub_stamp":"2023-10-30T13:59:24.387502+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.387502+0000","objects_scrubbed":159,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:33.212806+0000","scrub_duration":0.154922691,"objects_trimmed":0,"snaptrim_duration":0.001317775,"stat_sum":{"num_bytes":10165225,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":133,"num_read_kb":8205,"num_write":656,"num_write_kb":40955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"56'701","reported_seq":2905,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.510951+0000","last_change":"2023-10-30T13:45:28.597729+0000","last_active":"2023-10-30T13:59:03.510951+0000","last_peered":"2023-10-30T13:59:03.510951+0000","last_clean":"2023-10-30T13:59:03.510951+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:59:03.510951+0000","last_undegraded":"2023-10-30T13:59:03.510951+0000","last_fullsized":"2023-10-30T13:59:03.510951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007614844,"stat_sum":{"num_bytes":19907850,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":110,"num_read_kb":8208,"num_write":664,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"56'666","reported_seq":2864,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.525952+0000","last_change":"2023-10-30T13:45:27.941070+0000","last_active":"2023-10-30T13:59:03.525952+0000","last_peered":"2023-10-30T13:59:03.525952+0000","last_clean":"2023-10-30T13:59:03.525952+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:59:03.525952+0000","last_undegraded":"2023-10-30T13:59:03.525952+0000","last_fullsized":"2023-10-30T13:59:03.525952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030849649999999998,"stat_sum":{"num_bytes":19227960,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":104,"num_read_kb":7536,"num_write":633,"num_write_kb":20026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"56'856","reported_seq":3055,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:34.802613+0000","last_change":"2023-10-30T13:59:34.802759+0000","last_active":"2023-10-30T13:59:34.802613+0000","last_peered":"2023-10-30T13:59:34.802613+0000","last_clean":"2023-10-30T13:59:34.802613+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:59:34.802613+0000","last_undegraded":"2023-10-30T13:59:34.802613+0000","last_fullsized":"2023-10-30T13:59:34.802613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'856","last_scrub_stamp":"2023-10-30T13:59:34.802404+0000","last_deep_scrub":"56'856","last_deep_scrub_stamp":"2023-10-30T13:59:34.802404+0000","last_clean_scrub_stamp":"2023-10-30T13:59:34.802404+0000","objects_scrubbed":161,"log_size":856,"log_dups_size":0,"ondisk_log_size":856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:57.808507+0000","scrub_duration":0.29249388500000001,"objects_trimmed":0,"snaptrim_duration":0.0082986029999999999,"stat_sum":{"num_bytes":10009773,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":91,"num_read_kb":11,"num_write":659,"num_write_kb":29217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"56'828","reported_seq":3068,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:41.385142+0000","last_change":"2023-10-30T13:59:41.385188+0000","last_active":"2023-10-30T13:59:41.385142+0000","last_peered":"2023-10-30T13:59:41.385142+0000","last_clean":"2023-10-30T13:59:41.385142+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:59:41.385142+0000","last_undegraded":"2023-10-30T13:59:41.385142+0000","last_fullsized":"2023-10-30T13:59:41.385142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'828","last_scrub_stamp":"2023-10-30T13:59:41.385065+0000","last_deep_scrub":"56'828","last_deep_scrub_stamp":"2023-10-30T13:59:41.385065+0000","last_clean_scrub_stamp":"2023-10-30T13:59:41.385065+0000","objects_scrubbed":158,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:32.113907+0000","scrub_duration":0.28254902599999998,"objects_trimmed":0,"snaptrim_duration":0.0021728070000000001,"stat_sum":{"num_bytes":10221735,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":132,"num_read_kb":1147,"num_write":641,"num_write_kb":24778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"56'666","reported_seq":2884,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.089833+0000","last_change":"2023-10-30T13:45:28.601219+0000","last_active":"2023-10-30T13:59:11.089833+0000","last_peered":"2023-10-30T13:59:11.089833+0000","last_clean":"2023-10-30T13:59:11.089833+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:59:11.089833+0000","last_undegraded":"2023-10-30T13:59:11.089833+0000","last_fullsized":"2023-10-30T13:59:11.089833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010045174,"stat_sum":{"num_bytes":10718551,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":124,"num_read_kb":4115,"num_write":615,"num_write_kb":24187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"56'778","reported_seq":3000,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:43.984031+0000","last_change":"2023-10-30T13:59:43.984081+0000","last_active":"2023-10-30T13:59:43.984031+0000","last_peered":"2023-10-30T13:59:43.984031+0000","last_clean":"2023-10-30T13:59:43.984031+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:59:43.984031+0000","last_undegraded":"2023-10-30T13:59:43.984031+0000","last_fullsized":"2023-10-30T13:59:43.984031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'778","last_scrub_stamp":"2023-10-30T13:59:43.983948+0000","last_deep_scrub":"56'778","last_deep_scrub_stamp":"2023-10-30T13:59:43.983948+0000","last_clean_scrub_stamp":"2023-10-30T13:59:43.983948+0000","objects_scrubbed":161,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:21.972997+0000","scrub_duration":0.23974774600000001,"objects_trimmed":0,"snaptrim_duration":0.0047277760000000004,"stat_sum":{"num_bytes":20230336,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":113,"num_read_kb":28,"num_write":585,"num_write_kb":25967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"56'781","reported_seq":2987,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:46.984535+0000","last_change":"2023-10-30T13:59:46.984590+0000","last_active":"2023-10-30T13:59:46.984535+0000","last_peered":"2023-10-30T13:59:46.984535+0000","last_clean":"2023-10-30T13:59:46.984535+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:59:46.984535+0000","last_undegraded":"2023-10-30T13:59:46.984535+0000","last_fullsized":"2023-10-30T13:59:46.984535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'781","last_scrub_stamp":"2023-10-30T13:59:46.984450+0000","last_deep_scrub":"56'781","last_deep_scrub_stamp":"2023-10-30T13:59:46.984450+0000","last_clean_scrub_stamp":"2023-10-30T13:59:46.984450+0000","objects_scrubbed":169,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:12.021790+0000","scrub_duration":0.21481325800000001,"objects_trimmed":0,"snaptrim_duration":0.00429868,"stat_sum":{"num_bytes":2384023,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":97,"num_read_kb":17,"num_write":586,"num_write_kb":9055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"56'789","reported_seq":3015,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.827760+0000","last_change":"2023-10-30T13:59:19.827838+0000","last_active":"2023-10-30T13:59:19.827760+0000","last_peered":"2023-10-30T13:59:19.827760+0000","last_clean":"2023-10-30T13:59:19.827760+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:59:19.827760+0000","last_undegraded":"2023-10-30T13:59:19.827760+0000","last_fullsized":"2023-10-30T13:59:19.827760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'789","last_scrub_stamp":"2023-10-30T13:59:19.827650+0000","last_deep_scrub":"56'789","last_deep_scrub_stamp":"2023-10-30T13:59:19.827650+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.827650+0000","objects_scrubbed":175,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:07.295597+0000","scrub_duration":0.27225982599999998,"objects_trimmed":0,"snaptrim_duration":0.0066533010000000004,"stat_sum":{"num_bytes":18341132,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":8208,"num_write":596,"num_write_kb":32418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"56'648","reported_seq":2856,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.913193+0000","last_change":"2023-10-30T13:45:28.780475+0000","last_active":"2023-10-30T13:59:08.913193+0000","last_peered":"2023-10-30T13:59:08.913193+0000","last_clean":"2023-10-30T13:59:08.913193+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:59:08.913193+0000","last_undegraded":"2023-10-30T13:59:08.913193+0000","last_fullsized":"2023-10-30T13:59:08.913193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002338237,"stat_sum":{"num_bytes":14177749,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":114,"num_read_kb":12311,"num_write":601,"num_write_kb":37081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"56'871","reported_seq":3099,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.750623+0000","last_change":"2023-10-30T13:59:24.750679+0000","last_active":"2023-10-30T13:59:24.750623+0000","last_peered":"2023-10-30T13:59:24.750623+0000","last_clean":"2023-10-30T13:59:24.750623+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:59:24.750623+0000","last_undegraded":"2023-10-30T13:59:24.750623+0000","last_fullsized":"2023-10-30T13:59:24.750623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'871","last_scrub_stamp":"2023-10-30T13:59:24.750548+0000","last_deep_scrub":"56'871","last_deep_scrub_stamp":"2023-10-30T13:59:24.750548+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.750548+0000","objects_scrubbed":167,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:44.353295+0000","scrub_duration":0.19578169600000001,"objects_trimmed":0,"snaptrim_duration":0.0029248669999999998,"stat_sum":{"num_bytes":22804734,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":119,"num_read_kb":24041,"num_write":691,"num_write_kb":57342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"56'875","reported_seq":3096,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:39.045087+0000","last_change":"2023-10-30T13:59:39.045134+0000","last_active":"2023-10-30T13:59:39.045087+0000","last_peered":"2023-10-30T13:59:39.045087+0000","last_clean":"2023-10-30T13:59:39.045087+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:59:39.045087+0000","last_undegraded":"2023-10-30T13:59:39.045087+0000","last_fullsized":"2023-10-30T13:59:39.045087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'875","last_scrub_stamp":"2023-10-30T13:59:39.045006+0000","last_deep_scrub":"56'875","last_deep_scrub_stamp":"2023-10-30T13:59:39.045006+0000","last_clean_scrub_stamp":"2023-10-30T13:59:39.045006+0000","objects_scrubbed":159,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:49:37.890024+0000","scrub_duration":0.291907531,"objects_trimmed":0,"snaptrim_duration":0.0021050410000000002,"stat_sum":{"num_bytes":18271578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":112,"num_read_kb":7334,"num_write":676,"num_write_kb":39770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"56'884","reported_seq":3117,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:31.971189+0000","last_change":"2023-10-30T13:59:31.971276+0000","last_active":"2023-10-30T13:59:31.971189+0000","last_peered":"2023-10-30T13:59:31.971189+0000","last_clean":"2023-10-30T13:59:31.971189+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:59:31.971189+0000","last_undegraded":"2023-10-30T13:59:31.971189+0000","last_fullsized":"2023-10-30T13:59:31.971189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'884","last_scrub_stamp":"2023-10-30T13:59:31.971052+0000","last_deep_scrub":"56'884","last_deep_scrub_stamp":"2023-10-30T13:59:31.971052+0000","last_clean_scrub_stamp":"2023-10-30T13:59:31.971052+0000","objects_scrubbed":181,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:49:06.332092+0000","scrub_duration":0.37008379400000002,"objects_trimmed":0,"snaptrim_duration":0.0052579080000000004,"stat_sum":{"num_bytes":35259912,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":123,"num_read_kb":12306,"num_write":693,"num_write_kb":43349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"56'832","reported_seq":3043,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:29.855984+0000","last_change":"2023-10-30T13:59:29.856089+0000","last_active":"2023-10-30T13:59:29.855984+0000","last_peered":"2023-10-30T13:59:29.855984+0000","last_clean":"2023-10-30T13:59:29.855984+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:59:29.855984+0000","last_undegraded":"2023-10-30T13:59:29.855984+0000","last_fullsized":"2023-10-30T13:59:29.855984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'832","last_scrub_stamp":"2023-10-30T13:59:29.855862+0000","last_deep_scrub":"56'832","last_deep_scrub_stamp":"2023-10-30T13:59:29.855862+0000","last_clean_scrub_stamp":"2023-10-30T13:59:29.855862+0000","objects_scrubbed":154,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:19.015619+0000","scrub_duration":0.27485262900000001,"objects_trimmed":0,"snaptrim_duration":0.0052923609999999998,"stat_sum":{"num_bytes":14461920,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":103,"num_read_kb":4113,"num_write":659,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"56'763","reported_seq":2989,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.418430+0000","last_change":"2023-10-30T13:59:17.418479+0000","last_active":"2023-10-30T13:59:17.418430+0000","last_peered":"2023-10-30T13:59:17.418430+0000","last_clean":"2023-10-30T13:59:17.418430+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:59:17.418430+0000","last_undegraded":"2023-10-30T13:59:17.418430+0000","last_fullsized":"2023-10-30T13:59:17.418430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'763","last_scrub_stamp":"2023-10-30T13:59:17.418359+0000","last_deep_scrub":"56'763","last_deep_scrub_stamp":"2023-10-30T13:59:17.418359+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.418359+0000","objects_scrubbed":142,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:00.183579+0000","scrub_duration":0.20971843000000001,"objects_trimmed":0,"snaptrim_duration":0.001369274,"stat_sum":{"num_bytes":15414431,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":118,"num_read_kb":4123,"num_write":623,"num_write_kb":31369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"56'907","reported_seq":3150,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:44.853684+0000","last_change":"2023-10-30T13:59:44.853755+0000","last_active":"2023-10-30T13:59:44.853684+0000","last_peered":"2023-10-30T13:59:44.853684+0000","last_clean":"2023-10-30T13:59:44.853684+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:59:44.853684+0000","last_undegraded":"2023-10-30T13:59:44.853684+0000","last_fullsized":"2023-10-30T13:59:44.853684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'907","last_scrub_stamp":"2023-10-30T13:59:44.853571+0000","last_deep_scrub":"56'907","last_deep_scrub_stamp":"2023-10-30T13:59:44.853571+0000","last_clean_scrub_stamp":"2023-10-30T13:59:44.853571+0000","objects_scrubbed":179,"log_size":907,"log_dups_size":0,"ondisk_log_size":907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:17.124766+0000","scrub_duration":0.34510061199999997,"objects_trimmed":0,"snaptrim_duration":0.0092525119999999992,"stat_sum":{"num_bytes":8879597,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":134,"num_read_kb":8209,"num_write":693,"num_write_kb":31072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"56'821","reported_seq":3029,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:38.050131+0000","last_change":"2023-10-30T13:59:38.050201+0000","last_active":"2023-10-30T13:59:38.050131+0000","last_peered":"2023-10-30T13:59:38.050131+0000","last_clean":"2023-10-30T13:59:38.050131+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:59:38.050131+0000","last_undegraded":"2023-10-30T13:59:38.050131+0000","last_fullsized":"2023-10-30T13:59:38.050131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'821","last_scrub_stamp":"2023-10-30T13:59:38.049998+0000","last_deep_scrub":"56'821","last_deep_scrub_stamp":"2023-10-30T13:59:38.049998+0000","last_clean_scrub_stamp":"2023-10-30T13:59:38.049998+0000","objects_scrubbed":154,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:50:58.830199+0000","scrub_duration":0.30212535800000001,"objects_trimmed":0,"snaptrim_duration":0.00502631,"stat_sum":{"num_bytes":6211385,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":100,"num_read_kb":8209,"num_write":620,"num_write_kb":19966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"56'818","reported_seq":3038,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:29.620176+0000","last_change":"2023-10-30T13:59:29.620218+0000","last_active":"2023-10-30T13:59:29.620176+0000","last_peered":"2023-10-30T13:59:29.620176+0000","last_clean":"2023-10-30T13:59:29.620176+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:59:29.620176+0000","last_undegraded":"2023-10-30T13:59:29.620176+0000","last_fullsized":"2023-10-30T13:59:29.620176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'818","last_scrub_stamp":"2023-10-30T13:59:29.620101+0000","last_deep_scrub":"56'818","last_deep_scrub_stamp":"2023-10-30T13:59:29.620101+0000","last_clean_scrub_stamp":"2023-10-30T13:59:29.620101+0000","objects_scrubbed":152,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:31.441486+0000","scrub_duration":0.23420875999999999,"objects_trimmed":0,"snaptrim_duration":0.0026024469999999999,"stat_sum":{"num_bytes":18097476,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":111,"num_read_kb":21,"num_write":651,"num_write_kb":32402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"56'705","reported_seq":2905,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.526011+0000","last_change":"2023-10-30T13:45:27.947213+0000","last_active":"2023-10-30T13:59:03.526011+0000","last_peered":"2023-10-30T13:59:03.526011+0000","last_clean":"2023-10-30T13:59:03.526011+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:59:03.526011+0000","last_undegraded":"2023-10-30T13:59:03.526011+0000","last_fullsized":"2023-10-30T13:59:03.526011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054353270000000002,"stat_sum":{"num_bytes":10272429,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":105,"num_read_kb":4117,"num_write":665,"num_write_kb":16082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"56'847","reported_seq":3068,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:43.360123+0000","last_change":"2023-10-30T13:59:43.360180+0000","last_active":"2023-10-30T13:59:43.360123+0000","last_peered":"2023-10-30T13:59:43.360123+0000","last_clean":"2023-10-30T13:59:43.360123+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:59:43.360123+0000","last_undegraded":"2023-10-30T13:59:43.360123+0000","last_fullsized":"2023-10-30T13:59:43.360123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'847","last_scrub_stamp":"2023-10-30T13:59:43.360046+0000","last_deep_scrub":"56'847","last_deep_scrub_stamp":"2023-10-30T13:59:43.360046+0000","last_clean_scrub_stamp":"2023-10-30T13:59:43.360046+0000","objects_scrubbed":146,"log_size":847,"log_dups_size":0,"ondisk_log_size":847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:44.037800+0000","scrub_duration":0.25013997900000001,"objects_trimmed":0,"snaptrim_duration":0.0025617359999999998,"stat_sum":{"num_bytes":14333972,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":114,"num_read_kb":8213,"num_write":665,"num_write_kb":36470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"56'850","reported_seq":3086,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.738390+0000","last_change":"2023-10-30T13:59:24.738493+0000","last_active":"2023-10-30T13:59:24.738390+0000","last_peered":"2023-10-30T13:59:24.738390+0000","last_clean":"2023-10-30T13:59:24.738390+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:59:24.738390+0000","last_undegraded":"2023-10-30T13:59:24.738390+0000","last_fullsized":"2023-10-30T13:59:24.738390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'850","last_scrub_stamp":"2023-10-30T13:59:24.738213+0000","last_deep_scrub":"56'850","last_deep_scrub_stamp":"2023-10-30T13:59:24.738213+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.738213+0000","objects_scrubbed":157,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:41:27.710826+0000","scrub_duration":0.30834826700000001,"objects_trimmed":0,"snaptrim_duration":0.004034279,"stat_sum":{"num_bytes":1595822,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":126,"num_read_kb":16,"num_write":676,"num_write_kb":12551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"56'819","reported_seq":3045,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:38.313186+0000","last_change":"2023-10-30T13:59:38.313242+0000","last_active":"2023-10-30T13:59:38.313186+0000","last_peered":"2023-10-30T13:59:38.313186+0000","last_clean":"2023-10-30T13:59:38.313186+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:59:38.313186+0000","last_undegraded":"2023-10-30T13:59:38.313186+0000","last_fullsized":"2023-10-30T13:59:38.313186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'819","last_scrub_stamp":"2023-10-30T13:59:38.313111+0000","last_deep_scrub":"56'819","last_deep_scrub_stamp":"2023-10-30T13:59:38.313111+0000","last_clean_scrub_stamp":"2023-10-30T13:59:38.313111+0000","objects_scrubbed":145,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:35.451625+0000","scrub_duration":0.24463141199999999,"objects_trimmed":0,"snaptrim_duration":0.0030593479999999999,"stat_sum":{"num_bytes":7858557,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":117,"num_read_kb":4108,"num_write":634,"num_write_kb":18163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"56'858","reported_seq":3078,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:41.006397+0000","last_change":"2023-10-30T13:59:41.006480+0000","last_active":"2023-10-30T13:59:41.006397+0000","last_peered":"2023-10-30T13:59:41.006397+0000","last_clean":"2023-10-30T13:59:41.006397+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:59:41.006397+0000","last_undegraded":"2023-10-30T13:59:41.006397+0000","last_fullsized":"2023-10-30T13:59:41.006397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'858","last_scrub_stamp":"2023-10-30T13:59:41.006253+0000","last_deep_scrub":"56'858","last_deep_scrub_stamp":"2023-10-30T13:59:41.006253+0000","last_clean_scrub_stamp":"2023-10-30T13:59:41.006253+0000","objects_scrubbed":136,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:17.543137+0000","scrub_duration":0.28529553499999999,"objects_trimmed":0,"snaptrim_duration":0.0049754609999999996,"stat_sum":{"num_bytes":9799460,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":111,"num_read_kb":24,"num_write":679,"num_write_kb":16350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"56'822","reported_seq":3042,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:39.919298+0000","last_change":"2023-10-30T13:59:39.919348+0000","last_active":"2023-10-30T13:59:39.919298+0000","last_peered":"2023-10-30T13:59:39.919298+0000","last_clean":"2023-10-30T13:59:39.919298+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:59:39.919298+0000","last_undegraded":"2023-10-30T13:59:39.919298+0000","last_fullsized":"2023-10-30T13:59:39.919298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'822","last_scrub_stamp":"2023-10-30T13:59:39.919226+0000","last_deep_scrub":"56'822","last_deep_scrub_stamp":"2023-10-30T13:59:39.919226+0000","last_clean_scrub_stamp":"2023-10-30T13:59:39.919226+0000","objects_scrubbed":154,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:01.818651+0000","scrub_duration":0.18919842000000001,"objects_trimmed":0,"snaptrim_duration":0.0028468439999999999,"stat_sum":{"num_bytes":33375622,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":110,"num_read_kb":16400,"num_write":628,"num_write_kb":35724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"56'791","reported_seq":3012,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.370003+0000","last_change":"2023-10-30T13:59:25.370057+0000","last_active":"2023-10-30T13:59:25.370003+0000","last_peered":"2023-10-30T13:59:25.370003+0000","last_clean":"2023-10-30T13:59:25.370003+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:59:25.370003+0000","last_undegraded":"2023-10-30T13:59:25.370003+0000","last_fullsized":"2023-10-30T13:59:25.370003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'791","last_scrub_stamp":"2023-10-30T13:59:25.369932+0000","last_deep_scrub":"56'791","last_deep_scrub_stamp":"2023-10-30T13:59:25.369932+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.369932+0000","objects_scrubbed":174,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:57.726447+0000","scrub_duration":0.25855898900000002,"objects_trimmed":0,"snaptrim_duration":0.002839121,"stat_sum":{"num_bytes":9811427,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":112,"num_read_kb":10701,"num_write":625,"num_write_kb":15312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"56'859","reported_seq":3079,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:32.572493+0000","last_change":"2023-10-30T13:59:32.572574+0000","last_active":"2023-10-30T13:59:32.572493+0000","last_peered":"2023-10-30T13:59:32.572493+0000","last_clean":"2023-10-30T13:59:32.572493+0000","last_became_active":"2023-10-30T13:35:48.847825+0000","last_became_peered":"2023-10-30T13:35:48.847825+0000","last_unstale":"2023-10-30T13:59:32.572493+0000","last_undegraded":"2023-10-30T13:59:32.572493+0000","last_fullsized":"2023-10-30T13:59:32.572493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'859","last_scrub_stamp":"2023-10-30T13:59:32.572396+0000","last_deep_scrub":"56'859","last_deep_scrub_stamp":"2023-10-30T13:59:32.572396+0000","last_clean_scrub_stamp":"2023-10-30T13:59:32.572396+0000","objects_scrubbed":183,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:49:33.324446+0000","scrub_duration":0.17384617199999999,"objects_trimmed":0,"snaptrim_duration":0.0028647770000000002,"stat_sum":{"num_bytes":7231756,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":102,"num_read_kb":4848,"num_write":653,"num_write_kb":26441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"56'854","reported_seq":3084,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:45.005956+0000","last_change":"2023-10-30T13:59:45.006081+0000","last_active":"2023-10-30T13:59:45.005956+0000","last_peered":"2023-10-30T13:59:45.005956+0000","last_clean":"2023-10-30T13:59:45.005956+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:59:45.005956+0000","last_undegraded":"2023-10-30T13:59:45.005956+0000","last_fullsized":"2023-10-30T13:59:45.005956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'854","last_scrub_stamp":"2023-10-30T13:59:45.005743+0000","last_deep_scrub":"56'854","last_deep_scrub_stamp":"2023-10-30T13:59:45.005743+0000","last_clean_scrub_stamp":"2023-10-30T13:59:45.005743+0000","objects_scrubbed":162,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:19.548547+0000","scrub_duration":0.35207807699999999,"objects_trimmed":0,"snaptrim_duration":0.0060200970000000003,"stat_sum":{"num_bytes":22333767,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":120,"num_read_kb":4116,"num_write":656,"num_write_kb":43887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"56'925","reported_seq":3175,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.513765+0000","last_change":"2023-10-30T13:59:16.513815+0000","last_active":"2023-10-30T13:59:16.513765+0000","last_peered":"2023-10-30T13:59:16.513765+0000","last_clean":"2023-10-30T13:59:16.513765+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:59:16.513765+0000","last_undegraded":"2023-10-30T13:59:16.513765+0000","last_fullsized":"2023-10-30T13:59:16.513765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'925","last_scrub_stamp":"2023-10-30T13:59:16.513655+0000","last_deep_scrub":"56'925","last_deep_scrub_stamp":"2023-10-30T13:59:16.513655+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.513655+0000","objects_scrubbed":186,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:03.990730+0000","scrub_duration":0.27611799599999998,"objects_trimmed":0,"snaptrim_duration":0.001263171,"stat_sum":{"num_bytes":10546872,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":139,"num_read_kb":4118,"num_write":747,"num_write_kb":34715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"56'835","reported_seq":3055,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:32.085669+0000","last_change":"2023-10-30T13:59:32.085787+0000","last_active":"2023-10-30T13:59:32.085669+0000","last_peered":"2023-10-30T13:59:32.085669+0000","last_clean":"2023-10-30T13:59:32.085669+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:59:32.085669+0000","last_undegraded":"2023-10-30T13:59:32.085669+0000","last_fullsized":"2023-10-30T13:59:32.085669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'835","last_scrub_stamp":"2023-10-30T13:59:32.085458+0000","last_deep_scrub":"56'835","last_deep_scrub_stamp":"2023-10-30T13:59:32.085458+0000","last_clean_scrub_stamp":"2023-10-30T13:59:32.085458+0000","objects_scrubbed":153,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:28.945683+0000","scrub_duration":0.25482183200000003,"objects_trimmed":0,"snaptrim_duration":0.0036881090000000002,"stat_sum":{"num_bytes":14815628,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":110,"num_read_kb":20,"num_write":677,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"56'797","reported_seq":3013,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.874226+0000","last_change":"2023-10-30T13:59:15.874311+0000","last_active":"2023-10-30T13:59:15.874226+0000","last_peered":"2023-10-30T13:59:15.874226+0000","last_clean":"2023-10-30T13:59:15.874226+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:59:15.874226+0000","last_undegraded":"2023-10-30T13:59:15.874226+0000","last_fullsized":"2023-10-30T13:59:15.874226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'797","last_scrub_stamp":"2023-10-30T13:59:15.874087+0000","last_deep_scrub":"56'797","last_deep_scrub_stamp":"2023-10-30T13:59:15.874087+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.874087+0000","objects_scrubbed":163,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:30.985181+0000","scrub_duration":0.29104613299999998,"objects_trimmed":0,"snaptrim_duration":0.0065387930000000002,"stat_sum":{"num_bytes":14536361,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":108,"num_read_kb":8117,"num_write":641,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"56'698","reported_seq":776,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.085592+0000","last_change":"2023-10-30T13:59:12.085635+0000","last_active":"2023-10-30T13:59:12.085592+0000","last_peered":"2023-10-30T13:59:12.085592+0000","last_clean":"2023-10-30T13:59:12.085592+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:59:12.085592+0000","last_undegraded":"2023-10-30T13:59:12.085592+0000","last_fullsized":"2023-10-30T13:59:12.085592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'698","last_scrub_stamp":"2023-10-30T13:59:12.085518+0000","last_deep_scrub":"56'698","last_deep_scrub_stamp":"2023-10-30T13:59:12.085518+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.085518+0000","objects_scrubbed":4,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:55:05.427263+0000","scrub_duration":0.022533350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45057,"num_write":652,"num_write_kb":16749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":500,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"56'816","reported_seq":3051,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.871830+0000","last_change":"2023-10-30T13:59:20.871886+0000","last_active":"2023-10-30T13:59:20.871830+0000","last_peered":"2023-10-30T13:59:20.871830+0000","last_clean":"2023-10-30T13:59:20.871830+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:59:20.871830+0000","last_undegraded":"2023-10-30T13:59:20.871830+0000","last_fullsized":"2023-10-30T13:59:20.871830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'816","last_scrub_stamp":"2023-10-30T13:59:20.871754+0000","last_deep_scrub":"56'816","last_deep_scrub_stamp":"2023-10-30T13:59:20.871754+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.871754+0000","objects_scrubbed":166,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:54.494538+0000","scrub_duration":0.28592286,"objects_trimmed":0,"snaptrim_duration":0.0031306289999999998,"stat_sum":{"num_bytes":22886210,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":127,"num_read_kb":6230,"num_write":644,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"56'754","reported_seq":894,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:26.898884+0000","last_change":"2023-10-30T13:59:26.899016+0000","last_active":"2023-10-30T13:59:26.898884+0000","last_peered":"2023-10-30T13:59:26.898884+0000","last_clean":"2023-10-30T13:59:26.898884+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:59:26.898884+0000","last_undegraded":"2023-10-30T13:59:26.898884+0000","last_fullsized":"2023-10-30T13:59:26.898884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'754","last_scrub_stamp":"2023-10-30T13:59:26.898658+0000","last_deep_scrub":"56'754","last_deep_scrub_stamp":"2023-10-30T13:59:26.898658+0000","last_clean_scrub_stamp":"2023-10-30T13:59:26.898658+0000","objects_scrubbed":11,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:38.802448+0000","scrub_duration":0.095300314999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21047876,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":114393,"num_write":693,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21258,"num_omap_keys":40,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"56'146","reported_seq":264,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.093208+0000","last_change":"2023-10-30T13:59:09.093208+0000","last_active":"2023-10-30T13:59:09.093208+0000","last_peered":"2023-10-30T13:59:09.093208+0000","last_clean":"2023-10-30T13:59:09.093208+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:59:09.093208+0000","last_undegraded":"2023-10-30T13:59:09.093208+0000","last_fullsized":"2023-10-30T13:59:09.093208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'146","last_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_deep_scrub":"56'146","last_deep_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.093046+0000","objects_scrubbed":7,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:36.674466+0000","scrub_duration":0.022130225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297526,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":20619,"num_write":102,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33717,"num_omap_keys":64,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"56'987","reported_seq":3240,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:39.282813+0000","last_change":"2023-10-30T13:59:39.282871+0000","last_active":"2023-10-30T13:59:39.282813+0000","last_peered":"2023-10-30T13:59:39.282813+0000","last_clean":"2023-10-30T13:59:39.282813+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:59:39.282813+0000","last_undegraded":"2023-10-30T13:59:39.282813+0000","last_fullsized":"2023-10-30T13:59:39.282813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'987","last_scrub_stamp":"2023-10-30T13:59:39.282741+0000","last_deep_scrub":"56'987","last_deep_scrub_stamp":"2023-10-30T13:59:39.282741+0000","last_clean_scrub_stamp":"2023-10-30T13:59:39.282741+0000","objects_scrubbed":195,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:31.434310+0000","scrub_duration":0.182689608,"objects_trimmed":0,"snaptrim_duration":0.0027758349999999999,"stat_sum":{"num_bytes":1695799,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":144,"num_read_kb":4120,"num_write":754,"num_write_kb":40525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"56'600","reported_seq":692,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.512895+0000","last_change":"2023-10-30T13:59:10.512895+0000","last_active":"2023-10-30T13:59:10.512895+0000","last_peered":"2023-10-30T13:59:10.512895+0000","last_clean":"2023-10-30T13:59:10.512895+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:59:10.512895+0000","last_undegraded":"2023-10-30T13:59:10.512895+0000","last_fullsized":"2023-10-30T13:59:10.512895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'600","last_scrub_stamp":"2023-10-30T13:59:10.512557+0000","last_deep_scrub":"56'600","last_deep_scrub_stamp":"2023-10-30T13:59:10.512557+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.512557+0000","objects_scrubbed":4,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:37:46.271383+0000","scrub_duration":0.060732762000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":96432,"num_write":542,"num_write_kb":20769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7630,"num_omap_keys":14,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"56'774","reported_seq":2997,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.878582+0000","last_change":"2023-10-30T13:59:17.878582+0000","last_active":"2023-10-30T13:59:17.878582+0000","last_peered":"2023-10-30T13:59:17.878582+0000","last_clean":"2023-10-30T13:59:17.878582+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:59:17.878582+0000","last_undegraded":"2023-10-30T13:59:17.878582+0000","last_fullsized":"2023-10-30T13:59:17.878582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'774","last_scrub_stamp":"2023-10-30T13:59:17.878418+0000","last_deep_scrub":"56'774","last_deep_scrub_stamp":"2023-10-30T13:59:17.878418+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.878418+0000","objects_scrubbed":155,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:30.998088+0000","scrub_duration":0.157018885,"objects_trimmed":0,"snaptrim_duration":0.0017556029999999999,"stat_sum":{"num_bytes":7038572,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":115,"num_read_kb":8211,"num_write":611,"num_write_kb":33441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"56'131","reported_seq":200,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.802120+0000","last_change":"2023-10-30T13:59:20.802254+0000","last_active":"2023-10-30T13:59:20.802120+0000","last_peered":"2023-10-30T13:59:20.802120+0000","last_clean":"2023-10-30T13:59:20.802120+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:59:20.802120+0000","last_undegraded":"2023-10-30T13:59:20.802120+0000","last_fullsized":"2023-10-30T13:59:20.802120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'131","last_scrub_stamp":"2023-10-30T13:59:20.801920+0000","last_deep_scrub":"56'131","last_deep_scrub_stamp":"2023-10-30T13:59:20.801920+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.801920+0000","objects_scrubbed":7,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:04.676924+0000","scrub_duration":0.017330934999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":91,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20610,"num_omap_keys":38,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"56'827","reported_seq":3033,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:27.694964+0000","last_change":"2023-10-30T13:59:27.695039+0000","last_active":"2023-10-30T13:59:27.694964+0000","last_peered":"2023-10-30T13:59:27.694964+0000","last_clean":"2023-10-30T13:59:27.694964+0000","last_became_active":"2023-10-30T13:35:49.007835+0000","last_became_peered":"2023-10-30T13:35:49.007835+0000","last_unstale":"2023-10-30T13:59:27.694964+0000","last_undegraded":"2023-10-30T13:59:27.694964+0000","last_fullsized":"2023-10-30T13:59:27.694964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'827","last_scrub_stamp":"2023-10-30T13:59:27.694847+0000","last_deep_scrub":"56'827","last_deep_scrub_stamp":"2023-10-30T13:59:27.694847+0000","last_clean_scrub_stamp":"2023-10-30T13:59:27.694847+0000","objects_scrubbed":154,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:47.134642+0000","scrub_duration":0.465225628,"objects_trimmed":0,"snaptrim_duration":0.0011632000000000001,"stat_sum":{"num_bytes":15868329,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":101,"num_read_kb":5707,"num_write":652,"num_write_kb":27262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"56'390","reported_seq":473,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.497506+0000","last_change":"2023-10-30T13:59:13.497506+0000","last_active":"2023-10-30T13:59:13.497506+0000","last_peered":"2023-10-30T13:59:13.497506+0000","last_clean":"2023-10-30T13:59:13.497506+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:59:13.497506+0000","last_undegraded":"2023-10-30T13:59:13.497506+0000","last_fullsized":"2023-10-30T13:59:13.497506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'390","last_scrub_stamp":"2023-10-30T13:59:13.497014+0000","last_deep_scrub":"56'390","last_deep_scrub_stamp":"2023-10-30T13:59:13.497014+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.497014+0000","objects_scrubbed":6,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:50.064647+0000","scrub_duration":0.074294748999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":61443,"num_write":345,"num_write_kb":20724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"56'809","reported_seq":3047,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.828504+0000","last_change":"2023-10-30T13:59:19.828578+0000","last_active":"2023-10-30T13:59:19.828504+0000","last_peered":"2023-10-30T13:59:19.828504+0000","last_clean":"2023-10-30T13:59:19.828504+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:59:19.828504+0000","last_undegraded":"2023-10-30T13:59:19.828504+0000","last_fullsized":"2023-10-30T13:59:19.828504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'809","last_scrub_stamp":"2023-10-30T13:59:19.828412+0000","last_deep_scrub":"56'809","last_deep_scrub_stamp":"2023-10-30T13:59:19.828412+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.828412+0000","objects_scrubbed":177,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:57.553096+0000","scrub_duration":0.224065866,"objects_trimmed":0,"snaptrim_duration":0.0028607060000000002,"stat_sum":{"num_bytes":18966683,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":128,"num_read_kb":11963,"num_write":623,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"56'719","reported_seq":814,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.839263+0000","last_change":"2023-10-30T13:59:10.839263+0000","last_active":"2023-10-30T13:59:10.839263+0000","last_peered":"2023-10-30T13:59:10.839263+0000","last_clean":"2023-10-30T13:59:10.839263+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:59:10.839263+0000","last_undegraded":"2023-10-30T13:59:10.839263+0000","last_fullsized":"2023-10-30T13:59:10.839263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'719","last_scrub_stamp":"2023-10-30T13:59:10.839113+0000","last_deep_scrub":"56'719","last_deep_scrub_stamp":"2023-10-30T13:59:10.839113+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.839113+0000","objects_scrubbed":8,"log_size":719,"log_dups_size":0,"ondisk_log_size":719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:45.790385+0000","scrub_duration":0.068460164000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":49185,"num_write":677,"num_write_kb":16860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21363,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"56'788","reported_seq":2996,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:14.912002+0000","last_change":"2023-10-30T13:59:14.912126+0000","last_active":"2023-10-30T13:59:14.912002+0000","last_peered":"2023-10-30T13:59:14.912002+0000","last_clean":"2023-10-30T13:59:14.912002+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:59:14.912002+0000","last_undegraded":"2023-10-30T13:59:14.912002+0000","last_fullsized":"2023-10-30T13:59:14.912002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'788","last_scrub_stamp":"2023-10-30T13:59:14.911840+0000","last_deep_scrub":"56'788","last_deep_scrub_stamp":"2023-10-30T13:59:14.911840+0000","last_clean_scrub_stamp":"2023-10-30T13:59:14.911840+0000","objects_scrubbed":165,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:58.599422+0000","scrub_duration":0.31948987000000001,"objects_trimmed":0,"snaptrim_duration":0.0070741379999999998,"stat_sum":{"num_bytes":9996128,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":99,"num_read_kb":8209,"num_write":632,"num_write_kb":23925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"56'1294","reported_seq":1407,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:41.368145+0000","last_change":"2023-10-30T13:59:23.855056+0000","last_active":"2023-10-30T13:59:41.368145+0000","last_peered":"2023-10-30T13:59:41.368145+0000","last_clean":"2023-10-30T13:59:41.368145+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:59:41.368145+0000","last_undegraded":"2023-10-30T13:59:41.368145+0000","last_fullsized":"2023-10-30T13:59:41.368145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1274","last_scrub_stamp":"2023-10-30T13:59:23.854712+0000","last_deep_scrub":"56'1274","last_deep_scrub_stamp":"2023-10-30T13:59:23.854712+0000","last_clean_scrub_stamp":"2023-10-30T13:59:23.854712+0000","objects_scrubbed":10,"log_size":1294,"log_dups_size":0,"ondisk_log_size":1294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:21.240711+0000","scrub_duration":0.074187731000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16547844,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":86059,"num_write":1236,"num_write_kb":20987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29629,"num_omap_keys":53,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"56'671","reported_seq":2870,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.089956+0000","last_change":"2023-10-30T13:45:28.596691+0000","last_active":"2023-10-30T13:59:11.089956+0000","last_peered":"2023-10-30T13:59:11.089956+0000","last_clean":"2023-10-30T13:59:11.089956+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:59:11.089956+0000","last_undegraded":"2023-10-30T13:59:11.089956+0000","last_fullsized":"2023-10-30T13:59:11.089956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070810969999999997,"stat_sum":{"num_bytes":18214186,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":104,"num_read_kb":12299,"num_write":638,"num_write_kb":31808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"56'436","reported_seq":542,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.574205+0000","last_change":"2023-10-30T13:59:08.574261+0000","last_active":"2023-10-30T13:59:08.574205+0000","last_peered":"2023-10-30T13:59:08.574205+0000","last_clean":"2023-10-30T13:59:08.574205+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:59:08.574205+0000","last_undegraded":"2023-10-30T13:59:08.574205+0000","last_fullsized":"2023-10-30T13:59:08.574205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'436","last_scrub_stamp":"2023-10-30T13:59:08.574118+0000","last_deep_scrub":"56'436","last_deep_scrub_stamp":"2023-10-30T13:59:08.574118+0000","last_clean_scrub_stamp":"2023-10-30T13:59:08.574118+0000","objects_scrubbed":8,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:26.795616+0000","scrub_duration":0.024734227000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":77906,"num_write":384,"num_write_kb":12555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":59046,"num_omap_keys":109,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"56'843","reported_seq":3086,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:22.473412+0000","last_change":"2023-10-30T13:59:22.473466+0000","last_active":"2023-10-30T13:59:22.473412+0000","last_peered":"2023-10-30T13:59:22.473412+0000","last_clean":"2023-10-30T13:59:22.473412+0000","last_became_active":"2023-10-30T13:35:49.010460+0000","last_became_peered":"2023-10-30T13:35:49.010460+0000","last_unstale":"2023-10-30T13:59:22.473412+0000","last_undegraded":"2023-10-30T13:59:22.473412+0000","last_fullsized":"2023-10-30T13:59:22.473412+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'843","last_scrub_stamp":"2023-10-30T13:59:22.473336+0000","last_deep_scrub":"56'843","last_deep_scrub_stamp":"2023-10-30T13:59:22.473336+0000","last_clean_scrub_stamp":"2023-10-30T13:59:22.473336+0000","objects_scrubbed":164,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:50:11.354602+0000","scrub_duration":0.279252891,"objects_trimmed":0,"snaptrim_duration":0.001187553,"stat_sum":{"num_bytes":13816511,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":138,"num_read_kb":8210,"num_write":662,"num_write_kb":56040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"56'179","reported_seq":248,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.414864+0000","last_change":"2023-10-30T13:59:19.414972+0000","last_active":"2023-10-30T13:59:19.414864+0000","last_peered":"2023-10-30T13:59:19.414864+0000","last_clean":"2023-10-30T13:59:19.414864+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:59:19.414864+0000","last_undegraded":"2023-10-30T13:59:19.414864+0000","last_fullsized":"2023-10-30T13:59:19.414864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'179","last_scrub_stamp":"2023-10-30T13:59:19.414635+0000","last_deep_scrub":"56'179","last_deep_scrub_stamp":"2023-10-30T13:59:19.414635+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.414635+0000","objects_scrubbed":3,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.618899+0000","scrub_duration":0.0097075200000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":632,"num_write":147,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13012,"num_omap_keys":24,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"56'713","reported_seq":2933,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.520124+0000","last_change":"2023-10-30T13:45:28.599507+0000","last_active":"2023-10-30T13:59:03.520124+0000","last_peered":"2023-10-30T13:59:03.520124+0000","last_clean":"2023-10-30T13:59:03.520124+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:59:03.520124+0000","last_undegraded":"2023-10-30T13:59:03.520124+0000","last_fullsized":"2023-10-30T13:59:03.520124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092364379999999996,"stat_sum":{"num_bytes":16180540,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":126,"num_read_kb":8225,"num_write":670,"num_write_kb":34788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"56'176","reported_seq":248,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.774022+0000","last_change":"2023-10-30T13:59:25.774136+0000","last_active":"2023-10-30T13:59:25.774022+0000","last_peered":"2023-10-30T13:59:25.774022+0000","last_clean":"2023-10-30T13:59:25.774022+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:59:25.774022+0000","last_undegraded":"2023-10-30T13:59:25.774022+0000","last_fullsized":"2023-10-30T13:59:25.774022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'176","last_scrub_stamp":"2023-10-30T13:59:25.773814+0000","last_deep_scrub":"56'176","last_deep_scrub_stamp":"2023-10-30T13:59:25.773814+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.773814+0000","objects_scrubbed":5,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:01.029899+0000","scrub_duration":0.016157685000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4103,"num_write":119,"num_write_kb":4220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31622,"num_omap_keys":58,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"56'791","reported_seq":2999,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:29.956919+0000","last_change":"2023-10-30T13:59:29.956971+0000","last_active":"2023-10-30T13:59:29.956919+0000","last_peered":"2023-10-30T13:59:29.956919+0000","last_clean":"2023-10-30T13:59:29.956919+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:59:29.956919+0000","last_undegraded":"2023-10-30T13:59:29.956919+0000","last_fullsized":"2023-10-30T13:59:29.956919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'791","last_scrub_stamp":"2023-10-30T13:59:29.956832+0000","last_deep_scrub":"56'791","last_deep_scrub_stamp":"2023-10-30T13:59:29.956832+0000","last_clean_scrub_stamp":"2023-10-30T13:59:29.956832+0000","objects_scrubbed":152,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:19.118645+0000","scrub_duration":0.199579172,"objects_trimmed":0,"snaptrim_duration":0.0045350950000000003,"stat_sum":{"num_bytes":14031523,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":99,"num_read_kb":8208,"num_write":613,"num_write_kb":14857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"56'625","reported_seq":743,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.373340+0000","last_change":"2023-10-30T13:59:09.373401+0000","last_active":"2023-10-30T13:59:09.373340+0000","last_peered":"2023-10-30T13:59:09.373340+0000","last_clean":"2023-10-30T13:59:09.373340+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:59:09.373340+0000","last_undegraded":"2023-10-30T13:59:09.373340+0000","last_fullsized":"2023-10-30T13:59:09.373340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'625","last_scrub_stamp":"2023-10-30T13:59:09.373262+0000","last_deep_scrub":"56'625","last_deep_scrub_stamp":"2023-10-30T13:59:09.373262+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.373262+0000","objects_scrubbed":12,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:19.085134+0000","scrub_duration":0.038380236999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":130,"num_read_kb":41105,"num_write":605,"num_write_kb":12685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":87540,"num_omap_keys":149,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"56'854","reported_seq":3079,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.886946+0000","last_change":"2023-10-30T13:59:13.886999+0000","last_active":"2023-10-30T13:59:13.886946+0000","last_peered":"2023-10-30T13:59:13.886946+0000","last_clean":"2023-10-30T13:59:13.886946+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:59:13.886946+0000","last_undegraded":"2023-10-30T13:59:13.886946+0000","last_fullsized":"2023-10-30T13:59:13.886946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'854","last_scrub_stamp":"2023-10-30T13:59:13.886861+0000","last_deep_scrub":"56'854","last_deep_scrub_stamp":"2023-10-30T13:59:13.886861+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.886861+0000","objects_scrubbed":157,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:39.441296+0000","scrub_duration":0.25638365299999999,"objects_trimmed":0,"snaptrim_duration":0.0089036940000000002,"stat_sum":{"num_bytes":6058105,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":119,"num_read_kb":8210,"num_write":694,"num_write_kb":35073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'356","reported_seq":439,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.554847+0000","last_change":"2023-10-30T13:59:12.555089+0000","last_active":"2023-10-30T13:59:12.554847+0000","last_peered":"2023-10-30T13:59:12.554847+0000","last_clean":"2023-10-30T13:59:12.554847+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:59:12.554847+0000","last_undegraded":"2023-10-30T13:59:12.554847+0000","last_fullsized":"2023-10-30T13:59:12.554847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'356","last_scrub_stamp":"2023-10-30T13:59:12.554618+0000","last_deep_scrub":"56'356","last_deep_scrub_stamp":"2023-10-30T13:59:12.554618+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.554618+0000","objects_scrubbed":6,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:45.923483+0000","scrub_duration":0.048085234999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":28724,"num_write":357,"num_write_kb":8477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20972,"num_omap_keys":39,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"56'782","reported_seq":2990,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.912849+0000","last_change":"2023-10-30T13:59:12.912905+0000","last_active":"2023-10-30T13:59:12.912849+0000","last_peered":"2023-10-30T13:59:12.912849+0000","last_clean":"2023-10-30T13:59:12.912849+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:59:12.912849+0000","last_undegraded":"2023-10-30T13:59:12.912849+0000","last_fullsized":"2023-10-30T13:59:12.912849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'782","last_scrub_stamp":"2023-10-30T13:59:12.912738+0000","last_deep_scrub":"56'782","last_deep_scrub_stamp":"2023-10-30T13:59:12.912738+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.912738+0000","objects_scrubbed":175,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:22.069756+0000","scrub_duration":0.27457796699999998,"objects_trimmed":0,"snaptrim_duration":0.0094781629999999995,"stat_sum":{"num_bytes":6414224,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":99,"num_read_kb":8205,"num_write":617,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"56'644","reported_seq":730,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.281554+0000","last_change":"2023-10-30T13:59:12.281633+0000","last_active":"2023-10-30T13:59:12.281554+0000","last_peered":"2023-10-30T13:59:12.281554+0000","last_clean":"2023-10-30T13:59:12.281554+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:59:12.281554+0000","last_undegraded":"2023-10-30T13:59:12.281554+0000","last_fullsized":"2023-10-30T13:59:12.281554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'644","last_scrub_stamp":"2023-10-30T13:59:12.281461+0000","last_deep_scrub":"56'644","last_deep_scrub_stamp":"2023-10-30T13:59:12.281461+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.281461+0000","objects_scrubbed":6,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:09.939939+0000","scrub_duration":0.025851885000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":33457,"num_write":586,"num_write_kb":12711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33662,"num_omap_keys":63,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"56'676","reported_seq":2881,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:03.530680+0000","last_change":"2023-10-30T13:45:28.601986+0000","last_active":"2023-10-30T13:59:03.530680+0000","last_peered":"2023-10-30T13:59:03.530680+0000","last_clean":"2023-10-30T13:59:03.530680+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:59:03.530680+0000","last_undegraded":"2023-10-30T13:59:03.530680+0000","last_fullsized":"2023-10-30T13:59:03.530680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:35:07.244846+0000","last_clean_scrub_stamp":"2023-10-30T13:35:07.244846+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010320058,"stat_sum":{"num_bytes":13797516,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":111,"num_read_kb":8210,"num_write":631,"num_write_kb":35965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"56'405","reported_seq":488,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.759775+0000","last_change":"2023-10-30T13:59:09.759865+0000","last_active":"2023-10-30T13:59:09.759775+0000","last_peered":"2023-10-30T13:59:09.759775+0000","last_clean":"2023-10-30T13:59:09.759775+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:59:09.759775+0000","last_undegraded":"2023-10-30T13:59:09.759775+0000","last_fullsized":"2023-10-30T13:59:09.759775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'405","last_scrub_stamp":"2023-10-30T13:59:09.759704+0000","last_deep_scrub":"56'405","last_deep_scrub_stamp":"2023-10-30T13:59:09.759704+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.759704+0000","objects_scrubbed":4,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:26:19.730801+0000","scrub_duration":0.036239831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388678,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":36872,"num_write":370,"num_write_kb":12516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11466,"num_omap_keys":21,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"56'877","reported_seq":3121,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:30.927951+0000","last_change":"2023-10-30T13:59:30.928146+0000","last_active":"2023-10-30T13:59:30.927951+0000","last_peered":"2023-10-30T13:59:30.927951+0000","last_clean":"2023-10-30T13:59:30.927951+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:59:30.927951+0000","last_undegraded":"2023-10-30T13:59:30.927951+0000","last_fullsized":"2023-10-30T13:59:30.927951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'877","last_scrub_stamp":"2023-10-30T13:59:30.927723+0000","last_deep_scrub":"56'877","last_deep_scrub_stamp":"2023-10-30T13:59:30.927723+0000","last_clean_scrub_stamp":"2023-10-30T13:59:30.927723+0000","objects_scrubbed":150,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:06.613346+0000","scrub_duration":0.30010206900000003,"objects_trimmed":0,"snaptrim_duration":0.0066640720000000001,"stat_sum":{"num_bytes":18395347,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":135,"num_read_kb":4114,"num_write":711,"num_write_kb":24326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"56'227","reported_seq":304,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.292401+0000","last_change":"2023-10-30T13:59:11.292450+0000","last_active":"2023-10-30T13:59:11.292401+0000","last_peered":"2023-10-30T13:59:11.292401+0000","last_clean":"2023-10-30T13:59:11.292401+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:59:11.292401+0000","last_undegraded":"2023-10-30T13:59:11.292401+0000","last_fullsized":"2023-10-30T13:59:11.292401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'227","last_scrub_stamp":"2023-10-30T13:59:11.292333+0000","last_deep_scrub":"56'227","last_deep_scrub_stamp":"2023-10-30T13:59:11.292333+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.292333+0000","objects_scrubbed":5,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:09.441346+0000","scrub_duration":0.024139252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":20494,"num_write":198,"num_write_kb":4363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10920,"num_omap_keys":20,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"56'838","reported_seq":3052,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:36.884783+0000","last_change":"2023-10-30T13:59:36.884844+0000","last_active":"2023-10-30T13:59:36.884783+0000","last_peered":"2023-10-30T13:59:36.884783+0000","last_clean":"2023-10-30T13:59:36.884783+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:59:36.884783+0000","last_undegraded":"2023-10-30T13:59:36.884783+0000","last_fullsized":"2023-10-30T13:59:36.884783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'838","last_scrub_stamp":"2023-10-30T13:59:36.884678+0000","last_deep_scrub":"56'838","last_deep_scrub_stamp":"2023-10-30T13:59:36.884678+0000","last_clean_scrub_stamp":"2023-10-30T13:59:36.884678+0000","objects_scrubbed":161,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:17:27.989261+0000","scrub_duration":0.33098685500000002,"objects_trimmed":0,"snaptrim_duration":0.0084474579999999997,"stat_sum":{"num_bytes":18100965,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":106,"num_read_kb":8207,"num_write":635,"num_write_kb":40451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"56'119","reported_seq":223,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:14.603525+0000","last_change":"2023-10-30T13:59:14.603715+0000","last_active":"2023-10-30T13:59:14.603525+0000","last_peered":"2023-10-30T13:59:14.603525+0000","last_clean":"2023-10-30T13:59:14.603525+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:59:14.603525+0000","last_undegraded":"2023-10-30T13:59:14.603525+0000","last_fullsized":"2023-10-30T13:59:14.603525+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'119","last_scrub_stamp":"2023-10-30T13:59:14.603316+0000","last_deep_scrub":"56'119","last_deep_scrub_stamp":"2023-10-30T13:59:14.603316+0000","last_clean_scrub_stamp":"2023-10-30T13:59:14.603316+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:48.801390+0000","scrub_duration":0.012158702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'353","reported_seq":429,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.516018+0000","last_change":"2023-10-30T13:59:11.516196+0000","last_active":"2023-10-30T13:59:11.516018+0000","last_peered":"2023-10-30T13:59:11.516018+0000","last_clean":"2023-10-30T13:59:11.516018+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:59:11.516018+0000","last_undegraded":"2023-10-30T13:59:11.516018+0000","last_fullsized":"2023-10-30T13:59:11.516018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'353","last_scrub_stamp":"2023-10-30T13:59:11.515780+0000","last_deep_scrub":"56'353","last_deep_scrub_stamp":"2023-10-30T13:59:11.515780+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.515780+0000","objects_scrubbed":8,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:30.246583+0000","scrub_duration":0.054846111000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388674,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":20491,"num_write":340,"num_write_kb":8443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31208,"num_omap_keys":58,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'799","reported_seq":3017,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.131857+0000","last_change":"2023-10-30T13:59:17.131904+0000","last_active":"2023-10-30T13:59:17.131857+0000","last_peered":"2023-10-30T13:59:17.131857+0000","last_clean":"2023-10-30T13:59:17.131857+0000","last_became_active":"2023-10-30T13:35:48.847905+0000","last_became_peered":"2023-10-30T13:35:48.847905+0000","last_unstale":"2023-10-30T13:59:17.131857+0000","last_undegraded":"2023-10-30T13:59:17.131857+0000","last_fullsized":"2023-10-30T13:59:17.131857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'799","last_scrub_stamp":"2023-10-30T13:59:17.131773+0000","last_deep_scrub":"56'799","last_deep_scrub_stamp":"2023-10-30T13:59:17.131773+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.131773+0000","objects_scrubbed":182,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:20.503652+0000","scrub_duration":0.14335236700000001,"objects_trimmed":0,"snaptrim_duration":0.002389721,"stat_sum":{"num_bytes":10749534,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":100,"num_read_kb":11,"num_write":647,"num_write_kb":20249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"56'357","reported_seq":454,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:21.575803+0000","last_change":"2023-10-30T13:59:21.575920+0000","last_active":"2023-10-30T13:59:21.575803+0000","last_peered":"2023-10-30T13:59:21.575803+0000","last_clean":"2023-10-30T13:59:21.575803+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:59:21.575803+0000","last_undegraded":"2023-10-30T13:59:21.575803+0000","last_fullsized":"2023-10-30T13:59:21.575803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'357","last_scrub_stamp":"2023-10-30T13:59:21.575603+0000","last_deep_scrub":"56'357","last_deep_scrub_stamp":"2023-10-30T13:59:21.575603+0000","last_clean_scrub_stamp":"2023-10-30T13:59:21.575603+0000","objects_scrubbed":5,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:43.316435+0000","scrub_duration":0.021550125999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":102412,"num_write":307,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16214,"num_omap_keys":31,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"56'824","reported_seq":3051,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.910863+0000","last_change":"2023-10-30T13:59:16.910909+0000","last_active":"2023-10-30T13:59:16.910863+0000","last_peered":"2023-10-30T13:59:16.910863+0000","last_clean":"2023-10-30T13:59:16.910863+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:59:16.910863+0000","last_undegraded":"2023-10-30T13:59:16.910863+0000","last_fullsized":"2023-10-30T13:59:16.910863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'824","last_scrub_stamp":"2023-10-30T13:59:16.910789+0000","last_deep_scrub":"56'824","last_deep_scrub_stamp":"2023-10-30T13:59:16.910789+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.910789+0000","objects_scrubbed":156,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:06.670797+0000","scrub_duration":0.25255747699999997,"objects_trimmed":0,"snaptrim_duration":0.0018610090000000001,"stat_sum":{"num_bytes":24271212,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":117,"num_read_kb":10,"num_write":671,"num_write_kb":33782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"56'1505","reported_seq":2038,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.342054+0000","last_change":"2023-10-30T13:59:10.342107+0000","last_active":"2023-10-30T13:59:10.342054+0000","last_peered":"2023-10-30T13:59:10.342054+0000","last_clean":"2023-10-30T13:59:10.342054+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:59:10.342054+0000","last_undegraded":"2023-10-30T13:59:10.342054+0000","last_fullsized":"2023-10-30T13:59:10.342054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1505","last_scrub_stamp":"2023-10-30T13:59:10.341982+0000","last_deep_scrub":"56'1505","last_deep_scrub_stamp":"2023-10-30T13:59:10.341982+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.341982+0000","objects_scrubbed":9,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:03.471145+0000","scrub_duration":0.046294123999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959716,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":475,"num_read_kb":12479,"num_write":1460,"num_write_kb":17319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8598,"num_omap_keys":16,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'803","reported_seq":3030,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.925329+0000","last_change":"2023-10-30T13:59:20.925381+0000","last_active":"2023-10-30T13:59:20.925329+0000","last_peered":"2023-10-30T13:59:20.925329+0000","last_clean":"2023-10-30T13:59:20.925329+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:59:20.925329+0000","last_undegraded":"2023-10-30T13:59:20.925329+0000","last_fullsized":"2023-10-30T13:59:20.925329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'803","last_scrub_stamp":"2023-10-30T13:59:20.925256+0000","last_deep_scrub":"56'803","last_deep_scrub_stamp":"2023-10-30T13:59:20.925256+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.925256+0000","objects_scrubbed":166,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:15.483272+0000","scrub_duration":0.17049674500000001,"objects_trimmed":0,"snaptrim_duration":0.0022228600000000001,"stat_sum":{"num_bytes":9925305,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":116,"num_read_kb":4117,"num_write":612,"num_write_kb":24793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'634","reported_seq":716,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.440153+0000","last_change":"2023-10-30T13:59:16.440238+0000","last_active":"2023-10-30T13:59:16.440153+0000","last_peered":"2023-10-30T13:59:16.440153+0000","last_clean":"2023-10-30T13:59:16.440153+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:59:16.440153+0000","last_undegraded":"2023-10-30T13:59:16.440153+0000","last_fullsized":"2023-10-30T13:59:16.440153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'634","last_scrub_stamp":"2023-10-30T13:59:16.440060+0000","last_deep_scrub":"56'634","last_deep_scrub_stamp":"2023-10-30T13:59:16.440060+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.440060+0000","objects_scrubbed":4,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:53.542410+0000","scrub_duration":0.021825480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":61440,"num_write":567,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":28266,"num_omap_keys":53,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'840","reported_seq":3051,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.860098+0000","last_change":"2023-10-30T13:59:16.860153+0000","last_active":"2023-10-30T13:59:16.860098+0000","last_peered":"2023-10-30T13:59:16.860098+0000","last_clean":"2023-10-30T13:59:16.860098+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:59:16.860098+0000","last_undegraded":"2023-10-30T13:59:16.860098+0000","last_fullsized":"2023-10-30T13:59:16.860098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'840","last_scrub_stamp":"2023-10-30T13:59:16.860020+0000","last_deep_scrub":"56'840","last_deep_scrub_stamp":"2023-10-30T13:59:16.860020+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.860020+0000","objects_scrubbed":157,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:19.322252+0000","scrub_duration":0.29827281300000003,"objects_trimmed":0,"snaptrim_duration":0.0066434720000000001,"stat_sum":{"num_bytes":26554596,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":101,"num_read_kb":3528,"num_write":689,"num_write_kb":37408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"56'729","reported_seq":803,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.710386+0000","last_change":"2023-10-30T13:59:08.710437+0000","last_active":"2023-10-30T13:59:08.710386+0000","last_peered":"2023-10-30T13:59:08.710386+0000","last_clean":"2023-10-30T13:59:08.710386+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:59:08.710386+0000","last_undegraded":"2023-10-30T13:59:08.710386+0000","last_fullsized":"2023-10-30T13:59:08.710386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'729","last_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_deep_scrub":"56'729","last_deep_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_clean_scrub_stamp":"2023-10-30T13:59:08.710307+0000","objects_scrubbed":3,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:48.485825+0000","scrub_duration":0.018720046000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":18914,"num_write":694,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10538,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"56'858","reported_seq":3076,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:33.952048+0000","last_change":"2023-10-30T13:59:33.952262+0000","last_active":"2023-10-30T13:59:33.952048+0000","last_peered":"2023-10-30T13:59:33.952048+0000","last_clean":"2023-10-30T13:59:33.952048+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:59:33.952048+0000","last_undegraded":"2023-10-30T13:59:33.952048+0000","last_fullsized":"2023-10-30T13:59:33.952048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'858","last_scrub_stamp":"2023-10-30T13:59:33.951806+0000","last_deep_scrub":"56'858","last_deep_scrub_stamp":"2023-10-30T13:59:33.951806+0000","last_clean_scrub_stamp":"2023-10-30T13:59:33.951806+0000","objects_scrubbed":160,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:30.144633+0000","scrub_duration":0.36214772699999997,"objects_trimmed":0,"snaptrim_duration":0.0064085849999999996,"stat_sum":{"num_bytes":10071990,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":4109,"num_write":659,"num_write_kb":40765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"56'763","reported_seq":842,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:33.576155+0000","last_change":"2023-10-30T13:59:33.576384+0000","last_active":"2023-10-30T13:59:33.576155+0000","last_peered":"2023-10-30T13:59:33.576155+0000","last_clean":"2023-10-30T13:59:33.576155+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:59:33.576155+0000","last_undegraded":"2023-10-30T13:59:33.576155+0000","last_fullsized":"2023-10-30T13:59:33.576155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'763","last_scrub_stamp":"2023-10-30T13:59:33.575889+0000","last_deep_scrub":"56'763","last_deep_scrub_stamp":"2023-10-30T13:59:33.575889+0000","last_clean_scrub_stamp":"2023-10-30T13:59:33.575889+0000","objects_scrubbed":7,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:16.668864+0000","scrub_duration":0.061119272000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":707,"num_write_kb":16664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7670,"num_omap_keys":15,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"56'810","reported_seq":3028,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:27.812278+0000","last_change":"2023-10-30T13:59:27.812324+0000","last_active":"2023-10-30T13:59:27.812278+0000","last_peered":"2023-10-30T13:59:27.812278+0000","last_clean":"2023-10-30T13:59:27.812278+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:59:27.812278+0000","last_undegraded":"2023-10-30T13:59:27.812278+0000","last_fullsized":"2023-10-30T13:59:27.812278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'810","last_scrub_stamp":"2023-10-30T13:59:27.812198+0000","last_deep_scrub":"56'810","last_deep_scrub_stamp":"2023-10-30T13:59:27.812198+0000","last_clean_scrub_stamp":"2023-10-30T13:59:27.812198+0000","objects_scrubbed":155,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:19.443546+0000","scrub_duration":0.136984935,"objects_trimmed":0,"snaptrim_duration":0.001735594,"stat_sum":{"num_bytes":6415761,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":108,"num_read_kb":4114,"num_write":644,"num_write_kb":12898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"56'283","reported_seq":370,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.889445+0000","last_change":"2023-10-30T13:59:11.889543+0000","last_active":"2023-10-30T13:59:11.889445+0000","last_peered":"2023-10-30T13:59:11.889445+0000","last_clean":"2023-10-30T13:59:11.889445+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:59:11.889445+0000","last_undegraded":"2023-10-30T13:59:11.889445+0000","last_fullsized":"2023-10-30T13:59:11.889445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'283","last_scrub_stamp":"2023-10-30T13:59:11.889356+0000","last_deep_scrub":"56'283","last_deep_scrub_stamp":"2023-10-30T13:59:11.889356+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.889356+0000","objects_scrubbed":6,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:04.195980+0000","scrub_duration":0.13641584700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":698,"num_write":517,"num_write_kb":2842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":36848,"num_omap_keys":69,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"56'769","reported_seq":2985,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:35.232420+0000","last_change":"2023-10-30T13:59:35.232463+0000","last_active":"2023-10-30T13:59:35.232420+0000","last_peered":"2023-10-30T13:59:35.232420+0000","last_clean":"2023-10-30T13:59:35.232420+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:59:35.232420+0000","last_undegraded":"2023-10-30T13:59:35.232420+0000","last_fullsized":"2023-10-30T13:59:35.232420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'769","last_scrub_stamp":"2023-10-30T13:59:35.232349+0000","last_deep_scrub":"56'769","last_deep_scrub_stamp":"2023-10-30T13:59:35.232349+0000","last_clean_scrub_stamp":"2023-10-30T13:59:35.232349+0000","objects_scrubbed":151,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:25:20.721025+0000","scrub_duration":0.212346017,"objects_trimmed":0,"snaptrim_duration":0.002858771,"stat_sum":{"num_bytes":14417139,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":106,"num_read_kb":4110,"num_write":585,"num_write_kb":24357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"56'124","reported_seq":210,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.849710+0000","last_change":"2023-10-30T13:59:17.849829+0000","last_active":"2023-10-30T13:59:17.849710+0000","last_peered":"2023-10-30T13:59:17.849710+0000","last_clean":"2023-10-30T13:59:17.849710+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:59:17.849710+0000","last_undegraded":"2023-10-30T13:59:17.849710+0000","last_fullsized":"2023-10-30T13:59:17.849710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'124","last_scrub_stamp":"2023-10-30T13:59:17.849515+0000","last_deep_scrub":"56'124","last_deep_scrub_stamp":"2023-10-30T13:59:17.849515+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.849515+0000","objects_scrubbed":7,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:54.554158+0000","scrub_duration":0.025438601000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":20522,"num_write":112,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30984,"num_omap_keys":57,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"56'824","reported_seq":3046,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.809876+0000","last_change":"2023-10-30T13:59:17.809939+0000","last_active":"2023-10-30T13:59:17.809876+0000","last_peered":"2023-10-30T13:59:17.809876+0000","last_clean":"2023-10-30T13:59:17.809876+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:59:17.809876+0000","last_undegraded":"2023-10-30T13:59:17.809876+0000","last_fullsized":"2023-10-30T13:59:17.809876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'824","last_scrub_stamp":"2023-10-30T13:59:17.809790+0000","last_deep_scrub":"56'824","last_deep_scrub_stamp":"2023-10-30T13:59:17.809790+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.809790+0000","objects_scrubbed":162,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:00.364334+0000","scrub_duration":0.16950805699999999,"objects_trimmed":0,"snaptrim_duration":0.0027315170000000001,"stat_sum":{"num_bytes":18657914,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":112,"num_read_kb":8214,"num_write":646,"num_write_kb":32163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"56'148","reported_seq":226,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.625271+0000","last_change":"2023-10-30T13:59:17.625403+0000","last_active":"2023-10-30T13:59:17.625271+0000","last_peered":"2023-10-30T13:59:17.625271+0000","last_clean":"2023-10-30T13:59:17.625271+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:59:17.625271+0000","last_undegraded":"2023-10-30T13:59:17.625271+0000","last_fullsized":"2023-10-30T13:59:17.625271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'148","last_scrub_stamp":"2023-10-30T13:59:17.625064+0000","last_deep_scrub":"56'148","last_deep_scrub_stamp":"2023-10-30T13:59:17.625064+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.625064+0000","objects_scrubbed":3,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:01.340603+0000","scrub_duration":0.011205155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":30625,"num_write":106,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25608,"num_omap_keys":47,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"56'788","reported_seq":2995,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.814579+0000","last_change":"2023-10-30T13:59:13.814677+0000","last_active":"2023-10-30T13:59:13.814579+0000","last_peered":"2023-10-30T13:59:13.814579+0000","last_clean":"2023-10-30T13:59:13.814579+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:59:13.814579+0000","last_undegraded":"2023-10-30T13:59:13.814579+0000","last_fullsized":"2023-10-30T13:59:13.814579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'788","last_scrub_stamp":"2023-10-30T13:59:13.814451+0000","last_deep_scrub":"56'788","last_deep_scrub_stamp":"2023-10-30T13:59:13.814451+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.814451+0000","objects_scrubbed":145,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:05.475962+0000","scrub_duration":0.26681934600000001,"objects_trimmed":0,"snaptrim_duration":0.0075826330000000001,"stat_sum":{"num_bytes":5755271,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":98,"num_read_kb":7,"num_write":639,"num_write_kb":16186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'946","reported_seq":1021,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.799532+0000","last_change":"2023-10-30T13:59:16.799588+0000","last_active":"2023-10-30T13:59:16.799532+0000","last_peered":"2023-10-30T13:59:16.799532+0000","last_clean":"2023-10-30T13:59:16.799532+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:59:16.799532+0000","last_undegraded":"2023-10-30T13:59:16.799532+0000","last_fullsized":"2023-10-30T13:59:16.799532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'946","last_scrub_stamp":"2023-10-30T13:59:16.799459+0000","last_deep_scrub":"56'946","last_deep_scrub_stamp":"2023-10-30T13:59:16.799459+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.799459+0000","objects_scrubbed":6,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:25.410921+0000","scrub_duration":0.051098538999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":20487,"num_write":909,"num_write_kb":8631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16326,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"56'874","reported_seq":3095,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:36.184240+0000","last_change":"2023-10-30T13:59:36.184295+0000","last_active":"2023-10-30T13:59:36.184240+0000","last_peered":"2023-10-30T13:59:36.184240+0000","last_clean":"2023-10-30T13:59:36.184240+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:59:36.184240+0000","last_undegraded":"2023-10-30T13:59:36.184240+0000","last_fullsized":"2023-10-30T13:59:36.184240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'874","last_scrub_stamp":"2023-10-30T13:59:36.184161+0000","last_deep_scrub":"56'874","last_deep_scrub_stamp":"2023-10-30T13:59:36.184161+0000","last_clean_scrub_stamp":"2023-10-30T13:59:36.184161+0000","objects_scrubbed":147,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:44:59.440315+0000","scrub_duration":0.16968080599999999,"objects_trimmed":0,"snaptrim_duration":0.0029726169999999999,"stat_sum":{"num_bytes":19889578,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":112,"num_read_kb":12306,"num_write":688,"num_write_kb":31224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"56'485","reported_seq":567,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.778391+0000","last_change":"2023-10-30T13:59:15.778438+0000","last_active":"2023-10-30T13:59:15.778391+0000","last_peered":"2023-10-30T13:59:15.778391+0000","last_clean":"2023-10-30T13:59:15.778391+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:59:15.778391+0000","last_undegraded":"2023-10-30T13:59:15.778391+0000","last_fullsized":"2023-10-30T13:59:15.778391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'485","last_scrub_stamp":"2023-10-30T13:59:15.778326+0000","last_deep_scrub":"56'485","last_deep_scrub_stamp":"2023-10-30T13:59:15.778326+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.778326+0000","objects_scrubbed":8,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:34.958856+0000","scrub_duration":0.04450407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":53262,"num_write":506,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35586,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"56'849","reported_seq":3070,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:44.494907+0000","last_change":"2023-10-30T13:59:44.494969+0000","last_active":"2023-10-30T13:59:44.494907+0000","last_peered":"2023-10-30T13:59:44.494907+0000","last_clean":"2023-10-30T13:59:44.494907+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:59:44.494907+0000","last_undegraded":"2023-10-30T13:59:44.494907+0000","last_fullsized":"2023-10-30T13:59:44.494907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'849","last_scrub_stamp":"2023-10-30T13:59:44.494821+0000","last_deep_scrub":"56'849","last_deep_scrub_stamp":"2023-10-30T13:59:44.494821+0000","last_clean_scrub_stamp":"2023-10-30T13:59:44.494821+0000","objects_scrubbed":158,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:02.881643+0000","scrub_duration":0.38383446399999999,"objects_trimmed":0,"snaptrim_duration":0.002600925,"stat_sum":{"num_bytes":9733947,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":111,"num_read_kb":4112,"num_write":647,"num_write_kb":24811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"56'1293","reported_seq":1389,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.627944+0000","last_change":"2023-10-30T13:59:25.628077+0000","last_active":"2023-10-30T13:59:25.627944+0000","last_peered":"2023-10-30T13:59:25.627944+0000","last_clean":"2023-10-30T13:59:25.627944+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:59:25.627944+0000","last_undegraded":"2023-10-30T13:59:25.627944+0000","last_fullsized":"2023-10-30T13:59:25.627944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1293","last_scrub_stamp":"2023-10-30T13:59:25.627729+0000","last_deep_scrub":"56'1293","last_deep_scrub_stamp":"2023-10-30T13:59:25.627729+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.627729+0000","objects_scrubbed":8,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:46:32.130653+0000","scrub_duration":0.082082413000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":81960,"num_write":1252,"num_write_kb":21130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27956,"num_omap_keys":18,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"56'873","reported_seq":3091,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:35.136342+0000","last_change":"2023-10-30T13:59:35.136398+0000","last_active":"2023-10-30T13:59:35.136342+0000","last_peered":"2023-10-30T13:59:35.136342+0000","last_clean":"2023-10-30T13:59:35.136342+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:59:35.136342+0000","last_undegraded":"2023-10-30T13:59:35.136342+0000","last_fullsized":"2023-10-30T13:59:35.136342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'873","last_scrub_stamp":"2023-10-30T13:59:35.136239+0000","last_deep_scrub":"56'873","last_deep_scrub_stamp":"2023-10-30T13:59:35.136239+0000","last_clean_scrub_stamp":"2023-10-30T13:59:35.136239+0000","objects_scrubbed":158,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:12:25.717527+0000","scrub_duration":0.34861871900000002,"objects_trimmed":0,"snaptrim_duration":0.0051869250000000002,"stat_sum":{"num_bytes":18626842,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":5015,"num_write":667,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"56'799","reported_seq":3029,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:43.912820+0000","last_change":"2023-10-30T13:59:43.912906+0000","last_active":"2023-10-30T13:59:43.912820+0000","last_peered":"2023-10-30T13:59:43.912820+0000","last_clean":"2023-10-30T13:59:43.912820+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:59:43.912820+0000","last_undegraded":"2023-10-30T13:59:43.912820+0000","last_fullsized":"2023-10-30T13:59:43.912820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'799","last_scrub_stamp":"2023-10-30T13:59:43.912716+0000","last_deep_scrub":"56'799","last_deep_scrub_stamp":"2023-10-30T13:59:43.912716+0000","last_clean_scrub_stamp":"2023-10-30T13:59:43.912716+0000","objects_scrubbed":142,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:53.615287+0000","scrub_duration":0.26467791000000002,"objects_trimmed":0,"snaptrim_duration":0.0061609869999999997,"stat_sum":{"num_bytes":14245271,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":123,"num_read_kb":12313,"num_write":620,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"56'235","reported_seq":316,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.601546+0000","last_change":"2023-10-30T13:59:24.601686+0000","last_active":"2023-10-30T13:59:24.601546+0000","last_peered":"2023-10-30T13:59:24.601546+0000","last_clean":"2023-10-30T13:59:24.601546+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:59:24.601546+0000","last_undegraded":"2023-10-30T13:59:24.601546+0000","last_fullsized":"2023-10-30T13:59:24.601546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'235","last_scrub_stamp":"2023-10-30T13:59:24.601285+0000","last_deep_scrub":"56'235","last_deep_scrub_stamp":"2023-10-30T13:59:24.601285+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.601285+0000","objects_scrubbed":4,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:28.578041+0000","scrub_duration":0.016429274000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":24606,"num_write":206,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35337,"num_omap_keys":63,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"56'894","reported_seq":3115,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:46.931313+0000","last_change":"2023-10-30T13:59:46.931391+0000","last_active":"2023-10-30T13:59:46.931313+0000","last_peered":"2023-10-30T13:59:46.931313+0000","last_clean":"2023-10-30T13:59:46.931313+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:59:46.931313+0000","last_undegraded":"2023-10-30T13:59:46.931313+0000","last_fullsized":"2023-10-30T13:59:46.931313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'894","last_scrub_stamp":"2023-10-30T13:59:46.931198+0000","last_deep_scrub":"56'894","last_deep_scrub_stamp":"2023-10-30T13:59:46.931198+0000","last_clean_scrub_stamp":"2023-10-30T13:59:46.931198+0000","objects_scrubbed":181,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:39.907025+0000","scrub_duration":0.31800989200000002,"objects_trimmed":0,"snaptrim_duration":0.0061457359999999997,"stat_sum":{"num_bytes":11313833,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":112,"num_read_kb":16,"num_write":670,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"56'957","reported_seq":1060,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:30.546252+0000","last_change":"2023-10-30T13:59:30.546377+0000","last_active":"2023-10-30T13:59:30.546252+0000","last_peered":"2023-10-30T13:59:30.546252+0000","last_clean":"2023-10-30T13:59:30.546252+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:59:30.546252+0000","last_undegraded":"2023-10-30T13:59:30.546252+0000","last_fullsized":"2023-10-30T13:59:30.546252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'957","last_scrub_stamp":"2023-10-30T13:59:30.546025+0000","last_deep_scrub":"56'957","last_deep_scrub_stamp":"2023-10-30T13:59:30.546025+0000","last_clean_scrub_stamp":"2023-10-30T13:59:30.546025+0000","objects_scrubbed":7,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:04.224187+0000","scrub_duration":0.051378422999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":103644,"num_write":908,"num_write_kb":16901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17921,"num_omap_keys":34,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"56'829","reported_seq":3061,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:26.753705+0000","last_change":"2023-10-30T13:59:26.753762+0000","last_active":"2023-10-30T13:59:26.753705+0000","last_peered":"2023-10-30T13:59:26.753705+0000","last_clean":"2023-10-30T13:59:26.753705+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:59:26.753705+0000","last_undegraded":"2023-10-30T13:59:26.753705+0000","last_fullsized":"2023-10-30T13:59:26.753705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'829","last_scrub_stamp":"2023-10-30T13:59:26.753633+0000","last_deep_scrub":"56'829","last_deep_scrub_stamp":"2023-10-30T13:59:26.753633+0000","last_clean_scrub_stamp":"2023-10-30T13:59:26.753633+0000","objects_scrubbed":184,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:01:43.957619+0000","scrub_duration":0.27683662399999998,"objects_trimmed":0,"snaptrim_duration":0.0029730300000000002,"stat_sum":{"num_bytes":19886092,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":121,"num_read_kb":4113,"num_write":647,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"56'590","reported_seq":679,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.115956+0000","last_change":"2023-10-30T13:59:13.116019+0000","last_active":"2023-10-30T13:59:13.115956+0000","last_peered":"2023-10-30T13:59:13.115956+0000","last_clean":"2023-10-30T13:59:13.115956+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:59:13.115956+0000","last_undegraded":"2023-10-30T13:59:13.115956+0000","last_fullsized":"2023-10-30T13:59:13.115956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'590","last_scrub_stamp":"2023-10-30T13:59:13.115857+0000","last_deep_scrub":"56'590","last_deep_scrub_stamp":"2023-10-30T13:59:13.115857+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.115857+0000","objects_scrubbed":4,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:55.093851+0000","scrub_duration":0.036041838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":72355,"num_write":540,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14742,"num_omap_keys":27,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"56'816","reported_seq":3025,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:45.340376+0000","last_change":"2023-10-30T13:59:45.340430+0000","last_active":"2023-10-30T13:59:45.340376+0000","last_peered":"2023-10-30T13:59:45.340376+0000","last_clean":"2023-10-30T13:59:45.340376+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:59:45.340376+0000","last_undegraded":"2023-10-30T13:59:45.340376+0000","last_fullsized":"2023-10-30T13:59:45.340376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'816","last_scrub_stamp":"2023-10-30T13:59:45.340302+0000","last_deep_scrub":"56'816","last_deep_scrub_stamp":"2023-10-30T13:59:45.340302+0000","last_clean_scrub_stamp":"2023-10-30T13:59:45.340302+0000","objects_scrubbed":156,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:39.933055+0000","scrub_duration":0.26337302099999998,"objects_trimmed":0,"snaptrim_duration":0.0026777319999999999,"stat_sum":{"num_bytes":7974576,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":98,"num_read_kb":4107,"num_write":628,"num_write_kb":17606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"56'364","reported_seq":468,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.646097+0000","last_change":"2023-10-30T13:59:15.646200+0000","last_active":"2023-10-30T13:59:15.646097+0000","last_peered":"2023-10-30T13:59:15.646097+0000","last_clean":"2023-10-30T13:59:15.646097+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:59:15.646097+0000","last_undegraded":"2023-10-30T13:59:15.646097+0000","last_fullsized":"2023-10-30T13:59:15.646097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'364","last_scrub_stamp":"2023-10-30T13:59:15.645995+0000","last_deep_scrub":"56'364","last_deep_scrub_stamp":"2023-10-30T13:59:15.645995+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.645995+0000","objects_scrubbed":11,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:40.607663+0000","scrub_duration":0.031506254999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4418509,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":57427,"num_write":339,"num_write_kb":8706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35486,"num_omap_keys":15,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":925185184,"num_objects":10324,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":7268,"num_read_kb":400185,"num_write":41527,"num_write_kb":1950152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1902518272,"data_stored":1848830272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51876,"ondisk_log_size":51876,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":260405585,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":1553,"num_read_kb":1442867,"num_write":16574,"num_write_kb":377787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":782235,"num_omap_keys":1350,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520970240,"data_stored":520811170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17546,"ondisk_log_size":17546,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509744,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445620,"kb_used_data":333932,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93926220,"statfs":{"total":96636764160,"available":96180449280,"internally_reserved":0,"allocated":341946368,"data_stored":333745462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":114358731},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.84299999999999997,"15min":0.81000000000000005},"min":{"1min":0.51700000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.353,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.016},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.94899999999999995,"15min":0.91000000000000003},"min":{"1min":0.52700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2709999999999999,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.1779999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.82099999999999995,"15min":0.90900000000000003},"min":{"1min":0.503,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.04,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.002,"5min":0.97899999999999998,"15min":0.92500000000000004},"min":{"1min":0.58799999999999997,"5min":0.45200000000000001,"15min":0.33400000000000002},"max":{"1min":1.357,"5min":1.577,"15min":1.625},"last":1.3620000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.018,"15min":0.97899999999999998},"min":{"1min":0.622,"5min":0.51300000000000001,"15min":0.32000000000000001},"max":{"1min":1.4410000000000001,"5min":1.468,"15min":1.468},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.94899999999999995,"15min":0.96099999999999997},"min":{"1min":0.65000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.6279999999999999,"5min":3.7810000000000001,"15min":4.1120000000000001},"last":1.518}]},{"osd":3,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.1120000000000001,"15min":1.0900000000000001},"min":{"1min":0.78600000000000003,"5min":0.55300000000000005,"15min":0.45900000000000002},"max":{"1min":1.6659999999999999,"5min":1.7669999999999999,"15min":4.2069999999999999},"last":1.28},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.605,"15min":1.1950000000000001},"min":{"1min":0.50600000000000001,"5min":0.47199999999999998,"15min":0.44400000000000001},"max":{"1min":1.3220000000000001,"5min":59.844999999999999,"15min":59.844999999999999},"last":1.05}]},{"osd":4,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.163,"15min":1.1080000000000001},"min":{"1min":0.98099999999999998,"5min":0.623,"15min":0.38100000000000001},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":3.1059999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.45,"5min":1.3440000000000001,"15min":1.298},"min":{"1min":1.1930000000000001,"5min":0.754,"15min":0.42399999999999999},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":3.198},"last":1.4390000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.012,"15min":1.0860000000000001},"min":{"1min":0.66200000000000003,"5min":0.53100000000000003,"15min":0.40300000000000002},"max":{"1min":1.758,"5min":3.9510000000000001,"15min":4.2759999999999998},"last":0.63},{"interface":"front","average":{"1min":1.377,"5min":1.2809999999999999,"15min":1.258},"min":{"1min":1.0680000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":3.4380000000000002},"last":1.599}]},{"osd":6,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92700000000000005,"15min":1.0760000000000001},"min":{"1min":0.754,"5min":0.48999999999999999,"15min":0.48399999999999999},"max":{"1min":1.351,"5min":1.429,"15min":3.9630000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.133,"15min":1.2090000000000001},"min":{"1min":0.75,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.9019999999999999,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.95199999999999996}]}]},{"osd":3,"up_from":15,"seq":64424509743,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429856,"kb_used_data":318168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93941984,"statfs":{"total":96636764160,"available":96196591616,"internally_reserved":0,"allocated":325804032,"data_stored":317570953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":114358728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47799999999999998,"15min":0.52100000000000002},"min":{"1min":0.27100000000000002,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.74099999999999999,"5min":0.78600000000000003,"15min":9.3320000000000007},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.0349999999999999,"15min":0.72899999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.308},"max":{"1min":0.65800000000000003,"5min":38.082999999999998,"15min":38.082999999999998},"last":0.623}]},{"osd":1,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.69599999999999995,"15min":0.61399999999999999},"min":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.30099999999999999},"max":{"1min":0.67700000000000005,"5min":11.599,"15min":11.599},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.70599999999999996,"15min":0.66000000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.72199999999999998,"5min":15.715999999999999,"15min":15.715999999999999},"last":0.71899999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.73199999999999998,"15min":0.67300000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.88600000000000001,"5min":11.646000000000001,"15min":12.351000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.76200000000000001,"15min":0.65400000000000003},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":15.641999999999999,"15min":15.641999999999999},"last":0.65600000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.012,"15min":0.92300000000000004},"min":{"1min":0.59799999999999998,"5min":0.55400000000000005,"15min":0.51600000000000001},"max":{"1min":1.0620000000000001,"5min":15.759,"15min":15.759},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.752,"5min":1.268,"15min":1.054},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.296,"5min":38.137999999999998,"15min":38.137999999999998},"last":0.78900000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.4059999999999999,"15min":1.1000000000000001},"min":{"1min":0.626,"5min":0.59299999999999997,"15min":0.52600000000000002},"max":{"1min":1.0469999999999999,"5min":40.808,"15min":40.808},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.032,"15min":0.95899999999999996},"min":{"1min":0.56100000000000005,"5min":0.55800000000000005,"15min":0.55300000000000005},"max":{"1min":1.127,"5min":16.498000000000001,"15min":19.835999999999999},"last":0.871}]},{"osd":6,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.96999999999999997,"15min":0.92000000000000004},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.53500000000000003},"max":{"1min":1.085,"5min":11.701000000000001,"15min":11.701000000000001},"last":1.085},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.024,"15min":0.96299999999999997},"min":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":1.1870000000000001,"5min":11.747999999999999,"15min":11.747999999999999},"last":1.1759999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1020000000000001,"15min":0.96499999999999997},"min":{"1min":0.66000000000000003,"5min":0.60599999999999998,"15min":0.59999999999999998},"max":{"1min":1.1240000000000001,"5min":16.082999999999998,"15min":16.082999999999998},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.569,"15min":1.113},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.2370000000000001,"5min":52.902999999999999,"15min":52.902999999999999},"last":1.1220000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258464,"kb_used_data":187096,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94113376,"statfs":{"total":96636764160,"available":96372097024,"internally_reserved":0,"allocated":191586304,"data_stored":186614303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.29399999999999998,"5min":0.251,"15min":0.246},"max":{"1min":2.242,"5min":3.407,"15min":3.8969999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.9430000000000001,"5min":0.85899999999999999,"15min":0.67300000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33300000000000002},"max":{"1min":23.335999999999999,"5min":23.335999999999999,"15min":23.335999999999999},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.56299999999999994,"15min":0.62},"min":{"1min":0.33200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.5919999999999996,"5min":5.5750000000000002,"15min":36.109000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.337,"5min":0.73099999999999998,"15min":0.627},"min":{"1min":0.311,"5min":0.311,"15min":0.28599999999999998},"max":{"1min":13.584,"5min":13.584,"15min":13.584},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.65300000000000002,"15min":0.70199999999999996},"min":{"1min":0.437,"5min":0.35899999999999999,"15min":0.29199999999999998},"max":{"1min":2.339,"5min":5.532,"15min":35.890999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.431,"5min":0.83099999999999996,"15min":0.75600000000000001},"min":{"1min":0.47699999999999998,"5min":0.42899999999999999,"15min":0.371},"max":{"1min":13.706,"5min":13.706,"15min":22.846},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":3.8690000000000002,"5min":1.446,"15min":1.0429999999999999},"min":{"1min":0.66200000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":48.710999999999999,"5min":48.710999999999999,"15min":48.710999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.86399999999999999,"15min":0.85499999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.302,"5min":5.6059999999999999,"15min":7.8220000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":2.504,"5min":1.1919999999999999,"15min":0.98799999999999999},"min":{"1min":0.58499999999999996,"5min":0.442,"15min":0.442},"max":{"1min":26.788,"5min":26.788,"15min":26.788},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.71,"5min":1.0580000000000001,"15min":0.92500000000000004},"min":{"1min":0.66900000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":13.654999999999999,"5min":13.654999999999999,"15min":13.654999999999999},"last":0.90800000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:58:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":1.2030000000000001,"15min":0.95999999999999996},"min":{"1min":0.64100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0960000000000001,"15min":0.97599999999999998},"min":{"1min":0.68300000000000005,"5min":0.60099999999999998,"15min":0.56599999999999995},"max":{"1min":15.975,"5min":15.975,"15min":15.975},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.087,"15min":0.96399999999999997},"min":{"1min":0.70499999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":13.747999999999999,"5min":13.747999999999999,"15min":13.747999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.738,"5min":1.0860000000000001,"15min":0.97199999999999998},"min":{"1min":0.61399999999999999,"5min":0.53900000000000003,"15min":0.53700000000000003},"max":{"1min":13.779999999999999,"5min":13.779999999999999,"15min":13.779999999999999},"last":0.86899999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509742,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357220,"kb_used_data":245596,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94014620,"statfs":{"total":96636764160,"available":96270970880,"internally_reserved":0,"allocated":251490304,"data_stored":246176700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114293190},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.51000000000000001,"15min":0.46000000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.7469999999999999,"5min":4.7469999999999999,"15min":4.7469999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":6.577,"5min":6.577,"15min":22.149000000000001},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":1.905,"5min":0.76900000000000002,"15min":0.56399999999999995},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":25.263999999999999,"5min":25.263999999999999,"15min":25.263999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.66600000000000004,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":6.056,"5min":6.056,"15min":6.056},"last":0.53900000000000003}]},{"osd":3,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.64100000000000001,"15min":0.78100000000000003},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":54.530000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.72399999999999998,"15min":0.71499999999999997},"min":{"1min":0.437,"5min":0.40100000000000002,"15min":0.34100000000000003},"max":{"1min":7.1070000000000002,"5min":7.1070000000000002,"15min":25.457000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":0.97299999999999998,"15min":0.873},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":14.019,"5min":14.019,"15min":14.019},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.1319999999999999,"5min":0.88,"15min":0.81999999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":4.71,"5min":4.71,"15min":4.71},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":0.95299999999999996,"15min":0.91300000000000003},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.505},"max":{"1min":11.82,"5min":11.82,"15min":11.82},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.4119999999999999,"5min":0.95499999999999996,"15min":0.879},"min":{"1min":0.63300000000000001,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":10.568,"5min":10.568,"15min":10.568},"last":0.876}]},{"osd":6,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":2.9420000000000002,"5min":1.2350000000000001,"15min":0.97799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.497},"max":{"1min":38.530999999999999,"5min":38.530999999999999,"15min":38.530999999999999},"last":1.008},{"interface":"front","average":{"1min":1.7869999999999999,"5min":1.0249999999999999,"15min":0.91200000000000003},"min":{"1min":0.64100000000000001,"5min":0.56299999999999994,"15min":0.47999999999999998},"max":{"1min":14.568,"5min":14.568,"15min":14.568},"last":0.82899999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:58:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.002,"15min":0.96899999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":12.221,"5min":12.221,"15min":12.221},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.373,"5min":1.0209999999999999,"15min":0.94499999999999995},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":12.153,"5min":12.153,"15min":12.153},"last":0.98999999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332612,"kb_used_data":247356,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94039228,"statfs":{"total":96636764160,"available":96296169472,"internally_reserved":0,"allocated":253292544,"data_stored":248134929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87292359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.1299999999999999,"15min":1.0249999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.41999999999999998},"max":{"1min":1.1879999999999999,"5min":19.741,"15min":19.741},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.036,"5min":1.1359999999999999,"15min":1.002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.34899999999999998},"max":{"1min":1.742,"5min":13.863,"15min":13.863},"last":0.746}]},{"osd":1,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.92500000000000004,"15min":0.86799999999999999},"min":{"1min":0.52400000000000002,"5min":0.496,"15min":0.48299999999999998},"max":{"1min":1.1839999999999999,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.179,"15min":1.0369999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.4279999999999999,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":1.0880000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96299999999999997,"15min":0.97999999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.7130000000000001,"5min":3.5329999999999999,"15min":3.7709999999999999},"last":0.997},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0760000000000001,"15min":1.079},"min":{"1min":0.60199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7549999999999999,"5min":3.407,"15min":4.6840000000000002},"last":0.64500000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.036,"15min":1.0189999999999999},"min":{"1min":0.64300000000000002,"5min":0.52800000000000002,"15min":0.46700000000000003},"max":{"1min":2.0059999999999998,"5min":3.7999999999999998,"15min":4.7629999999999999},"last":1.143},{"interface":"front","average":{"1min":1.224,"5min":1.1870000000000001,"15min":1.109},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.909,"5min":3.968,"15min":3.968},"last":1.1879999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1739999999999999,"15min":1.202},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5309999999999999,"5min":3.8900000000000001,"15min":16.204000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2250000000000001,"15min":1.266},"min":{"1min":0.70599999999999996,"5min":0.622,"15min":0.42699999999999999},"max":{"1min":2.0899999999999999,"5min":19.398,"15min":43.198},"last":1.2889999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96399999999999997,"15min":1.0409999999999999},"min":{"1min":0.51200000000000001,"5min":0.47699999999999998,"15min":0.41699999999999998},"max":{"1min":1.484,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.373,"5min":1.3129999999999999,"15min":1.3069999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.58599999999999997},"max":{"1min":2.1819999999999999,"5min":3.7269999999999999,"15min":3.802},"last":1.2370000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.8340000000000001,"15min":1.5900000000000001},"min":{"1min":0.65200000000000002,"5min":0.58899999999999997,"15min":0.55400000000000005},"max":{"1min":1.6559999999999999,"5min":70.335999999999999,"15min":73.228999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.097,"5min":1.661,"15min":1.629},"min":{"1min":0.625,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6100000000000001,"5min":42.164999999999999,"15min":73.289000000000001},"last":1.05}]}]},{"osd":5,"up_from":14,"seq":60129542446,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425100,"kb_used_data":335620,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93946740,"statfs":{"total":96636764160,"available":96201461760,"internally_reserved":0,"allocated":343674880,"data_stored":335552063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.88300000000000001,"15min":0.872},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.40400000000000003},"max":{"1min":1.7350000000000001,"5min":3.9159999999999999,"15min":5.0599999999999996},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.0429999999999999,"5min":0.86699999999999999,"15min":0.876},"min":{"1min":0.47699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.89300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.95499999999999996,"15min":0.876},"min":{"1min":0.52300000000000002,"5min":0.40600000000000003,"15min":0.33600000000000002},"max":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"last":1},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.85499999999999998,"15min":0.90200000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.5300000000000002,"5min":4.5300000000000002,"15min":4.5300000000000002},"last":0.85599999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.997,"15min":0.96899999999999997},"min":{"1min":0.59499999999999997,"5min":0.55000000000000004,"15min":0.51100000000000001},"max":{"1min":5.109,"5min":5.109,"15min":5.3710000000000004},"last":1.321},{"interface":"front","average":{"1min":1.361,"5min":1.1639999999999999,"15min":1.0780000000000001},"min":{"1min":0.70099999999999996,"5min":0.63800000000000001,"15min":0.53200000000000003},"max":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":1.2130000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1200000000000001,"15min":1.0760000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.2709999999999999,"5min":5.2709999999999999,"15min":5.476},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.014,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.6950000000000003,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.78300000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.0009999999999999,"15min":1.004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":5.0229999999999997,"5min":5.0229999999999997,"15min":5.0229999999999997},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.319,"5min":1.109,"15min":1.123},"min":{"1min":0.752,"5min":0.49199999999999999,"15min":0.41799999999999998},"max":{"1min":4.843,"5min":4.843,"15min":4.843},"last":1.1579999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2250000000000001,"15min":1.1930000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.6029999999999998},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.1819999999999999,"15min":1.1879999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41899999999999998},"max":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":1.1180000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":1.24,"15min":1.2629999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.45100000000000001},"max":{"1min":10.884,"5min":10.884,"15min":10.884},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.272,"15min":1.3},"min":{"1min":0.5,"5min":0.5,"15min":0.48499999999999999},"max":{"1min":5.335,"5min":5.335,"15min":5.335},"last":1.2829999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426512,"kb_used_data":337032,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93945328,"statfs":{"total":96636764160,"available":96200015872,"internally_reserved":0,"allocated":345120768,"data_stored":336789947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.872},"min":{"1min":0.45000000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.3159999999999998,"5min":8.7680000000000007,"15min":8.7680000000000007},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.329,"5min":0.999,"15min":0.89600000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":10.52,"5min":10.52,"15min":10.52},"last":1}]},{"osd":1,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.82299999999999995,"15min":0.88900000000000001},"min":{"1min":0.51500000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":1.236,"15min":5.8010000000000002},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.016,"5min":0.86499999999999999,"15min":0.874},"min":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.85899999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.55400000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":3.323},"last":1.091},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0329999999999999,"15min":0.98199999999999998},"min":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.0549999999999997,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":1.069}]},{"osd":3,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.105,"15min":1.0209999999999999},"min":{"1min":0.69299999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.268,"5min":8.0570000000000004,"15min":8.0570000000000004},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.0529999999999999,"15min":0.995},"min":{"1min":0.66800000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1459999999999999,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":1.286}]},{"osd":5,"last update":"Mon Oct 30 13:59:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":1.1599999999999999,"15min":1.103},"min":{"1min":0.54200000000000004,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":12.798,"5min":12.798,"15min":12.798},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.1459999999999999,"15min":1.169},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5229999999999999,"5min":4.0609999999999999,"15min":8.5399999999999991},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.2010000000000001,"15min":1.0900000000000001},"min":{"1min":0.56000000000000005,"5min":0.503,"15min":0.45100000000000001},"max":{"1min":1.3680000000000001,"5min":16.989999999999998,"15min":16.989999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1890000000000001,"15min":1.1599999999999999},"min":{"1min":0.81599999999999995,"5min":0.41299999999999998,"15min":0.36699999999999999},"max":{"1min":1.4690000000000001,"5min":4.4530000000000003,"15min":4.4530000000000003},"last":1.1879999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:59:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.194,"15min":1.1739999999999999},"min":{"1min":0.69999999999999996,"5min":0.45100000000000001,"15min":0.42199999999999999},"max":{"1min":1.6799999999999999,"5min":2.3220000000000001,"15min":3.4510000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.3109999999999999,"15min":1.3120000000000001},"min":{"1min":0.754,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9259999999999999,"5min":2.3570000000000002,"15min":3.5470000000000002},"last":1.24}]}]},{"osd":0,"up_from":14,"seq":60129542448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449200,"kb_used_data":368040,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93922640,"statfs":{"total":96636764160,"available":96176783360,"internally_reserved":0,"allocated":376872960,"data_stored":368821573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.99199999999999999,"15min":0.64000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.63700000000000001,"5min":45.311,"15min":45.311},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.52200000000000002,"15min":0.59499999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27200000000000002},"max":{"1min":0.79400000000000004,"5min":2.2410000000000001,"15min":27.616},"last":0.71699999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.29099999999999998},"max":{"1min":0.70799999999999996,"5min":0.88100000000000001,"15min":2.5449999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.61499999999999999,"15min":0.60299999999999998},"min":{"1min":0.442,"5min":0.40899999999999997,"15min":0.35199999999999998},"max":{"1min":0.74299999999999999,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.78400000000000003}]},{"osd":3,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.67400000000000004,"15min":0.56000000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.29299999999999998},"max":{"1min":0.77100000000000002,"5min":12.368,"15min":12.368},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.75700000000000001,"15min":0.67200000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82199999999999995,"5min":12.442,"15min":12.442},"last":0.82999999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.59599999999999997,"5min":0.54000000000000004,"15min":0.41599999999999998},"max":{"1min":1.196,"5min":2.6949999999999998,"15min":9.4979999999999993},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.86199999999999999,"15min":0.86699999999999999},"min":{"1min":0.65700000000000003,"5min":0.60199999999999998,"15min":0.46700000000000003},"max":{"1min":1.1699999999999999,"5min":2.6440000000000001,"15min":9.5429999999999993},"last":1.036}]},{"osd":5,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89100000000000001,"15min":0.88500000000000001},"min":{"1min":0.74399999999999999,"5min":0.47099999999999997,"15min":0.317},"max":{"1min":1.1339999999999999,"5min":1.1359999999999999,"15min":11.026},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.3700000000000001,"15min":1.0289999999999999},"min":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.46999999999999997},"max":{"1min":1.1120000000000001,"5min":44.225999999999999,"15min":44.225999999999999},"last":0.93500000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.89000000000000001,"15min":0.878},"min":{"1min":0.68200000000000005,"5min":0.63,"15min":0.48799999999999999},"max":{"1min":1.234,"5min":2.5169999999999999,"15min":2.9300000000000002},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.86499999999999999,"15min":0.879},"min":{"1min":0.64000000000000001,"5min":0.60499999999999998,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":1.236,"15min":11.087999999999999},"last":1.008}]},{"osd":7,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.91100000000000003,"15min":0.877},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.50900000000000001},"max":{"1min":1.2529999999999999,"5min":5.6849999999999996,"15min":5.6849999999999996},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.92400000000000004,"15min":0.93100000000000005},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.57199999999999995},"max":{"1min":1.0860000000000001,"5min":2.6030000000000002,"15min":9.5890000000000004},"last":1.2230000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50573312,"data_stored":50555931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46346240,"data_stored":46320604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75579392,"data_stored":75553888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":71393280,"data_stored":71379650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67129344,"data_stored":67110274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":75464704,"data_stored":75447768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":67350528,"data_stored":67333247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325066752,"data_stored":317352305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183717888,"data_stored":178743923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144601088,"data_stored":139970730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":249585664,"data_stored":241694096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":273088512,"data_stored":265087328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":275906560,"data_stored":268118820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":177188864,"data_stored":172364192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":273362944,"data_stored":265498878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:59:52.354 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.354 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.355 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.355 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.355 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.356 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.357 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.357 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.357 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:35:07.244846+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=59, tm_sec=5, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:59:52.358 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:59:59.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:59.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:12.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:00:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.511+0000 7fdae310e640 1 Processor -- start 2023-10-30T14:00:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae310e640 1 -- start start 2023-10-30T14:00:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae310e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fdadc153c80 0x7fdadc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae310e640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdadc159f20 con 0x7fdadc0c42a0 2023-10-30T14:00:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae310e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdadc15d620 con 0x7fdadc0c4a10 2023-10-30T14:00:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae310e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdadc15dec0 con 0x7fdadc153c80 2023-10-30T14:00:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae0eaa640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fdadc153c80 0x7fdadc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae0eaa640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fdadc153c80 0x7fdadc15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52500/0 (socket says 172.21.15.31:52500) 2023-10-30T14:00:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae0eaa640 1 -- 172.21.15.31:0/54921874 learned_addr learned my addr 172.21.15.31:0/54921874 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae16ab640 1 --1- 172.21.15.31:0/54921874 >> v1:172.21.15.31:6789/0 conn(0x7fdadc0c42a0 0x7fdadc0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54080/0 (socket says 172.21.15.31:54080) 2023-10-30T14:00:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae0eaa640 1 -- 172.21.15.31:0/54921874 >> v1:172.21.15.33:6789/0 conn(0x7fdadc0c4a10 legacy=0x7fdadc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae0eaa640 1 -- 172.21.15.31:0/54921874 >> v1:172.21.15.31:6789/0 conn(0x7fdadc0c42a0 legacy=0x7fdadc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdae0eaa640 1 -- 172.21.15.31:0/54921874 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc15d900 con 0x7fdadc153c80 2023-10-30T14:00:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.515+0000 7fdadb7fe640 1 -- 172.21.15.31:0/54921874 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410737050 0 0) 0x7fdadc159f20 con 0x7fdadc0c42a0 2023-10-30T14:00:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae0eaa640 1 --2- 172.21.15.31:0/54921874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc153c80 0x7fdadc15cf90 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fdad0009870 tx=0x7fdad001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f74603bc4e43fea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdadb7fe640 1 -- 172.21.15.31:0/54921874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad00a89f0 con 0x7fdadc153c80 2023-10-30T14:00:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdadb7fe640 1 -- 172.21.15.31:0/54921874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad00a8b50 con 0x7fdadc153c80 2023-10-30T14:00:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdadb7fe640 1 -- 172.21.15.31:0/54921874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad00b0440 con 0x7fdadc153c80 2023-10-30T14:00:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 -- 172.21.15.31:0/54921874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc153c80 msgr2=0x7fdadc15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 --2- 172.21.15.31:0/54921874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc153c80 0x7fdadc15cf90 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fdad0009870 tx=0x7fdad001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 -- 172.21.15.31:0/54921874 shutdown_connections 2023-10-30T14:00:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 --2- 172.21.15.31:0/54921874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc153c80 0x7fdadc15cf90 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 -- 172.21.15.31:0/54921874 >> 172.21.15.31:0/54921874 conn(0x7fdadc0715c0 msgr2=0x7fdadc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 -- 172.21.15.31:0/54921874 shutdown_connections 2023-10-30T14:00:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 -- 172.21.15.31:0/54921874 wait complete. 2023-10-30T14:00:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 Processor -- start 2023-10-30T14:00:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.519+0000 7fdae310e640 1 -- start start 2023-10-30T14:00:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc0c42a0 0x7fdadc0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc0c4a10 0x7fdadc0c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdadc153c80 0x7fdadc1f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc1f00f0 con 0x7fdadc0c42a0 2023-10-30T14:00:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc1f0990 con 0x7fdadc0c4a10 2023-10-30T14:00:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc0b2d20 con 0x7fdadc153c80 2023-10-30T14:00:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc0c42a0 0x7fdadc0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc0c42a0 0x7fdadc0cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52504/0 (socket says 172.21.15.31:52504) 2023-10-30T14:00:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 -- 172.21.15.31:0/1644313393 learned_addr learned my addr 172.21.15.31:0/1644313393 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae16ab640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdadc153c80 0x7fdadc1f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdadbfff640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc0c4a10 0x7fdadc0c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 -- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdadc153c80 msgr2=0x7fdadc1f0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdadc153c80 0x7fdadc1f0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 -- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc0c4a10 msgr2=0x7fdadc0c5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc0c4a10 0x7fdadc0c5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 -- 172.21.15.31:0/1644313393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad00094d0 con 0x7fdadc0c42a0 2023-10-30T14:00:12.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae16ab640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdadc153c80 0x7fdadc1f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:00:12.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae0eaa640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc0c42a0 0x7fdadc0cda50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fdad00b3040 tx=0x7fdad001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:12.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdad97fa640 1 -- 172.21.15.31:0/1644313393 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad00b0e40 con 0x7fdadc0c42a0 2023-10-30T14:00:12.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdad97fa640 1 -- 172.21.15.31:0/1644313393 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad00bb470 con 0x7fdadc0c42a0 2023-10-30T14:00:12.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc0ced50 con 0x7fdadc0c42a0 2023-10-30T14:00:12.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdad97fa640 1 -- 172.21.15.31:0/1644313393 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad00bb820 con 0x7fdadc0c42a0 2023-10-30T14:00:12.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdadc1f0340 con 0x7fdadc0c42a0 2023-10-30T14:00:12.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.523+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdadc04e170 con 0x7fdadc0c42a0 2023-10-30T14:00:12.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.527+0000 7fdad97fa640 1 -- 172.21.15.31:0/1644313393 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdad00b7070 con 0x7fdadc0c42a0 2023-10-30T14:00:12.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.527+0000 7fdad97fa640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdac00664a0 0x7fdac0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.527+0000 7fdad97fa640 1 -- 172.21.15.31:0/1644313393 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fdad0122700 con 0x7fdadc0c42a0 2023-10-30T14:00:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.527+0000 7fdadbfff640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdac00664a0 0x7fdac0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.527+0000 7fdadbfff640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdac00664a0 0x7fdac0068960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdadc153580 tx=0x7fdac8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.531+0000 7fdad97fa640 1 -- 172.21.15.31:0/1644313393 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdad00ef6b0 con 0x7fdadc0c42a0 2023-10-30T14:00:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.683+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 --> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdadc159bb0 con 0x7fdac00664a0 2023-10-30T14:00:12.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.707+0000 7fdad97fa640 1 -- 172.21.15.31:0/1644313393 <== mgr.4099 v2:172.21.15.33:6832/2139658642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294704 (secure 0 0 0) 0x7fdac8005380 con 0x7fdac00664a0 2023-10-30T14:00:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T14:00:12.715 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-30T14:00:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdac00664a0 msgr2=0x7fdac0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdac00664a0 0x7fdac0068960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdadc153580 tx=0x7fdac8009040 comp rx=0 tx=0).stop 2023-10-30T14:00:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc0c42a0 msgr2=0x7fdadc0cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc0c42a0 0x7fdadc0cda50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fdad00b3040 tx=0x7fdad001ba90 comp rx=0 tx=0).stop 2023-10-30T14:00:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 shutdown_connections 2023-10-30T14:00:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fdac00664a0 0x7fdac0068960 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fdadc153c80 0x7fdadc1f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc0c4a10 0x7fdadc0c5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 --2- 172.21.15.31:0/1644313393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdadc0c42a0 0x7fdadc0cda50 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 >> 172.21.15.31:0/1644313393 conn(0x7fdadc0715c0 msgr2=0x7fdadc0cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:12.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 shutdown_connections 2023-10-30T14:00:12.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.711+0000 7fdae310e640 1 -- 172.21.15.31:0/1644313393 wait complete. 2023-10-30T14:00:12.733 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":808,"stamp":"2023-10-30T14:00:11.739770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186202618,"num_objects":10532,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10532,"num_whiteouts":0,"num_read":8937,"num_read_kb":1843152,"num_write":58261,"num_write_kb":2330376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":782235,"num_omap_keys":1350,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70982,"ondisk_log_size":70982,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3116312,"kb_used_data":2372888,"kb_used_omap":12,"kb_used_meta":743347,"kb_avail":751858408,"statfs":{"total":773094113280,"available":769903009792,"internally_reserved":0,"allocated":2429837312,"data_stored":2373448892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":761187896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004588"},"pg_stats":[{"pgid":"3.3f","version":"56'822","reported_seq":3056,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:53.896168+0000","last_change":"2023-10-30T13:59:53.896168+0000","last_active":"2023-10-30T13:59:53.896168+0000","last_peered":"2023-10-30T13:59:53.896168+0000","last_clean":"2023-10-30T13:59:53.896168+0000","last_became_active":"2023-10-30T13:35:07.791023+0000","last_became_peered":"2023-10-30T13:35:07.791023+0000","last_unstale":"2023-10-30T13:59:53.896168+0000","last_undegraded":"2023-10-30T13:59:53.896168+0000","last_fullsized":"2023-10-30T13:59:53.896168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'822","last_scrub_stamp":"2023-10-30T13:59:53.895920+0000","last_deep_scrub":"56'822","last_deep_scrub_stamp":"2023-10-30T13:59:53.895920+0000","last_clean_scrub_stamp":"2023-10-30T13:59:53.895920+0000","objects_scrubbed":148,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:09.072914+0000","scrub_duration":0.345961505,"objects_trimmed":0,"snaptrim_duration":0.0098613959999999997,"stat_sum":{"num_bytes":30553715,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":124,"num_read_kb":8207,"num_write":638,"num_write_kb":39180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"56'818","reported_seq":3033,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:33.959488+0000","last_change":"2023-10-30T13:59:33.959488+0000","last_active":"2023-10-30T13:59:33.959488+0000","last_peered":"2023-10-30T13:59:33.959488+0000","last_clean":"2023-10-30T13:59:33.959488+0000","last_became_active":"2023-10-30T13:35:07.754882+0000","last_became_peered":"2023-10-30T13:35:07.754882+0000","last_unstale":"2023-10-30T13:59:33.959488+0000","last_undegraded":"2023-10-30T13:59:33.959488+0000","last_fullsized":"2023-10-30T13:59:33.959488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'818","last_scrub_stamp":"2023-10-30T13:59:33.959361+0000","last_deep_scrub":"56'818","last_deep_scrub_stamp":"2023-10-30T13:59:33.959361+0000","last_clean_scrub_stamp":"2023-10-30T13:59:33.959361+0000","objects_scrubbed":161,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:47.001349+0000","scrub_duration":0.182073817,"objects_trimmed":0,"snaptrim_duration":0.001955118,"stat_sum":{"num_bytes":9994664,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":104,"num_read_kb":11,"num_write":617,"num_write_kb":23604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"56'815","reported_seq":3033,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:22.708475+0000","last_change":"2023-10-30T13:59:22.708475+0000","last_active":"2023-10-30T13:59:22.708475+0000","last_peered":"2023-10-30T13:59:22.708475+0000","last_clean":"2023-10-30T13:59:22.708475+0000","last_became_active":"2023-10-30T13:35:07.753974+0000","last_became_peered":"2023-10-30T13:35:07.753974+0000","last_unstale":"2023-10-30T13:59:22.708475+0000","last_undegraded":"2023-10-30T13:59:22.708475+0000","last_fullsized":"2023-10-30T13:59:22.708475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'815","last_scrub_stamp":"2023-10-30T13:59:22.708353+0000","last_deep_scrub":"56'815","last_deep_scrub_stamp":"2023-10-30T13:59:22.708353+0000","last_clean_scrub_stamp":"2023-10-30T13:59:22.708353+0000","objects_scrubbed":162,"log_size":815,"log_dups_size":0,"ondisk_log_size":815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:43:36.533585+0000","scrub_duration":0.169844142,"objects_trimmed":0,"snaptrim_duration":0.0021120940000000001,"stat_sum":{"num_bytes":23390266,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":110,"num_read_kb":20497,"num_write":638,"num_write_kb":47001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"56'828","reported_seq":3069,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.387579+0000","last_change":"2023-10-30T13:59:24.387627+0000","last_active":"2023-10-30T13:59:24.387579+0000","last_peered":"2023-10-30T13:59:24.387579+0000","last_clean":"2023-10-30T13:59:24.387579+0000","last_became_active":"2023-10-30T13:35:48.779909+0000","last_became_peered":"2023-10-30T13:35:48.779909+0000","last_unstale":"2023-10-30T13:59:24.387579+0000","last_undegraded":"2023-10-30T13:59:24.387579+0000","last_fullsized":"2023-10-30T13:59:24.387579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'828","last_scrub_stamp":"2023-10-30T13:59:24.387502+0000","last_deep_scrub":"56'828","last_deep_scrub_stamp":"2023-10-30T13:59:24.387502+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.387502+0000","objects_scrubbed":159,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:33.212806+0000","scrub_duration":0.154922691,"objects_trimmed":0,"snaptrim_duration":0.001317775,"stat_sum":{"num_bytes":10165225,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":133,"num_read_kb":8205,"num_write":656,"num_write_kb":40955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"56'867","reported_seq":3086,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:50.901630+0000","last_change":"2023-10-30T13:59:50.901630+0000","last_active":"2023-10-30T13:59:50.901630+0000","last_peered":"2023-10-30T13:59:50.901630+0000","last_clean":"2023-10-30T13:59:50.901630+0000","last_became_active":"2023-10-30T13:35:07.792125+0000","last_became_peered":"2023-10-30T13:35:07.792125+0000","last_unstale":"2023-10-30T13:59:50.901630+0000","last_undegraded":"2023-10-30T13:59:50.901630+0000","last_fullsized":"2023-10-30T13:59:50.901630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'867","last_scrub_stamp":"2023-10-30T13:59:50.901304+0000","last_deep_scrub":"56'867","last_deep_scrub_stamp":"2023-10-30T13:59:50.901304+0000","last_clean_scrub_stamp":"2023-10-30T13:59:50.901304+0000","objects_scrubbed":166,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:51.378945+0000","scrub_duration":0.352873096,"objects_trimmed":0,"snaptrim_duration":0.007614844,"stat_sum":{"num_bytes":19907850,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":110,"num_read_kb":8208,"num_write":664,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"56'828","reported_seq":3041,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:49.985553+0000","last_change":"2023-10-30T13:59:49.985553+0000","last_active":"2023-10-30T13:59:49.985553+0000","last_peered":"2023-10-30T13:59:49.985553+0000","last_clean":"2023-10-30T13:59:49.985553+0000","last_became_active":"2023-10-30T13:35:07.791271+0000","last_became_peered":"2023-10-30T13:35:07.791271+0000","last_unstale":"2023-10-30T13:59:49.985553+0000","last_undegraded":"2023-10-30T13:59:49.985553+0000","last_fullsized":"2023-10-30T13:59:49.985553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'828","last_scrub_stamp":"2023-10-30T13:59:49.985412+0000","last_deep_scrub":"56'828","last_deep_scrub_stamp":"2023-10-30T13:59:49.985412+0000","last_clean_scrub_stamp":"2023-10-30T13:59:49.985412+0000","objects_scrubbed":162,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:27:04.740039+0000","scrub_duration":0.41252158300000002,"objects_trimmed":0,"snaptrim_duration":0.0030849649999999998,"stat_sum":{"num_bytes":19227960,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":104,"num_read_kb":7536,"num_write":633,"num_write_kb":20026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"56'856","reported_seq":3056,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:34.802759+0000","last_change":"2023-10-30T13:59:34.802759+0000","last_active":"2023-10-30T13:59:34.802759+0000","last_peered":"2023-10-30T13:59:34.802759+0000","last_clean":"2023-10-30T13:59:34.802759+0000","last_became_active":"2023-10-30T13:35:07.775536+0000","last_became_peered":"2023-10-30T13:35:07.775536+0000","last_unstale":"2023-10-30T13:59:34.802759+0000","last_undegraded":"2023-10-30T13:59:34.802759+0000","last_fullsized":"2023-10-30T13:59:34.802759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'856","last_scrub_stamp":"2023-10-30T13:59:34.802404+0000","last_deep_scrub":"56'856","last_deep_scrub_stamp":"2023-10-30T13:59:34.802404+0000","last_clean_scrub_stamp":"2023-10-30T13:59:34.802404+0000","objects_scrubbed":161,"log_size":856,"log_dups_size":0,"ondisk_log_size":856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:57.808507+0000","scrub_duration":0.29249388500000001,"objects_trimmed":0,"snaptrim_duration":0.0082986029999999999,"stat_sum":{"num_bytes":10009773,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":91,"num_read_kb":11,"num_write":659,"num_write_kb":29217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"56'828","reported_seq":3069,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:41.385188+0000","last_change":"2023-10-30T13:59:41.385188+0000","last_active":"2023-10-30T13:59:41.385188+0000","last_peered":"2023-10-30T13:59:41.385188+0000","last_clean":"2023-10-30T13:59:41.385188+0000","last_became_active":"2023-10-30T13:35:07.789970+0000","last_became_peered":"2023-10-30T13:35:07.789970+0000","last_unstale":"2023-10-30T13:59:41.385188+0000","last_undegraded":"2023-10-30T13:59:41.385188+0000","last_fullsized":"2023-10-30T13:59:41.385188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'828","last_scrub_stamp":"2023-10-30T13:59:41.385065+0000","last_deep_scrub":"56'828","last_deep_scrub_stamp":"2023-10-30T13:59:41.385065+0000","last_clean_scrub_stamp":"2023-10-30T13:59:41.385065+0000","objects_scrubbed":158,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:32.113907+0000","scrub_duration":0.28254902599999998,"objects_trimmed":0,"snaptrim_duration":0.0021728070000000001,"stat_sum":{"num_bytes":10221735,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":132,"num_read_kb":1147,"num_write":641,"num_write_kb":24778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"56'838","reported_seq":3070,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:56.867490+0000","last_change":"2023-10-30T13:59:56.867579+0000","last_active":"2023-10-30T13:59:56.867490+0000","last_peered":"2023-10-30T13:59:56.867490+0000","last_clean":"2023-10-30T13:59:56.867490+0000","last_became_active":"2023-10-30T13:35:07.803307+0000","last_became_peered":"2023-10-30T13:35:07.803307+0000","last_unstale":"2023-10-30T13:59:56.867490+0000","last_undegraded":"2023-10-30T13:59:56.867490+0000","last_fullsized":"2023-10-30T13:59:56.867490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'838","last_scrub_stamp":"2023-10-30T13:59:56.867381+0000","last_deep_scrub":"56'838","last_deep_scrub_stamp":"2023-10-30T13:59:56.867381+0000","last_clean_scrub_stamp":"2023-10-30T13:59:56.867381+0000","objects_scrubbed":172,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.064080+0000","scrub_duration":0.30969610400000003,"objects_trimmed":0,"snaptrim_duration":0.010045174,"stat_sum":{"num_bytes":10718551,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":124,"num_read_kb":4115,"num_write":615,"num_write_kb":24187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"56'778","reported_seq":3000,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:43.984031+0000","last_change":"2023-10-30T13:59:43.984081+0000","last_active":"2023-10-30T13:59:43.984031+0000","last_peered":"2023-10-30T13:59:43.984031+0000","last_clean":"2023-10-30T13:59:43.984031+0000","last_became_active":"2023-10-30T13:35:07.800079+0000","last_became_peered":"2023-10-30T13:35:07.800079+0000","last_unstale":"2023-10-30T13:59:43.984031+0000","last_undegraded":"2023-10-30T13:59:43.984031+0000","last_fullsized":"2023-10-30T13:59:43.984031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'778","last_scrub_stamp":"2023-10-30T13:59:43.983948+0000","last_deep_scrub":"56'778","last_deep_scrub_stamp":"2023-10-30T13:59:43.983948+0000","last_clean_scrub_stamp":"2023-10-30T13:59:43.983948+0000","objects_scrubbed":161,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:21.972997+0000","scrub_duration":0.23974774600000001,"objects_trimmed":0,"snaptrim_duration":0.0047277760000000004,"stat_sum":{"num_bytes":20230336,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":113,"num_read_kb":28,"num_write":585,"num_write_kb":25967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"56'781","reported_seq":2987,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:46.984535+0000","last_change":"2023-10-30T13:59:46.984590+0000","last_active":"2023-10-30T13:59:46.984535+0000","last_peered":"2023-10-30T13:59:46.984535+0000","last_clean":"2023-10-30T13:59:46.984535+0000","last_became_active":"2023-10-30T13:35:07.801577+0000","last_became_peered":"2023-10-30T13:35:07.801577+0000","last_unstale":"2023-10-30T13:59:46.984535+0000","last_undegraded":"2023-10-30T13:59:46.984535+0000","last_fullsized":"2023-10-30T13:59:46.984535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'781","last_scrub_stamp":"2023-10-30T13:59:46.984450+0000","last_deep_scrub":"56'781","last_deep_scrub_stamp":"2023-10-30T13:59:46.984450+0000","last_clean_scrub_stamp":"2023-10-30T13:59:46.984450+0000","objects_scrubbed":169,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:12.021790+0000","scrub_duration":0.21481325800000001,"objects_trimmed":0,"snaptrim_duration":0.00429868,"stat_sum":{"num_bytes":2384023,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":97,"num_read_kb":17,"num_write":586,"num_write_kb":9055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"56'789","reported_seq":3015,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.827760+0000","last_change":"2023-10-30T13:59:19.827838+0000","last_active":"2023-10-30T13:59:19.827760+0000","last_peered":"2023-10-30T13:59:19.827760+0000","last_clean":"2023-10-30T13:59:19.827760+0000","last_became_active":"2023-10-30T13:35:07.782879+0000","last_became_peered":"2023-10-30T13:35:07.782879+0000","last_unstale":"2023-10-30T13:59:19.827760+0000","last_undegraded":"2023-10-30T13:59:19.827760+0000","last_fullsized":"2023-10-30T13:59:19.827760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'789","last_scrub_stamp":"2023-10-30T13:59:19.827650+0000","last_deep_scrub":"56'789","last_deep_scrub_stamp":"2023-10-30T13:59:19.827650+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.827650+0000","objects_scrubbed":175,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:07.295597+0000","scrub_duration":0.27225982599999998,"objects_trimmed":0,"snaptrim_duration":0.0066533010000000004,"stat_sum":{"num_bytes":18341132,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":116,"num_read_kb":8208,"num_write":596,"num_write_kb":32418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"56'800","reported_seq":3022,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:56.325396+0000","last_change":"2023-10-30T13:59:56.325437+0000","last_active":"2023-10-30T13:59:56.325396+0000","last_peered":"2023-10-30T13:59:56.325396+0000","last_clean":"2023-10-30T13:59:56.325396+0000","last_became_active":"2023-10-30T13:35:07.779909+0000","last_became_peered":"2023-10-30T13:35:07.779909+0000","last_unstale":"2023-10-30T13:59:56.325396+0000","last_undegraded":"2023-10-30T13:59:56.325396+0000","last_fullsized":"2023-10-30T13:59:56.325396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'800","last_scrub_stamp":"2023-10-30T13:59:56.325317+0000","last_deep_scrub":"56'800","last_deep_scrub_stamp":"2023-10-30T13:59:56.325317+0000","last_clean_scrub_stamp":"2023-10-30T13:59:56.325317+0000","objects_scrubbed":152,"log_size":800,"log_dups_size":0,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:13.442110+0000","scrub_duration":0.267080225,"objects_trimmed":0,"snaptrim_duration":0.002338237,"stat_sum":{"num_bytes":14177749,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":114,"num_read_kb":12311,"num_write":601,"num_write_kb":37081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"56'871","reported_seq":3099,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.750623+0000","last_change":"2023-10-30T13:59:24.750679+0000","last_active":"2023-10-30T13:59:24.750623+0000","last_peered":"2023-10-30T13:59:24.750623+0000","last_clean":"2023-10-30T13:59:24.750623+0000","last_became_active":"2023-10-30T13:35:07.750749+0000","last_became_peered":"2023-10-30T13:35:07.750749+0000","last_unstale":"2023-10-30T13:59:24.750623+0000","last_undegraded":"2023-10-30T13:59:24.750623+0000","last_fullsized":"2023-10-30T13:59:24.750623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'871","last_scrub_stamp":"2023-10-30T13:59:24.750548+0000","last_deep_scrub":"56'871","last_deep_scrub_stamp":"2023-10-30T13:59:24.750548+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.750548+0000","objects_scrubbed":167,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:44.353295+0000","scrub_duration":0.19578169600000001,"objects_trimmed":0,"snaptrim_duration":0.0029248669999999998,"stat_sum":{"num_bytes":22804734,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":119,"num_read_kb":24041,"num_write":691,"num_write_kb":57342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"56'875","reported_seq":3096,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:39.045087+0000","last_change":"2023-10-30T13:59:39.045134+0000","last_active":"2023-10-30T13:59:39.045087+0000","last_peered":"2023-10-30T13:59:39.045087+0000","last_clean":"2023-10-30T13:59:39.045087+0000","last_became_active":"2023-10-30T13:35:07.767933+0000","last_became_peered":"2023-10-30T13:35:07.767933+0000","last_unstale":"2023-10-30T13:59:39.045087+0000","last_undegraded":"2023-10-30T13:59:39.045087+0000","last_fullsized":"2023-10-30T13:59:39.045087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'875","last_scrub_stamp":"2023-10-30T13:59:39.045006+0000","last_deep_scrub":"56'875","last_deep_scrub_stamp":"2023-10-30T13:59:39.045006+0000","last_clean_scrub_stamp":"2023-10-30T13:59:39.045006+0000","objects_scrubbed":159,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:49:37.890024+0000","scrub_duration":0.291907531,"objects_trimmed":0,"snaptrim_duration":0.0021050410000000002,"stat_sum":{"num_bytes":18271578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":112,"num_read_kb":7334,"num_write":676,"num_write_kb":39770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"56'884","reported_seq":3117,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:31.971189+0000","last_change":"2023-10-30T13:59:31.971276+0000","last_active":"2023-10-30T13:59:31.971189+0000","last_peered":"2023-10-30T13:59:31.971189+0000","last_clean":"2023-10-30T13:59:31.971189+0000","last_became_active":"2023-10-30T13:35:07.788520+0000","last_became_peered":"2023-10-30T13:35:07.788520+0000","last_unstale":"2023-10-30T13:59:31.971189+0000","last_undegraded":"2023-10-30T13:59:31.971189+0000","last_fullsized":"2023-10-30T13:59:31.971189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'884","last_scrub_stamp":"2023-10-30T13:59:31.971052+0000","last_deep_scrub":"56'884","last_deep_scrub_stamp":"2023-10-30T13:59:31.971052+0000","last_clean_scrub_stamp":"2023-10-30T13:59:31.971052+0000","objects_scrubbed":181,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:49:06.332092+0000","scrub_duration":0.37008379400000002,"objects_trimmed":0,"snaptrim_duration":0.0052579080000000004,"stat_sum":{"num_bytes":35259912,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":123,"num_read_kb":12306,"num_write":693,"num_write_kb":43349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"56'832","reported_seq":3043,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:29.855984+0000","last_change":"2023-10-30T13:59:29.856089+0000","last_active":"2023-10-30T13:59:29.855984+0000","last_peered":"2023-10-30T13:59:29.855984+0000","last_clean":"2023-10-30T13:59:29.855984+0000","last_became_active":"2023-10-30T13:35:07.795836+0000","last_became_peered":"2023-10-30T13:35:07.795836+0000","last_unstale":"2023-10-30T13:59:29.855984+0000","last_undegraded":"2023-10-30T13:59:29.855984+0000","last_fullsized":"2023-10-30T13:59:29.855984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'832","last_scrub_stamp":"2023-10-30T13:59:29.855862+0000","last_deep_scrub":"56'832","last_deep_scrub_stamp":"2023-10-30T13:59:29.855862+0000","last_clean_scrub_stamp":"2023-10-30T13:59:29.855862+0000","objects_scrubbed":154,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:19.015619+0000","scrub_duration":0.27485262900000001,"objects_trimmed":0,"snaptrim_duration":0.0052923609999999998,"stat_sum":{"num_bytes":14461920,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":103,"num_read_kb":4113,"num_write":659,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"56'763","reported_seq":2989,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.418430+0000","last_change":"2023-10-30T13:59:17.418479+0000","last_active":"2023-10-30T13:59:17.418430+0000","last_peered":"2023-10-30T13:59:17.418430+0000","last_clean":"2023-10-30T13:59:17.418430+0000","last_became_active":"2023-10-30T13:35:07.779794+0000","last_became_peered":"2023-10-30T13:35:07.779794+0000","last_unstale":"2023-10-30T13:59:17.418430+0000","last_undegraded":"2023-10-30T13:59:17.418430+0000","last_fullsized":"2023-10-30T13:59:17.418430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'763","last_scrub_stamp":"2023-10-30T13:59:17.418359+0000","last_deep_scrub":"56'763","last_deep_scrub_stamp":"2023-10-30T13:59:17.418359+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.418359+0000","objects_scrubbed":142,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:00.183579+0000","scrub_duration":0.20971843000000001,"objects_trimmed":0,"snaptrim_duration":0.001369274,"stat_sum":{"num_bytes":15414431,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":118,"num_read_kb":4123,"num_write":623,"num_write_kb":31369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"56'907","reported_seq":3151,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:44.853755+0000","last_change":"2023-10-30T13:59:44.853755+0000","last_active":"2023-10-30T13:59:44.853755+0000","last_peered":"2023-10-30T13:59:44.853755+0000","last_clean":"2023-10-30T13:59:44.853755+0000","last_became_active":"2023-10-30T13:35:07.790156+0000","last_became_peered":"2023-10-30T13:35:07.790156+0000","last_unstale":"2023-10-30T13:59:44.853755+0000","last_undegraded":"2023-10-30T13:59:44.853755+0000","last_fullsized":"2023-10-30T13:59:44.853755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'907","last_scrub_stamp":"2023-10-30T13:59:44.853571+0000","last_deep_scrub":"56'907","last_deep_scrub_stamp":"2023-10-30T13:59:44.853571+0000","last_clean_scrub_stamp":"2023-10-30T13:59:44.853571+0000","objects_scrubbed":179,"log_size":907,"log_dups_size":0,"ondisk_log_size":907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:17.124766+0000","scrub_duration":0.34510061199999997,"objects_trimmed":0,"snaptrim_duration":0.0092525119999999992,"stat_sum":{"num_bytes":8879597,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":134,"num_read_kb":8209,"num_write":693,"num_write_kb":31072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"56'821","reported_seq":3030,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:38.050201+0000","last_change":"2023-10-30T13:59:38.050201+0000","last_active":"2023-10-30T13:59:38.050201+0000","last_peered":"2023-10-30T13:59:38.050201+0000","last_clean":"2023-10-30T13:59:38.050201+0000","last_became_active":"2023-10-30T13:35:07.781349+0000","last_became_peered":"2023-10-30T13:35:07.781349+0000","last_unstale":"2023-10-30T13:59:38.050201+0000","last_undegraded":"2023-10-30T13:59:38.050201+0000","last_fullsized":"2023-10-30T13:59:38.050201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'821","last_scrub_stamp":"2023-10-30T13:59:38.049998+0000","last_deep_scrub":"56'821","last_deep_scrub_stamp":"2023-10-30T13:59:38.049998+0000","last_clean_scrub_stamp":"2023-10-30T13:59:38.049998+0000","objects_scrubbed":154,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:50:58.830199+0000","scrub_duration":0.30212535800000001,"objects_trimmed":0,"snaptrim_duration":0.00502631,"stat_sum":{"num_bytes":6211385,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":100,"num_read_kb":8209,"num_write":620,"num_write_kb":19966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"56'818","reported_seq":3039,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:29.620218+0000","last_change":"2023-10-30T13:59:29.620218+0000","last_active":"2023-10-30T13:59:29.620218+0000","last_peered":"2023-10-30T13:59:29.620218+0000","last_clean":"2023-10-30T13:59:29.620218+0000","last_became_active":"2023-10-30T13:35:07.794370+0000","last_became_peered":"2023-10-30T13:35:07.794370+0000","last_unstale":"2023-10-30T13:59:29.620218+0000","last_undegraded":"2023-10-30T13:59:29.620218+0000","last_fullsized":"2023-10-30T13:59:29.620218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'818","last_scrub_stamp":"2023-10-30T13:59:29.620101+0000","last_deep_scrub":"56'818","last_deep_scrub_stamp":"2023-10-30T13:59:29.620101+0000","last_clean_scrub_stamp":"2023-10-30T13:59:29.620101+0000","objects_scrubbed":152,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:31.441486+0000","scrub_duration":0.23420875999999999,"objects_trimmed":0,"snaptrim_duration":0.0026024469999999999,"stat_sum":{"num_bytes":18097476,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":111,"num_read_kb":21,"num_write":651,"num_write_kb":32402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"56'869","reported_seq":3084,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:48.924922+0000","last_change":"2023-10-30T13:59:48.924922+0000","last_active":"2023-10-30T13:59:48.924922+0000","last_peered":"2023-10-30T13:59:48.924922+0000","last_clean":"2023-10-30T13:59:48.924922+0000","last_became_active":"2023-10-30T13:35:07.796076+0000","last_became_peered":"2023-10-30T13:35:07.796076+0000","last_unstale":"2023-10-30T13:59:48.924922+0000","last_undegraded":"2023-10-30T13:59:48.924922+0000","last_fullsized":"2023-10-30T13:59:48.924922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'869","last_scrub_stamp":"2023-10-30T13:59:48.924732+0000","last_deep_scrub":"56'869","last_deep_scrub_stamp":"2023-10-30T13:59:48.924732+0000","last_clean_scrub_stamp":"2023-10-30T13:59:48.924732+0000","objects_scrubbed":164,"log_size":869,"log_dups_size":0,"ondisk_log_size":869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:48.609029+0000","scrub_duration":0.30822743600000002,"objects_trimmed":0,"snaptrim_duration":0.0054353270000000002,"stat_sum":{"num_bytes":10272429,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":105,"num_read_kb":4117,"num_write":665,"num_write_kb":16082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"56'847","reported_seq":3069,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:43.360180+0000","last_change":"2023-10-30T13:59:43.360180+0000","last_active":"2023-10-30T13:59:43.360180+0000","last_peered":"2023-10-30T13:59:43.360180+0000","last_clean":"2023-10-30T13:59:43.360180+0000","last_became_active":"2023-10-30T13:35:07.767567+0000","last_became_peered":"2023-10-30T13:35:07.767567+0000","last_unstale":"2023-10-30T13:59:43.360180+0000","last_undegraded":"2023-10-30T13:59:43.360180+0000","last_fullsized":"2023-10-30T13:59:43.360180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'847","last_scrub_stamp":"2023-10-30T13:59:43.360046+0000","last_deep_scrub":"56'847","last_deep_scrub_stamp":"2023-10-30T13:59:43.360046+0000","last_clean_scrub_stamp":"2023-10-30T13:59:43.360046+0000","objects_scrubbed":146,"log_size":847,"log_dups_size":0,"ondisk_log_size":847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:44.037800+0000","scrub_duration":0.25013997900000001,"objects_trimmed":0,"snaptrim_duration":0.0025617359999999998,"stat_sum":{"num_bytes":14333972,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":114,"num_read_kb":8213,"num_write":665,"num_write_kb":36470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"56'850","reported_seq":3086,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.738390+0000","last_change":"2023-10-30T13:59:24.738493+0000","last_active":"2023-10-30T13:59:24.738390+0000","last_peered":"2023-10-30T13:59:24.738390+0000","last_clean":"2023-10-30T13:59:24.738390+0000","last_became_active":"2023-10-30T13:35:07.771270+0000","last_became_peered":"2023-10-30T13:35:07.771270+0000","last_unstale":"2023-10-30T13:59:24.738390+0000","last_undegraded":"2023-10-30T13:59:24.738390+0000","last_fullsized":"2023-10-30T13:59:24.738390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'850","last_scrub_stamp":"2023-10-30T13:59:24.738213+0000","last_deep_scrub":"56'850","last_deep_scrub_stamp":"2023-10-30T13:59:24.738213+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.738213+0000","objects_scrubbed":157,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:41:27.710826+0000","scrub_duration":0.30834826700000001,"objects_trimmed":0,"snaptrim_duration":0.004034279,"stat_sum":{"num_bytes":1595822,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":126,"num_read_kb":16,"num_write":676,"num_write_kb":12551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"56'819","reported_seq":3045,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:38.313186+0000","last_change":"2023-10-30T13:59:38.313242+0000","last_active":"2023-10-30T13:59:38.313186+0000","last_peered":"2023-10-30T13:59:38.313186+0000","last_clean":"2023-10-30T13:59:38.313186+0000","last_became_active":"2023-10-30T13:35:07.758020+0000","last_became_peered":"2023-10-30T13:35:07.758020+0000","last_unstale":"2023-10-30T13:59:38.313186+0000","last_undegraded":"2023-10-30T13:59:38.313186+0000","last_fullsized":"2023-10-30T13:59:38.313186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'819","last_scrub_stamp":"2023-10-30T13:59:38.313111+0000","last_deep_scrub":"56'819","last_deep_scrub_stamp":"2023-10-30T13:59:38.313111+0000","last_clean_scrub_stamp":"2023-10-30T13:59:38.313111+0000","objects_scrubbed":145,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:35.451625+0000","scrub_duration":0.24463141199999999,"objects_trimmed":0,"snaptrim_duration":0.0030593479999999999,"stat_sum":{"num_bytes":7858557,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":117,"num_read_kb":4108,"num_write":634,"num_write_kb":18163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"56'858","reported_seq":3078,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:41.006397+0000","last_change":"2023-10-30T13:59:41.006480+0000","last_active":"2023-10-30T13:59:41.006397+0000","last_peered":"2023-10-30T13:59:41.006397+0000","last_clean":"2023-10-30T13:59:41.006397+0000","last_became_active":"2023-10-30T13:35:07.802590+0000","last_became_peered":"2023-10-30T13:35:07.802590+0000","last_unstale":"2023-10-30T13:59:41.006397+0000","last_undegraded":"2023-10-30T13:59:41.006397+0000","last_fullsized":"2023-10-30T13:59:41.006397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'858","last_scrub_stamp":"2023-10-30T13:59:41.006253+0000","last_deep_scrub":"56'858","last_deep_scrub_stamp":"2023-10-30T13:59:41.006253+0000","last_clean_scrub_stamp":"2023-10-30T13:59:41.006253+0000","objects_scrubbed":136,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:17.543137+0000","scrub_duration":0.28529553499999999,"objects_trimmed":0,"snaptrim_duration":0.0049754609999999996,"stat_sum":{"num_bytes":9799460,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":111,"num_read_kb":24,"num_write":679,"num_write_kb":16350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"56'822","reported_seq":3042,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:39.919298+0000","last_change":"2023-10-30T13:59:39.919348+0000","last_active":"2023-10-30T13:59:39.919298+0000","last_peered":"2023-10-30T13:59:39.919298+0000","last_clean":"2023-10-30T13:59:39.919298+0000","last_became_active":"2023-10-30T13:35:07.753011+0000","last_became_peered":"2023-10-30T13:35:07.753011+0000","last_unstale":"2023-10-30T13:59:39.919298+0000","last_undegraded":"2023-10-30T13:59:39.919298+0000","last_fullsized":"2023-10-30T13:59:39.919298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'822","last_scrub_stamp":"2023-10-30T13:59:39.919226+0000","last_deep_scrub":"56'822","last_deep_scrub_stamp":"2023-10-30T13:59:39.919226+0000","last_clean_scrub_stamp":"2023-10-30T13:59:39.919226+0000","objects_scrubbed":154,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:01.818651+0000","scrub_duration":0.18919842000000001,"objects_trimmed":0,"snaptrim_duration":0.0028468439999999999,"stat_sum":{"num_bytes":33375622,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":110,"num_read_kb":16400,"num_write":628,"num_write_kb":35724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"56'791","reported_seq":3012,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.370003+0000","last_change":"2023-10-30T13:59:25.370057+0000","last_active":"2023-10-30T13:59:25.370003+0000","last_peered":"2023-10-30T13:59:25.370003+0000","last_clean":"2023-10-30T13:59:25.370003+0000","last_became_active":"2023-10-30T13:35:07.767505+0000","last_became_peered":"2023-10-30T13:35:07.767505+0000","last_unstale":"2023-10-30T13:59:25.370003+0000","last_undegraded":"2023-10-30T13:59:25.370003+0000","last_fullsized":"2023-10-30T13:59:25.370003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'791","last_scrub_stamp":"2023-10-30T13:59:25.369932+0000","last_deep_scrub":"56'791","last_deep_scrub_stamp":"2023-10-30T13:59:25.369932+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.369932+0000","objects_scrubbed":174,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:57.726447+0000","scrub_duration":0.25855898900000002,"objects_trimmed":0,"snaptrim_duration":0.002839121,"stat_sum":{"num_bytes":9811427,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":112,"num_read_kb":10701,"num_write":625,"num_write_kb":15312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"56'859","reported_seq":3079,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:32.572493+0000","last_change":"2023-10-30T13:59:32.572574+0000","last_active":"2023-10-30T13:59:32.572493+0000","last_peered":"2023-10-30T13:59:32.572493+0000","last_clean":"2023-10-30T13:59:32.572493+0000","last_became_active":"2023-10-30T13:35:48.847825+0000","last_became_peered":"2023-10-30T13:35:48.847825+0000","last_unstale":"2023-10-30T13:59:32.572493+0000","last_undegraded":"2023-10-30T13:59:32.572493+0000","last_fullsized":"2023-10-30T13:59:32.572493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'859","last_scrub_stamp":"2023-10-30T13:59:32.572396+0000","last_deep_scrub":"56'859","last_deep_scrub_stamp":"2023-10-30T13:59:32.572396+0000","last_clean_scrub_stamp":"2023-10-30T13:59:32.572396+0000","objects_scrubbed":183,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:49:33.324446+0000","scrub_duration":0.17384617199999999,"objects_trimmed":0,"snaptrim_duration":0.0028647770000000002,"stat_sum":{"num_bytes":7231756,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":102,"num_read_kb":4848,"num_write":653,"num_write_kb":26441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"56'854","reported_seq":3085,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:45.006081+0000","last_change":"2023-10-30T13:59:45.006081+0000","last_active":"2023-10-30T13:59:45.006081+0000","last_peered":"2023-10-30T13:59:45.006081+0000","last_clean":"2023-10-30T13:59:45.006081+0000","last_became_active":"2023-10-30T13:35:07.788687+0000","last_became_peered":"2023-10-30T13:35:07.788687+0000","last_unstale":"2023-10-30T13:59:45.006081+0000","last_undegraded":"2023-10-30T13:59:45.006081+0000","last_fullsized":"2023-10-30T13:59:45.006081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'854","last_scrub_stamp":"2023-10-30T13:59:45.005743+0000","last_deep_scrub":"56'854","last_deep_scrub_stamp":"2023-10-30T13:59:45.005743+0000","last_clean_scrub_stamp":"2023-10-30T13:59:45.005743+0000","objects_scrubbed":162,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:19.548547+0000","scrub_duration":0.35207807699999999,"objects_trimmed":0,"snaptrim_duration":0.0060200970000000003,"stat_sum":{"num_bytes":22333767,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":120,"num_read_kb":4116,"num_write":656,"num_write_kb":43887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"56'925","reported_seq":3176,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.513815+0000","last_change":"2023-10-30T13:59:16.513815+0000","last_active":"2023-10-30T13:59:16.513815+0000","last_peered":"2023-10-30T13:59:16.513815+0000","last_clean":"2023-10-30T13:59:16.513815+0000","last_became_active":"2023-10-30T13:35:07.790285+0000","last_became_peered":"2023-10-30T13:35:07.790285+0000","last_unstale":"2023-10-30T13:59:16.513815+0000","last_undegraded":"2023-10-30T13:59:16.513815+0000","last_fullsized":"2023-10-30T13:59:16.513815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'925","last_scrub_stamp":"2023-10-30T13:59:16.513655+0000","last_deep_scrub":"56'925","last_deep_scrub_stamp":"2023-10-30T13:59:16.513655+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.513655+0000","objects_scrubbed":186,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:03.990730+0000","scrub_duration":0.27611799599999998,"objects_trimmed":0,"snaptrim_duration":0.001263171,"stat_sum":{"num_bytes":10546872,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":139,"num_read_kb":4118,"num_write":747,"num_write_kb":34715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"56'835","reported_seq":3056,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:32.085787+0000","last_change":"2023-10-30T13:59:32.085787+0000","last_active":"2023-10-30T13:59:32.085787+0000","last_peered":"2023-10-30T13:59:32.085787+0000","last_clean":"2023-10-30T13:59:32.085787+0000","last_became_active":"2023-10-30T13:35:07.779747+0000","last_became_peered":"2023-10-30T13:35:07.779747+0000","last_unstale":"2023-10-30T13:59:32.085787+0000","last_undegraded":"2023-10-30T13:59:32.085787+0000","last_fullsized":"2023-10-30T13:59:32.085787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'835","last_scrub_stamp":"2023-10-30T13:59:32.085458+0000","last_deep_scrub":"56'835","last_deep_scrub_stamp":"2023-10-30T13:59:32.085458+0000","last_clean_scrub_stamp":"2023-10-30T13:59:32.085458+0000","objects_scrubbed":153,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:28.945683+0000","scrub_duration":0.25482183200000003,"objects_trimmed":0,"snaptrim_duration":0.0036881090000000002,"stat_sum":{"num_bytes":14815628,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":110,"num_read_kb":20,"num_write":677,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"56'797","reported_seq":3014,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.874311+0000","last_change":"2023-10-30T13:59:15.874311+0000","last_active":"2023-10-30T13:59:15.874311+0000","last_peered":"2023-10-30T13:59:15.874311+0000","last_clean":"2023-10-30T13:59:15.874311+0000","last_became_active":"2023-10-30T13:35:07.754656+0000","last_became_peered":"2023-10-30T13:35:07.754656+0000","last_unstale":"2023-10-30T13:59:15.874311+0000","last_undegraded":"2023-10-30T13:59:15.874311+0000","last_fullsized":"2023-10-30T13:59:15.874311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'797","last_scrub_stamp":"2023-10-30T13:59:15.874087+0000","last_deep_scrub":"56'797","last_deep_scrub_stamp":"2023-10-30T13:59:15.874087+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.874087+0000","objects_scrubbed":163,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:30.985181+0000","scrub_duration":0.29104613299999998,"objects_trimmed":0,"snaptrim_duration":0.0065387930000000002,"stat_sum":{"num_bytes":14536361,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":108,"num_read_kb":8117,"num_write":641,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"56'698","reported_seq":777,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.085635+0000","last_change":"2023-10-30T13:59:12.085635+0000","last_active":"2023-10-30T13:59:12.085635+0000","last_peered":"2023-10-30T13:59:12.085635+0000","last_clean":"2023-10-30T13:59:12.085635+0000","last_became_active":"2023-10-30T13:35:07.286051+0000","last_became_peered":"2023-10-30T13:35:07.286051+0000","last_unstale":"2023-10-30T13:59:12.085635+0000","last_undegraded":"2023-10-30T13:59:12.085635+0000","last_fullsized":"2023-10-30T13:59:12.085635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'698","last_scrub_stamp":"2023-10-30T13:59:12.085518+0000","last_deep_scrub":"56'698","last_deep_scrub_stamp":"2023-10-30T13:59:12.085518+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.085518+0000","objects_scrubbed":4,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:55:05.427263+0000","scrub_duration":0.022533350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45057,"num_write":652,"num_write_kb":16749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":500,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"56'816","reported_seq":3052,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.871886+0000","last_change":"2023-10-30T13:59:20.871886+0000","last_active":"2023-10-30T13:59:20.871886+0000","last_peered":"2023-10-30T13:59:20.871886+0000","last_clean":"2023-10-30T13:59:20.871886+0000","last_became_active":"2023-10-30T13:35:07.762202+0000","last_became_peered":"2023-10-30T13:35:07.762202+0000","last_unstale":"2023-10-30T13:59:20.871886+0000","last_undegraded":"2023-10-30T13:59:20.871886+0000","last_fullsized":"2023-10-30T13:59:20.871886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'816","last_scrub_stamp":"2023-10-30T13:59:20.871754+0000","last_deep_scrub":"56'816","last_deep_scrub_stamp":"2023-10-30T13:59:20.871754+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.871754+0000","objects_scrubbed":166,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:54.494538+0000","scrub_duration":0.28592286,"objects_trimmed":0,"snaptrim_duration":0.0031306289999999998,"stat_sum":{"num_bytes":22886210,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":127,"num_read_kb":6230,"num_write":644,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"56'754","reported_seq":895,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:26.899016+0000","last_change":"2023-10-30T13:59:26.899016+0000","last_active":"2023-10-30T13:59:26.899016+0000","last_peered":"2023-10-30T13:59:26.899016+0000","last_clean":"2023-10-30T13:59:26.899016+0000","last_became_active":"2023-10-30T13:35:07.303255+0000","last_became_peered":"2023-10-30T13:35:07.303255+0000","last_unstale":"2023-10-30T13:59:26.899016+0000","last_undegraded":"2023-10-30T13:59:26.899016+0000","last_fullsized":"2023-10-30T13:59:26.899016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'754","last_scrub_stamp":"2023-10-30T13:59:26.898658+0000","last_deep_scrub":"56'754","last_deep_scrub_stamp":"2023-10-30T13:59:26.898658+0000","last_clean_scrub_stamp":"2023-10-30T13:59:26.898658+0000","objects_scrubbed":11,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:38.802448+0000","scrub_duration":0.095300314999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21047876,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":114393,"num_write":693,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21258,"num_omap_keys":40,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"56'146","reported_seq":264,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.093208+0000","last_change":"2023-10-30T13:59:09.093208+0000","last_active":"2023-10-30T13:59:09.093208+0000","last_peered":"2023-10-30T13:59:09.093208+0000","last_clean":"2023-10-30T13:59:09.093208+0000","last_became_active":"2023-10-30T13:35:07.293121+0000","last_became_peered":"2023-10-30T13:35:07.293121+0000","last_unstale":"2023-10-30T13:59:09.093208+0000","last_undegraded":"2023-10-30T13:59:09.093208+0000","last_fullsized":"2023-10-30T13:59:09.093208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'146","last_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_deep_scrub":"56'146","last_deep_scrub_stamp":"2023-10-30T13:59:09.093046+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.093046+0000","objects_scrubbed":7,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:36.674466+0000","scrub_duration":0.022130225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297526,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":20619,"num_write":102,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33717,"num_omap_keys":64,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"56'987","reported_seq":3240,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:39.282813+0000","last_change":"2023-10-30T13:59:39.282871+0000","last_active":"2023-10-30T13:59:39.282813+0000","last_peered":"2023-10-30T13:59:39.282813+0000","last_clean":"2023-10-30T13:59:39.282813+0000","last_became_active":"2023-10-30T13:35:07.750855+0000","last_became_peered":"2023-10-30T13:35:07.750855+0000","last_unstale":"2023-10-30T13:59:39.282813+0000","last_undegraded":"2023-10-30T13:59:39.282813+0000","last_fullsized":"2023-10-30T13:59:39.282813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'987","last_scrub_stamp":"2023-10-30T13:59:39.282741+0000","last_deep_scrub":"56'987","last_deep_scrub_stamp":"2023-10-30T13:59:39.282741+0000","last_clean_scrub_stamp":"2023-10-30T13:59:39.282741+0000","objects_scrubbed":195,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:31.434310+0000","scrub_duration":0.182689608,"objects_trimmed":0,"snaptrim_duration":0.0027758349999999999,"stat_sum":{"num_bytes":1695799,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":144,"num_read_kb":4120,"num_write":754,"num_write_kb":40525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"56'600","reported_seq":692,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.512895+0000","last_change":"2023-10-30T13:59:10.512895+0000","last_active":"2023-10-30T13:59:10.512895+0000","last_peered":"2023-10-30T13:59:10.512895+0000","last_clean":"2023-10-30T13:59:10.512895+0000","last_became_active":"2023-10-30T13:35:07.268847+0000","last_became_peered":"2023-10-30T13:35:07.268847+0000","last_unstale":"2023-10-30T13:59:10.512895+0000","last_undegraded":"2023-10-30T13:59:10.512895+0000","last_fullsized":"2023-10-30T13:59:10.512895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'600","last_scrub_stamp":"2023-10-30T13:59:10.512557+0000","last_deep_scrub":"56'600","last_deep_scrub_stamp":"2023-10-30T13:59:10.512557+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.512557+0000","objects_scrubbed":4,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:37:46.271383+0000","scrub_duration":0.060732762000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":96432,"num_write":542,"num_write_kb":20769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7630,"num_omap_keys":14,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"56'774","reported_seq":2997,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.878582+0000","last_change":"2023-10-30T13:59:17.878582+0000","last_active":"2023-10-30T13:59:17.878582+0000","last_peered":"2023-10-30T13:59:17.878582+0000","last_clean":"2023-10-30T13:59:17.878582+0000","last_became_active":"2023-10-30T13:35:07.749913+0000","last_became_peered":"2023-10-30T13:35:07.749913+0000","last_unstale":"2023-10-30T13:59:17.878582+0000","last_undegraded":"2023-10-30T13:59:17.878582+0000","last_fullsized":"2023-10-30T13:59:17.878582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'774","last_scrub_stamp":"2023-10-30T13:59:17.878418+0000","last_deep_scrub":"56'774","last_deep_scrub_stamp":"2023-10-30T13:59:17.878418+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.878418+0000","objects_scrubbed":155,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:30.998088+0000","scrub_duration":0.157018885,"objects_trimmed":0,"snaptrim_duration":0.0017556029999999999,"stat_sum":{"num_bytes":7038572,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":115,"num_read_kb":8211,"num_write":611,"num_write_kb":33441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"56'131","reported_seq":200,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.802120+0000","last_change":"2023-10-30T13:59:20.802254+0000","last_active":"2023-10-30T13:59:20.802120+0000","last_peered":"2023-10-30T13:59:20.802120+0000","last_clean":"2023-10-30T13:59:20.802120+0000","last_became_active":"2023-10-30T13:35:07.300686+0000","last_became_peered":"2023-10-30T13:35:07.300686+0000","last_unstale":"2023-10-30T13:59:20.802120+0000","last_undegraded":"2023-10-30T13:59:20.802120+0000","last_fullsized":"2023-10-30T13:59:20.802120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'131","last_scrub_stamp":"2023-10-30T13:59:20.801920+0000","last_deep_scrub":"56'131","last_deep_scrub_stamp":"2023-10-30T13:59:20.801920+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.801920+0000","objects_scrubbed":7,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:04.676924+0000","scrub_duration":0.017330934999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":91,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20610,"num_omap_keys":38,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"56'827","reported_seq":3034,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:27.695039+0000","last_change":"2023-10-30T13:59:27.695039+0000","last_active":"2023-10-30T13:59:27.695039+0000","last_peered":"2023-10-30T13:59:27.695039+0000","last_clean":"2023-10-30T13:59:27.695039+0000","last_became_active":"2023-10-30T13:35:49.007835+0000","last_became_peered":"2023-10-30T13:35:49.007835+0000","last_unstale":"2023-10-30T13:59:27.695039+0000","last_undegraded":"2023-10-30T13:59:27.695039+0000","last_fullsized":"2023-10-30T13:59:27.695039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'827","last_scrub_stamp":"2023-10-30T13:59:27.694847+0000","last_deep_scrub":"56'827","last_deep_scrub_stamp":"2023-10-30T13:59:27.694847+0000","last_clean_scrub_stamp":"2023-10-30T13:59:27.694847+0000","objects_scrubbed":154,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:47.134642+0000","scrub_duration":0.465225628,"objects_trimmed":0,"snaptrim_duration":0.0011632000000000001,"stat_sum":{"num_bytes":15868329,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":101,"num_read_kb":5707,"num_write":652,"num_write_kb":27262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"56'390","reported_seq":473,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.497506+0000","last_change":"2023-10-30T13:59:13.497506+0000","last_active":"2023-10-30T13:59:13.497506+0000","last_peered":"2023-10-30T13:59:13.497506+0000","last_clean":"2023-10-30T13:59:13.497506+0000","last_became_active":"2023-10-30T13:35:07.286069+0000","last_became_peered":"2023-10-30T13:35:07.286069+0000","last_unstale":"2023-10-30T13:59:13.497506+0000","last_undegraded":"2023-10-30T13:59:13.497506+0000","last_fullsized":"2023-10-30T13:59:13.497506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'390","last_scrub_stamp":"2023-10-30T13:59:13.497014+0000","last_deep_scrub":"56'390","last_deep_scrub_stamp":"2023-10-30T13:59:13.497014+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.497014+0000","objects_scrubbed":6,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:50.064647+0000","scrub_duration":0.074294748999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":61443,"num_write":345,"num_write_kb":20724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"56'809","reported_seq":3047,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.828504+0000","last_change":"2023-10-30T13:59:19.828578+0000","last_active":"2023-10-30T13:59:19.828504+0000","last_peered":"2023-10-30T13:59:19.828504+0000","last_clean":"2023-10-30T13:59:19.828504+0000","last_became_active":"2023-10-30T13:35:07.789678+0000","last_became_peered":"2023-10-30T13:35:07.789678+0000","last_unstale":"2023-10-30T13:59:19.828504+0000","last_undegraded":"2023-10-30T13:59:19.828504+0000","last_fullsized":"2023-10-30T13:59:19.828504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'809","last_scrub_stamp":"2023-10-30T13:59:19.828412+0000","last_deep_scrub":"56'809","last_deep_scrub_stamp":"2023-10-30T13:59:19.828412+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.828412+0000","objects_scrubbed":177,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:57.553096+0000","scrub_duration":0.224065866,"objects_trimmed":0,"snaptrim_duration":0.0028607060000000002,"stat_sum":{"num_bytes":18966683,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":128,"num_read_kb":11963,"num_write":623,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"56'719","reported_seq":814,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.839263+0000","last_change":"2023-10-30T13:59:10.839263+0000","last_active":"2023-10-30T13:59:10.839263+0000","last_peered":"2023-10-30T13:59:10.839263+0000","last_clean":"2023-10-30T13:59:10.839263+0000","last_became_active":"2023-10-30T13:35:07.295850+0000","last_became_peered":"2023-10-30T13:35:07.295850+0000","last_unstale":"2023-10-30T13:59:10.839263+0000","last_undegraded":"2023-10-30T13:59:10.839263+0000","last_fullsized":"2023-10-30T13:59:10.839263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'719","last_scrub_stamp":"2023-10-30T13:59:10.839113+0000","last_deep_scrub":"56'719","last_deep_scrub_stamp":"2023-10-30T13:59:10.839113+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.839113+0000","objects_scrubbed":8,"log_size":719,"log_dups_size":0,"ondisk_log_size":719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:45.790385+0000","scrub_duration":0.068460164000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":49185,"num_write":677,"num_write_kb":16860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21363,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"56'788","reported_seq":2996,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:14.912002+0000","last_change":"2023-10-30T13:59:14.912126+0000","last_active":"2023-10-30T13:59:14.912002+0000","last_peered":"2023-10-30T13:59:14.912002+0000","last_clean":"2023-10-30T13:59:14.912002+0000","last_became_active":"2023-10-30T13:35:07.794255+0000","last_became_peered":"2023-10-30T13:35:07.794255+0000","last_unstale":"2023-10-30T13:59:14.912002+0000","last_undegraded":"2023-10-30T13:59:14.912002+0000","last_fullsized":"2023-10-30T13:59:14.912002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'788","last_scrub_stamp":"2023-10-30T13:59:14.911840+0000","last_deep_scrub":"56'788","last_deep_scrub_stamp":"2023-10-30T13:59:14.911840+0000","last_clean_scrub_stamp":"2023-10-30T13:59:14.911840+0000","objects_scrubbed":165,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:58.599422+0000","scrub_duration":0.31948987000000001,"objects_trimmed":0,"snaptrim_duration":0.0070741379999999998,"stat_sum":{"num_bytes":9996128,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":99,"num_read_kb":8209,"num_write":632,"num_write_kb":23925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"56'1295","reported_seq":1408,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:41.373284+0000","last_change":"2023-10-30T13:59:23.855056+0000","last_active":"2023-10-30T13:59:41.373284+0000","last_peered":"2023-10-30T13:59:41.373284+0000","last_clean":"2023-10-30T13:59:41.373284+0000","last_became_active":"2023-10-30T13:35:07.285725+0000","last_became_peered":"2023-10-30T13:35:07.285725+0000","last_unstale":"2023-10-30T13:59:41.373284+0000","last_undegraded":"2023-10-30T13:59:41.373284+0000","last_fullsized":"2023-10-30T13:59:41.373284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1274","last_scrub_stamp":"2023-10-30T13:59:23.854712+0000","last_deep_scrub":"56'1274","last_deep_scrub_stamp":"2023-10-30T13:59:23.854712+0000","last_clean_scrub_stamp":"2023-10-30T13:59:23.854712+0000","objects_scrubbed":10,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:21.240711+0000","scrub_duration":0.074187731000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16569325,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":86059,"num_write":1237,"num_write_kb":21008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29629,"num_omap_keys":53,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"56'819","reported_seq":3031,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:49.969972+0000","last_change":"2023-10-30T13:59:49.970050+0000","last_active":"2023-10-30T13:59:49.969972+0000","last_peered":"2023-10-30T13:59:49.969972+0000","last_clean":"2023-10-30T13:59:49.969972+0000","last_became_active":"2023-10-30T13:35:07.790881+0000","last_became_peered":"2023-10-30T13:35:07.790881+0000","last_unstale":"2023-10-30T13:59:49.969972+0000","last_undegraded":"2023-10-30T13:59:49.969972+0000","last_fullsized":"2023-10-30T13:59:49.969972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'819","last_scrub_stamp":"2023-10-30T13:59:49.969877+0000","last_deep_scrub":"56'819","last_deep_scrub_stamp":"2023-10-30T13:59:49.969877+0000","last_clean_scrub_stamp":"2023-10-30T13:59:49.969877+0000","objects_scrubbed":148,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:34.089787+0000","scrub_duration":0.40625873000000001,"objects_trimmed":0,"snaptrim_duration":0.0070810969999999997,"stat_sum":{"num_bytes":18214186,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":104,"num_read_kb":12299,"num_write":638,"num_write_kb":31808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"56'436","reported_seq":542,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.574205+0000","last_change":"2023-10-30T13:59:08.574261+0000","last_active":"2023-10-30T13:59:08.574205+0000","last_peered":"2023-10-30T13:59:08.574205+0000","last_clean":"2023-10-30T13:59:08.574205+0000","last_became_active":"2023-10-30T13:35:07.276016+0000","last_became_peered":"2023-10-30T13:35:07.276016+0000","last_unstale":"2023-10-30T13:59:08.574205+0000","last_undegraded":"2023-10-30T13:59:08.574205+0000","last_fullsized":"2023-10-30T13:59:08.574205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'436","last_scrub_stamp":"2023-10-30T13:59:08.574118+0000","last_deep_scrub":"56'436","last_deep_scrub_stamp":"2023-10-30T13:59:08.574118+0000","last_clean_scrub_stamp":"2023-10-30T13:59:08.574118+0000","objects_scrubbed":8,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:26.795616+0000","scrub_duration":0.024734227000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":77906,"num_write":384,"num_write_kb":12555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":59046,"num_omap_keys":109,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"56'843","reported_seq":3086,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:22.473412+0000","last_change":"2023-10-30T13:59:22.473466+0000","last_active":"2023-10-30T13:59:22.473412+0000","last_peered":"2023-10-30T13:59:22.473412+0000","last_clean":"2023-10-30T13:59:22.473412+0000","last_became_active":"2023-10-30T13:35:49.010460+0000","last_became_peered":"2023-10-30T13:35:49.010460+0000","last_unstale":"2023-10-30T13:59:22.473412+0000","last_undegraded":"2023-10-30T13:59:22.473412+0000","last_fullsized":"2023-10-30T13:59:22.473412+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'843","last_scrub_stamp":"2023-10-30T13:59:22.473336+0000","last_deep_scrub":"56'843","last_deep_scrub_stamp":"2023-10-30T13:59:22.473336+0000","last_clean_scrub_stamp":"2023-10-30T13:59:22.473336+0000","objects_scrubbed":164,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:50:11.354602+0000","scrub_duration":0.279252891,"objects_trimmed":0,"snaptrim_duration":0.001187553,"stat_sum":{"num_bytes":13816511,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":138,"num_read_kb":8210,"num_write":662,"num_write_kb":56040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"56'179","reported_seq":248,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:19.414864+0000","last_change":"2023-10-30T13:59:19.414972+0000","last_active":"2023-10-30T13:59:19.414864+0000","last_peered":"2023-10-30T13:59:19.414864+0000","last_clean":"2023-10-30T13:59:19.414864+0000","last_became_active":"2023-10-30T13:35:07.285151+0000","last_became_peered":"2023-10-30T13:35:07.285151+0000","last_unstale":"2023-10-30T13:59:19.414864+0000","last_undegraded":"2023-10-30T13:59:19.414864+0000","last_fullsized":"2023-10-30T13:59:19.414864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'179","last_scrub_stamp":"2023-10-30T13:59:19.414635+0000","last_deep_scrub":"56'179","last_deep_scrub_stamp":"2023-10-30T13:59:19.414635+0000","last_clean_scrub_stamp":"2023-10-30T13:59:19.414635+0000","objects_scrubbed":3,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:23.618899+0000","scrub_duration":0.0097075200000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":632,"num_write":147,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13012,"num_omap_keys":24,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"56'881","reported_seq":3115,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:52.874352+0000","last_change":"2023-10-30T13:59:52.874430+0000","last_active":"2023-10-30T13:59:52.874352+0000","last_peered":"2023-10-30T13:59:52.874352+0000","last_clean":"2023-10-30T13:59:52.874352+0000","last_became_active":"2023-10-30T13:35:07.796457+0000","last_became_peered":"2023-10-30T13:35:07.796457+0000","last_unstale":"2023-10-30T13:59:52.874352+0000","last_undegraded":"2023-10-30T13:59:52.874352+0000","last_fullsized":"2023-10-30T13:59:52.874352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'881","last_scrub_stamp":"2023-10-30T13:59:52.874247+0000","last_deep_scrub":"56'881","last_deep_scrub_stamp":"2023-10-30T13:59:52.874247+0000","last_clean_scrub_stamp":"2023-10-30T13:59:52.874247+0000","objects_scrubbed":168,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:05:03.993455+0000","scrub_duration":0.36518218899999999,"objects_trimmed":0,"snaptrim_duration":0.0092364379999999996,"stat_sum":{"num_bytes":16180540,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":126,"num_read_kb":8225,"num_write":670,"num_write_kb":34788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"56'176","reported_seq":248,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.774022+0000","last_change":"2023-10-30T13:59:25.774136+0000","last_active":"2023-10-30T13:59:25.774022+0000","last_peered":"2023-10-30T13:59:25.774022+0000","last_clean":"2023-10-30T13:59:25.774022+0000","last_became_active":"2023-10-30T13:35:07.301775+0000","last_became_peered":"2023-10-30T13:35:07.301775+0000","last_unstale":"2023-10-30T13:59:25.774022+0000","last_undegraded":"2023-10-30T13:59:25.774022+0000","last_fullsized":"2023-10-30T13:59:25.774022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'176","last_scrub_stamp":"2023-10-30T13:59:25.773814+0000","last_deep_scrub":"56'176","last_deep_scrub_stamp":"2023-10-30T13:59:25.773814+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.773814+0000","objects_scrubbed":5,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:01.029899+0000","scrub_duration":0.016157685000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4103,"num_write":119,"num_write_kb":4220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31622,"num_omap_keys":58,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"56'791","reported_seq":2999,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:29.956919+0000","last_change":"2023-10-30T13:59:29.956971+0000","last_active":"2023-10-30T13:59:29.956919+0000","last_peered":"2023-10-30T13:59:29.956919+0000","last_clean":"2023-10-30T13:59:29.956919+0000","last_became_active":"2023-10-30T13:35:07.772876+0000","last_became_peered":"2023-10-30T13:35:07.772876+0000","last_unstale":"2023-10-30T13:59:29.956919+0000","last_undegraded":"2023-10-30T13:59:29.956919+0000","last_fullsized":"2023-10-30T13:59:29.956919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'791","last_scrub_stamp":"2023-10-30T13:59:29.956832+0000","last_deep_scrub":"56'791","last_deep_scrub_stamp":"2023-10-30T13:59:29.956832+0000","last_clean_scrub_stamp":"2023-10-30T13:59:29.956832+0000","objects_scrubbed":152,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:19.118645+0000","scrub_duration":0.199579172,"objects_trimmed":0,"snaptrim_duration":0.0045350950000000003,"stat_sum":{"num_bytes":14031523,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":99,"num_read_kb":8208,"num_write":613,"num_write_kb":14857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"56'625","reported_seq":743,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.373340+0000","last_change":"2023-10-30T13:59:09.373401+0000","last_active":"2023-10-30T13:59:09.373340+0000","last_peered":"2023-10-30T13:59:09.373340+0000","last_clean":"2023-10-30T13:59:09.373340+0000","last_became_active":"2023-10-30T13:35:07.277280+0000","last_became_peered":"2023-10-30T13:35:07.277280+0000","last_unstale":"2023-10-30T13:59:09.373340+0000","last_undegraded":"2023-10-30T13:59:09.373340+0000","last_fullsized":"2023-10-30T13:59:09.373340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'625","last_scrub_stamp":"2023-10-30T13:59:09.373262+0000","last_deep_scrub":"56'625","last_deep_scrub_stamp":"2023-10-30T13:59:09.373262+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.373262+0000","objects_scrubbed":12,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:19.085134+0000","scrub_duration":0.038380236999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":130,"num_read_kb":41105,"num_write":605,"num_write_kb":12685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":87540,"num_omap_keys":149,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"56'854","reported_seq":3079,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.886946+0000","last_change":"2023-10-30T13:59:13.886999+0000","last_active":"2023-10-30T13:59:13.886946+0000","last_peered":"2023-10-30T13:59:13.886946+0000","last_clean":"2023-10-30T13:59:13.886946+0000","last_became_active":"2023-10-30T13:35:07.790454+0000","last_became_peered":"2023-10-30T13:35:07.790454+0000","last_unstale":"2023-10-30T13:59:13.886946+0000","last_undegraded":"2023-10-30T13:59:13.886946+0000","last_fullsized":"2023-10-30T13:59:13.886946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'854","last_scrub_stamp":"2023-10-30T13:59:13.886861+0000","last_deep_scrub":"56'854","last_deep_scrub_stamp":"2023-10-30T13:59:13.886861+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.886861+0000","objects_scrubbed":157,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:39.441296+0000","scrub_duration":0.25638365299999999,"objects_trimmed":0,"snaptrim_duration":0.0089036940000000002,"stat_sum":{"num_bytes":6058105,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":119,"num_read_kb":8210,"num_write":694,"num_write_kb":35073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"56'356","reported_seq":440,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.555089+0000","last_change":"2023-10-30T13:59:12.555089+0000","last_active":"2023-10-30T13:59:12.555089+0000","last_peered":"2023-10-30T13:59:12.555089+0000","last_clean":"2023-10-30T13:59:12.555089+0000","last_became_active":"2023-10-30T13:35:07.276665+0000","last_became_peered":"2023-10-30T13:35:07.276665+0000","last_unstale":"2023-10-30T13:59:12.555089+0000","last_undegraded":"2023-10-30T13:59:12.555089+0000","last_fullsized":"2023-10-30T13:59:12.555089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'356","last_scrub_stamp":"2023-10-30T13:59:12.554618+0000","last_deep_scrub":"56'356","last_deep_scrub_stamp":"2023-10-30T13:59:12.554618+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.554618+0000","objects_scrubbed":6,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:45.923483+0000","scrub_duration":0.048085234999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":28724,"num_write":357,"num_write_kb":8477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20972,"num_omap_keys":39,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"56'782","reported_seq":2991,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.912905+0000","last_change":"2023-10-30T13:59:12.912905+0000","last_active":"2023-10-30T13:59:12.912905+0000","last_peered":"2023-10-30T13:59:12.912905+0000","last_clean":"2023-10-30T13:59:12.912905+0000","last_became_active":"2023-10-30T13:35:07.793495+0000","last_became_peered":"2023-10-30T13:35:07.793495+0000","last_unstale":"2023-10-30T13:59:12.912905+0000","last_undegraded":"2023-10-30T13:59:12.912905+0000","last_fullsized":"2023-10-30T13:59:12.912905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'782","last_scrub_stamp":"2023-10-30T13:59:12.912738+0000","last_deep_scrub":"56'782","last_deep_scrub_stamp":"2023-10-30T13:59:12.912738+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.912738+0000","objects_scrubbed":175,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:22.069756+0000","scrub_duration":0.27457796699999998,"objects_trimmed":0,"snaptrim_duration":0.0094781629999999995,"stat_sum":{"num_bytes":6414224,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":99,"num_read_kb":8205,"num_write":617,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"56'644","reported_seq":731,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:12.281633+0000","last_change":"2023-10-30T13:59:12.281633+0000","last_active":"2023-10-30T13:59:12.281633+0000","last_peered":"2023-10-30T13:59:12.281633+0000","last_clean":"2023-10-30T13:59:12.281633+0000","last_became_active":"2023-10-30T13:35:07.286651+0000","last_became_peered":"2023-10-30T13:35:07.286651+0000","last_unstale":"2023-10-30T13:59:12.281633+0000","last_undegraded":"2023-10-30T13:59:12.281633+0000","last_fullsized":"2023-10-30T13:59:12.281633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'644","last_scrub_stamp":"2023-10-30T13:59:12.281461+0000","last_deep_scrub":"56'644","last_deep_scrub_stamp":"2023-10-30T13:59:12.281461+0000","last_clean_scrub_stamp":"2023-10-30T13:59:12.281461+0000","objects_scrubbed":6,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:09.939939+0000","scrub_duration":0.025851885000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":33457,"num_write":586,"num_write_kb":12711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33662,"num_omap_keys":63,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"56'836","reported_seq":3056,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:58.904904+0000","last_change":"2023-10-30T13:59:58.904904+0000","last_active":"2023-10-30T13:59:58.904904+0000","last_peered":"2023-10-30T13:59:58.904904+0000","last_clean":"2023-10-30T13:59:58.904904+0000","last_became_active":"2023-10-30T13:35:07.791975+0000","last_became_peered":"2023-10-30T13:35:07.791975+0000","last_unstale":"2023-10-30T13:59:58.904904+0000","last_undegraded":"2023-10-30T13:59:58.904904+0000","last_fullsized":"2023-10-30T13:59:58.904904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'836","last_scrub_stamp":"2023-10-30T13:59:58.904712+0000","last_deep_scrub":"56'836","last_deep_scrub_stamp":"2023-10-30T13:59:58.904712+0000","last_clean_scrub_stamp":"2023-10-30T13:59:58.904712+0000","objects_scrubbed":160,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:11.044874+0000","scrub_duration":0.31030290700000002,"objects_trimmed":0,"snaptrim_duration":0.010320058,"stat_sum":{"num_bytes":13797516,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":111,"num_read_kb":8210,"num_write":631,"num_write_kb":35965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"56'405","reported_seq":489,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:09.759865+0000","last_change":"2023-10-30T13:59:09.759865+0000","last_active":"2023-10-30T13:59:09.759865+0000","last_peered":"2023-10-30T13:59:09.759865+0000","last_clean":"2023-10-30T13:59:09.759865+0000","last_became_active":"2023-10-30T13:35:07.279122+0000","last_became_peered":"2023-10-30T13:35:07.279122+0000","last_unstale":"2023-10-30T13:59:09.759865+0000","last_undegraded":"2023-10-30T13:59:09.759865+0000","last_fullsized":"2023-10-30T13:59:09.759865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'405","last_scrub_stamp":"2023-10-30T13:59:09.759704+0000","last_deep_scrub":"56'405","last_deep_scrub_stamp":"2023-10-30T13:59:09.759704+0000","last_clean_scrub_stamp":"2023-10-30T13:59:09.759704+0000","objects_scrubbed":4,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:26:19.730801+0000","scrub_duration":0.036239831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388678,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":36872,"num_write":370,"num_write_kb":12516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11466,"num_omap_keys":21,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"56'877","reported_seq":3122,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:30.928146+0000","last_change":"2023-10-30T13:59:30.928146+0000","last_active":"2023-10-30T13:59:30.928146+0000","last_peered":"2023-10-30T13:59:30.928146+0000","last_clean":"2023-10-30T13:59:30.928146+0000","last_became_active":"2023-10-30T13:35:07.790586+0000","last_became_peered":"2023-10-30T13:35:07.790586+0000","last_unstale":"2023-10-30T13:59:30.928146+0000","last_undegraded":"2023-10-30T13:59:30.928146+0000","last_fullsized":"2023-10-30T13:59:30.928146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'877","last_scrub_stamp":"2023-10-30T13:59:30.927723+0000","last_deep_scrub":"56'877","last_deep_scrub_stamp":"2023-10-30T13:59:30.927723+0000","last_clean_scrub_stamp":"2023-10-30T13:59:30.927723+0000","objects_scrubbed":150,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:06.613346+0000","scrub_duration":0.30010206900000003,"objects_trimmed":0,"snaptrim_duration":0.0066640720000000001,"stat_sum":{"num_bytes":18395347,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":135,"num_read_kb":4114,"num_write":711,"num_write_kb":24326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"56'227","reported_seq":305,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.292450+0000","last_change":"2023-10-30T13:59:11.292450+0000","last_active":"2023-10-30T13:59:11.292450+0000","last_peered":"2023-10-30T13:59:11.292450+0000","last_clean":"2023-10-30T13:59:11.292450+0000","last_became_active":"2023-10-30T13:35:07.282147+0000","last_became_peered":"2023-10-30T13:35:07.282147+0000","last_unstale":"2023-10-30T13:59:11.292450+0000","last_undegraded":"2023-10-30T13:59:11.292450+0000","last_fullsized":"2023-10-30T13:59:11.292450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'227","last_scrub_stamp":"2023-10-30T13:59:11.292333+0000","last_deep_scrub":"56'227","last_deep_scrub_stamp":"2023-10-30T13:59:11.292333+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.292333+0000","objects_scrubbed":5,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:09.441346+0000","scrub_duration":0.024139252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":20494,"num_write":198,"num_write_kb":4363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10920,"num_omap_keys":20,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"56'838","reported_seq":3053,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:36.884844+0000","last_change":"2023-10-30T13:59:36.884844+0000","last_active":"2023-10-30T13:59:36.884844+0000","last_peered":"2023-10-30T13:59:36.884844+0000","last_clean":"2023-10-30T13:59:36.884844+0000","last_became_active":"2023-10-30T13:35:07.802499+0000","last_became_peered":"2023-10-30T13:35:07.802499+0000","last_unstale":"2023-10-30T13:59:36.884844+0000","last_undegraded":"2023-10-30T13:59:36.884844+0000","last_fullsized":"2023-10-30T13:59:36.884844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'838","last_scrub_stamp":"2023-10-30T13:59:36.884678+0000","last_deep_scrub":"56'838","last_deep_scrub_stamp":"2023-10-30T13:59:36.884678+0000","last_clean_scrub_stamp":"2023-10-30T13:59:36.884678+0000","objects_scrubbed":161,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:17:27.989261+0000","scrub_duration":0.33098685500000002,"objects_trimmed":0,"snaptrim_duration":0.0084474579999999997,"stat_sum":{"num_bytes":18100965,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":106,"num_read_kb":8207,"num_write":635,"num_write_kb":40451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"56'119","reported_seq":223,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:14.603525+0000","last_change":"2023-10-30T13:59:14.603715+0000","last_active":"2023-10-30T13:59:14.603525+0000","last_peered":"2023-10-30T13:59:14.603525+0000","last_clean":"2023-10-30T13:59:14.603525+0000","last_became_active":"2023-10-30T13:34:55.136508+0000","last_became_peered":"2023-10-30T13:34:55.136508+0000","last_unstale":"2023-10-30T13:59:14.603525+0000","last_undegraded":"2023-10-30T13:59:14.603525+0000","last_fullsized":"2023-10-30T13:59:14.603525+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'119","last_scrub_stamp":"2023-10-30T13:59:14.603316+0000","last_deep_scrub":"56'119","last_deep_scrub_stamp":"2023-10-30T13:59:14.603316+0000","last_clean_scrub_stamp":"2023-10-30T13:59:14.603316+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:48.801390+0000","scrub_duration":0.012158702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'353","reported_seq":430,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.516196+0000","last_change":"2023-10-30T13:59:11.516196+0000","last_active":"2023-10-30T13:59:11.516196+0000","last_peered":"2023-10-30T13:59:11.516196+0000","last_clean":"2023-10-30T13:59:11.516196+0000","last_became_active":"2023-10-30T13:35:07.278498+0000","last_became_peered":"2023-10-30T13:35:07.278498+0000","last_unstale":"2023-10-30T13:59:11.516196+0000","last_undegraded":"2023-10-30T13:59:11.516196+0000","last_fullsized":"2023-10-30T13:59:11.516196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'353","last_scrub_stamp":"2023-10-30T13:59:11.515780+0000","last_deep_scrub":"56'353","last_deep_scrub_stamp":"2023-10-30T13:59:11.515780+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.515780+0000","objects_scrubbed":8,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:30.246583+0000","scrub_duration":0.054846111000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388674,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":20491,"num_write":340,"num_write_kb":8443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31208,"num_omap_keys":58,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'799","reported_seq":3018,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.131904+0000","last_change":"2023-10-30T13:59:17.131904+0000","last_active":"2023-10-30T13:59:17.131904+0000","last_peered":"2023-10-30T13:59:17.131904+0000","last_clean":"2023-10-30T13:59:17.131904+0000","last_became_active":"2023-10-30T13:35:48.847905+0000","last_became_peered":"2023-10-30T13:35:48.847905+0000","last_unstale":"2023-10-30T13:59:17.131904+0000","last_undegraded":"2023-10-30T13:59:17.131904+0000","last_fullsized":"2023-10-30T13:59:17.131904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'799","last_scrub_stamp":"2023-10-30T13:59:17.131773+0000","last_deep_scrub":"56'799","last_deep_scrub_stamp":"2023-10-30T13:59:17.131773+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.131773+0000","objects_scrubbed":182,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:20.503652+0000","scrub_duration":0.14335236700000001,"objects_trimmed":0,"snaptrim_duration":0.002389721,"stat_sum":{"num_bytes":10749534,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":100,"num_read_kb":11,"num_write":647,"num_write_kb":20249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"56'357","reported_seq":454,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:21.575803+0000","last_change":"2023-10-30T13:59:21.575920+0000","last_active":"2023-10-30T13:59:21.575803+0000","last_peered":"2023-10-30T13:59:21.575803+0000","last_clean":"2023-10-30T13:59:21.575803+0000","last_became_active":"2023-10-30T13:35:07.299914+0000","last_became_peered":"2023-10-30T13:35:07.299914+0000","last_unstale":"2023-10-30T13:59:21.575803+0000","last_undegraded":"2023-10-30T13:59:21.575803+0000","last_fullsized":"2023-10-30T13:59:21.575803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'357","last_scrub_stamp":"2023-10-30T13:59:21.575603+0000","last_deep_scrub":"56'357","last_deep_scrub_stamp":"2023-10-30T13:59:21.575603+0000","last_clean_scrub_stamp":"2023-10-30T13:59:21.575603+0000","objects_scrubbed":5,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:43.316435+0000","scrub_duration":0.021550125999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":102412,"num_write":307,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16214,"num_omap_keys":31,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"56'824","reported_seq":3052,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.910909+0000","last_change":"2023-10-30T13:59:16.910909+0000","last_active":"2023-10-30T13:59:16.910909+0000","last_peered":"2023-10-30T13:59:16.910909+0000","last_clean":"2023-10-30T13:59:16.910909+0000","last_became_active":"2023-10-30T13:35:07.759852+0000","last_became_peered":"2023-10-30T13:35:07.759852+0000","last_unstale":"2023-10-30T13:59:16.910909+0000","last_undegraded":"2023-10-30T13:59:16.910909+0000","last_fullsized":"2023-10-30T13:59:16.910909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'824","last_scrub_stamp":"2023-10-30T13:59:16.910789+0000","last_deep_scrub":"56'824","last_deep_scrub_stamp":"2023-10-30T13:59:16.910789+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.910789+0000","objects_scrubbed":156,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:06.670797+0000","scrub_duration":0.25255747699999997,"objects_trimmed":0,"snaptrim_duration":0.0018610090000000001,"stat_sum":{"num_bytes":24271212,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":117,"num_read_kb":10,"num_write":671,"num_write_kb":33782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"56'1505","reported_seq":2039,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:10.342107+0000","last_change":"2023-10-30T13:59:10.342107+0000","last_active":"2023-10-30T13:59:10.342107+0000","last_peered":"2023-10-30T13:59:10.342107+0000","last_clean":"2023-10-30T13:59:10.342107+0000","last_became_active":"2023-10-30T13:35:07.281369+0000","last_became_peered":"2023-10-30T13:35:07.281369+0000","last_unstale":"2023-10-30T13:59:10.342107+0000","last_undegraded":"2023-10-30T13:59:10.342107+0000","last_fullsized":"2023-10-30T13:59:10.342107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1505","last_scrub_stamp":"2023-10-30T13:59:10.341982+0000","last_deep_scrub":"56'1505","last_deep_scrub_stamp":"2023-10-30T13:59:10.341982+0000","last_clean_scrub_stamp":"2023-10-30T13:59:10.341982+0000","objects_scrubbed":9,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:03.471145+0000","scrub_duration":0.046294123999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959716,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":475,"num_read_kb":12479,"num_write":1460,"num_write_kb":17319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8598,"num_omap_keys":16,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'803","reported_seq":3030,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:20.925329+0000","last_change":"2023-10-30T13:59:20.925381+0000","last_active":"2023-10-30T13:59:20.925329+0000","last_peered":"2023-10-30T13:59:20.925329+0000","last_clean":"2023-10-30T13:59:20.925329+0000","last_became_active":"2023-10-30T13:35:07.750726+0000","last_became_peered":"2023-10-30T13:35:07.750726+0000","last_unstale":"2023-10-30T13:59:20.925329+0000","last_undegraded":"2023-10-30T13:59:20.925329+0000","last_fullsized":"2023-10-30T13:59:20.925329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'803","last_scrub_stamp":"2023-10-30T13:59:20.925256+0000","last_deep_scrub":"56'803","last_deep_scrub_stamp":"2023-10-30T13:59:20.925256+0000","last_clean_scrub_stamp":"2023-10-30T13:59:20.925256+0000","objects_scrubbed":166,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:15.483272+0000","scrub_duration":0.17049674500000001,"objects_trimmed":0,"snaptrim_duration":0.0022228600000000001,"stat_sum":{"num_bytes":9925305,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":116,"num_read_kb":4117,"num_write":612,"num_write_kb":24793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'634","reported_seq":717,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.440238+0000","last_change":"2023-10-30T13:59:16.440238+0000","last_active":"2023-10-30T13:59:16.440238+0000","last_peered":"2023-10-30T13:59:16.440238+0000","last_clean":"2023-10-30T13:59:16.440238+0000","last_became_active":"2023-10-30T13:35:07.280991+0000","last_became_peered":"2023-10-30T13:35:07.280991+0000","last_unstale":"2023-10-30T13:59:16.440238+0000","last_undegraded":"2023-10-30T13:59:16.440238+0000","last_fullsized":"2023-10-30T13:59:16.440238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'634","last_scrub_stamp":"2023-10-30T13:59:16.440060+0000","last_deep_scrub":"56'634","last_deep_scrub_stamp":"2023-10-30T13:59:16.440060+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.440060+0000","objects_scrubbed":4,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:53.542410+0000","scrub_duration":0.021825480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":61440,"num_write":567,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":28266,"num_omap_keys":53,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'840","reported_seq":3052,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.860153+0000","last_change":"2023-10-30T13:59:16.860153+0000","last_active":"2023-10-30T13:59:16.860153+0000","last_peered":"2023-10-30T13:59:16.860153+0000","last_clean":"2023-10-30T13:59:16.860153+0000","last_became_active":"2023-10-30T13:35:07.761890+0000","last_became_peered":"2023-10-30T13:35:07.761890+0000","last_unstale":"2023-10-30T13:59:16.860153+0000","last_undegraded":"2023-10-30T13:59:16.860153+0000","last_fullsized":"2023-10-30T13:59:16.860153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'840","last_scrub_stamp":"2023-10-30T13:59:16.860020+0000","last_deep_scrub":"56'840","last_deep_scrub_stamp":"2023-10-30T13:59:16.860020+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.860020+0000","objects_scrubbed":157,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:19.322252+0000","scrub_duration":0.29827281300000003,"objects_trimmed":0,"snaptrim_duration":0.0066434720000000001,"stat_sum":{"num_bytes":26554596,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":101,"num_read_kb":3528,"num_write":689,"num_write_kb":37408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"56'729","reported_seq":804,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:08.710437+0000","last_change":"2023-10-30T13:59:08.710437+0000","last_active":"2023-10-30T13:59:08.710437+0000","last_peered":"2023-10-30T13:59:08.710437+0000","last_clean":"2023-10-30T13:59:08.710437+0000","last_became_active":"2023-10-30T13:35:07.276326+0000","last_became_peered":"2023-10-30T13:35:07.276326+0000","last_unstale":"2023-10-30T13:59:08.710437+0000","last_undegraded":"2023-10-30T13:59:08.710437+0000","last_fullsized":"2023-10-30T13:59:08.710437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'729","last_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_deep_scrub":"56'729","last_deep_scrub_stamp":"2023-10-30T13:59:08.710307+0000","last_clean_scrub_stamp":"2023-10-30T13:59:08.710307+0000","objects_scrubbed":3,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:48.485825+0000","scrub_duration":0.018720046000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":18914,"num_write":694,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10538,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"56'858","reported_seq":3076,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:33.952048+0000","last_change":"2023-10-30T13:59:33.952262+0000","last_active":"2023-10-30T13:59:33.952048+0000","last_peered":"2023-10-30T13:59:33.952048+0000","last_clean":"2023-10-30T13:59:33.952048+0000","last_became_active":"2023-10-30T13:35:07.798667+0000","last_became_peered":"2023-10-30T13:35:07.798667+0000","last_unstale":"2023-10-30T13:59:33.952048+0000","last_undegraded":"2023-10-30T13:59:33.952048+0000","last_fullsized":"2023-10-30T13:59:33.952048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'858","last_scrub_stamp":"2023-10-30T13:59:33.951806+0000","last_deep_scrub":"56'858","last_deep_scrub_stamp":"2023-10-30T13:59:33.951806+0000","last_clean_scrub_stamp":"2023-10-30T13:59:33.951806+0000","objects_scrubbed":160,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:30.144633+0000","scrub_duration":0.36214772699999997,"objects_trimmed":0,"snaptrim_duration":0.0064085849999999996,"stat_sum":{"num_bytes":10071990,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":4109,"num_write":659,"num_write_kb":40765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"56'763","reported_seq":842,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:33.576155+0000","last_change":"2023-10-30T13:59:33.576384+0000","last_active":"2023-10-30T13:59:33.576155+0000","last_peered":"2023-10-30T13:59:33.576155+0000","last_clean":"2023-10-30T13:59:33.576155+0000","last_became_active":"2023-10-30T13:35:07.298857+0000","last_became_peered":"2023-10-30T13:35:07.298857+0000","last_unstale":"2023-10-30T13:59:33.576155+0000","last_undegraded":"2023-10-30T13:59:33.576155+0000","last_fullsized":"2023-10-30T13:59:33.576155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'763","last_scrub_stamp":"2023-10-30T13:59:33.575889+0000","last_deep_scrub":"56'763","last_deep_scrub_stamp":"2023-10-30T13:59:33.575889+0000","last_clean_scrub_stamp":"2023-10-30T13:59:33.575889+0000","objects_scrubbed":7,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:16.668864+0000","scrub_duration":0.061119272000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":707,"num_write_kb":16664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7670,"num_omap_keys":15,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"56'810","reported_seq":3029,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:27.812324+0000","last_change":"2023-10-30T13:59:27.812324+0000","last_active":"2023-10-30T13:59:27.812324+0000","last_peered":"2023-10-30T13:59:27.812324+0000","last_clean":"2023-10-30T13:59:27.812324+0000","last_became_active":"2023-10-30T13:35:07.753914+0000","last_became_peered":"2023-10-30T13:35:07.753914+0000","last_unstale":"2023-10-30T13:59:27.812324+0000","last_undegraded":"2023-10-30T13:59:27.812324+0000","last_fullsized":"2023-10-30T13:59:27.812324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'810","last_scrub_stamp":"2023-10-30T13:59:27.812198+0000","last_deep_scrub":"56'810","last_deep_scrub_stamp":"2023-10-30T13:59:27.812198+0000","last_clean_scrub_stamp":"2023-10-30T13:59:27.812198+0000","objects_scrubbed":155,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:19.443546+0000","scrub_duration":0.136984935,"objects_trimmed":0,"snaptrim_duration":0.001735594,"stat_sum":{"num_bytes":6415761,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":108,"num_read_kb":4114,"num_write":644,"num_write_kb":12898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"56'283","reported_seq":370,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:11.889445+0000","last_change":"2023-10-30T13:59:11.889543+0000","last_active":"2023-10-30T13:59:11.889445+0000","last_peered":"2023-10-30T13:59:11.889445+0000","last_clean":"2023-10-30T13:59:11.889445+0000","last_became_active":"2023-10-30T13:35:07.294048+0000","last_became_peered":"2023-10-30T13:35:07.294048+0000","last_unstale":"2023-10-30T13:59:11.889445+0000","last_undegraded":"2023-10-30T13:59:11.889445+0000","last_fullsized":"2023-10-30T13:59:11.889445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'283","last_scrub_stamp":"2023-10-30T13:59:11.889356+0000","last_deep_scrub":"56'283","last_deep_scrub_stamp":"2023-10-30T13:59:11.889356+0000","last_clean_scrub_stamp":"2023-10-30T13:59:11.889356+0000","objects_scrubbed":6,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:04.195980+0000","scrub_duration":0.13641584700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":698,"num_write":517,"num_write_kb":2842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":36848,"num_omap_keys":69,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"56'769","reported_seq":2985,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:35.232420+0000","last_change":"2023-10-30T13:59:35.232463+0000","last_active":"2023-10-30T13:59:35.232420+0000","last_peered":"2023-10-30T13:59:35.232420+0000","last_clean":"2023-10-30T13:59:35.232420+0000","last_became_active":"2023-10-30T13:35:07.794050+0000","last_became_peered":"2023-10-30T13:35:07.794050+0000","last_unstale":"2023-10-30T13:59:35.232420+0000","last_undegraded":"2023-10-30T13:59:35.232420+0000","last_fullsized":"2023-10-30T13:59:35.232420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'769","last_scrub_stamp":"2023-10-30T13:59:35.232349+0000","last_deep_scrub":"56'769","last_deep_scrub_stamp":"2023-10-30T13:59:35.232349+0000","last_clean_scrub_stamp":"2023-10-30T13:59:35.232349+0000","objects_scrubbed":151,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:25:20.721025+0000","scrub_duration":0.212346017,"objects_trimmed":0,"snaptrim_duration":0.002858771,"stat_sum":{"num_bytes":14417139,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":106,"num_read_kb":4110,"num_write":585,"num_write_kb":24357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"56'124","reported_seq":210,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.849710+0000","last_change":"2023-10-30T13:59:17.849829+0000","last_active":"2023-10-30T13:59:17.849710+0000","last_peered":"2023-10-30T13:59:17.849710+0000","last_clean":"2023-10-30T13:59:17.849710+0000","last_became_active":"2023-10-30T13:35:07.295661+0000","last_became_peered":"2023-10-30T13:35:07.295661+0000","last_unstale":"2023-10-30T13:59:17.849710+0000","last_undegraded":"2023-10-30T13:59:17.849710+0000","last_fullsized":"2023-10-30T13:59:17.849710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'124","last_scrub_stamp":"2023-10-30T13:59:17.849515+0000","last_deep_scrub":"56'124","last_deep_scrub_stamp":"2023-10-30T13:59:17.849515+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.849515+0000","objects_scrubbed":7,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:54.554158+0000","scrub_duration":0.025438601000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":20522,"num_write":112,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30984,"num_omap_keys":57,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"56'824","reported_seq":3046,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.809876+0000","last_change":"2023-10-30T13:59:17.809939+0000","last_active":"2023-10-30T13:59:17.809876+0000","last_peered":"2023-10-30T13:59:17.809876+0000","last_clean":"2023-10-30T13:59:17.809876+0000","last_became_active":"2023-10-30T13:35:07.751757+0000","last_became_peered":"2023-10-30T13:35:07.751757+0000","last_unstale":"2023-10-30T13:59:17.809876+0000","last_undegraded":"2023-10-30T13:59:17.809876+0000","last_fullsized":"2023-10-30T13:59:17.809876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'824","last_scrub_stamp":"2023-10-30T13:59:17.809790+0000","last_deep_scrub":"56'824","last_deep_scrub_stamp":"2023-10-30T13:59:17.809790+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.809790+0000","objects_scrubbed":162,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:00.364334+0000","scrub_duration":0.16950805699999999,"objects_trimmed":0,"snaptrim_duration":0.0027315170000000001,"stat_sum":{"num_bytes":18657914,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":112,"num_read_kb":8214,"num_write":646,"num_write_kb":32163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"56'148","reported_seq":226,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:17.625271+0000","last_change":"2023-10-30T13:59:17.625403+0000","last_active":"2023-10-30T13:59:17.625271+0000","last_peered":"2023-10-30T13:59:17.625271+0000","last_clean":"2023-10-30T13:59:17.625271+0000","last_became_active":"2023-10-30T13:35:07.298937+0000","last_became_peered":"2023-10-30T13:35:07.298937+0000","last_unstale":"2023-10-30T13:59:17.625271+0000","last_undegraded":"2023-10-30T13:59:17.625271+0000","last_fullsized":"2023-10-30T13:59:17.625271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'148","last_scrub_stamp":"2023-10-30T13:59:17.625064+0000","last_deep_scrub":"56'148","last_deep_scrub_stamp":"2023-10-30T13:59:17.625064+0000","last_clean_scrub_stamp":"2023-10-30T13:59:17.625064+0000","objects_scrubbed":3,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:01.340603+0000","scrub_duration":0.011205155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":30625,"num_write":106,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25608,"num_omap_keys":47,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"56'788","reported_seq":2995,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.814579+0000","last_change":"2023-10-30T13:59:13.814677+0000","last_active":"2023-10-30T13:59:13.814579+0000","last_peered":"2023-10-30T13:59:13.814579+0000","last_clean":"2023-10-30T13:59:13.814579+0000","last_became_active":"2023-10-30T13:35:07.758413+0000","last_became_peered":"2023-10-30T13:35:07.758413+0000","last_unstale":"2023-10-30T13:59:13.814579+0000","last_undegraded":"2023-10-30T13:59:13.814579+0000","last_fullsized":"2023-10-30T13:59:13.814579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'788","last_scrub_stamp":"2023-10-30T13:59:13.814451+0000","last_deep_scrub":"56'788","last_deep_scrub_stamp":"2023-10-30T13:59:13.814451+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.814451+0000","objects_scrubbed":145,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:05.475962+0000","scrub_duration":0.26681934600000001,"objects_trimmed":0,"snaptrim_duration":0.0075826330000000001,"stat_sum":{"num_bytes":5755271,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":98,"num_read_kb":7,"num_write":639,"num_write_kb":16186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'946","reported_seq":1021,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:16.799532+0000","last_change":"2023-10-30T13:59:16.799588+0000","last_active":"2023-10-30T13:59:16.799532+0000","last_peered":"2023-10-30T13:59:16.799532+0000","last_clean":"2023-10-30T13:59:16.799532+0000","last_became_active":"2023-10-30T13:35:07.284933+0000","last_became_peered":"2023-10-30T13:35:07.284933+0000","last_unstale":"2023-10-30T13:59:16.799532+0000","last_undegraded":"2023-10-30T13:59:16.799532+0000","last_fullsized":"2023-10-30T13:59:16.799532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'946","last_scrub_stamp":"2023-10-30T13:59:16.799459+0000","last_deep_scrub":"56'946","last_deep_scrub_stamp":"2023-10-30T13:59:16.799459+0000","last_clean_scrub_stamp":"2023-10-30T13:59:16.799459+0000","objects_scrubbed":6,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:25.410921+0000","scrub_duration":0.051098538999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":20487,"num_write":909,"num_write_kb":8631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16326,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"56'874","reported_seq":3095,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:36.184240+0000","last_change":"2023-10-30T13:59:36.184295+0000","last_active":"2023-10-30T13:59:36.184240+0000","last_peered":"2023-10-30T13:59:36.184240+0000","last_clean":"2023-10-30T13:59:36.184240+0000","last_became_active":"2023-10-30T13:35:07.753411+0000","last_became_peered":"2023-10-30T13:35:07.753411+0000","last_unstale":"2023-10-30T13:59:36.184240+0000","last_undegraded":"2023-10-30T13:59:36.184240+0000","last_fullsized":"2023-10-30T13:59:36.184240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'874","last_scrub_stamp":"2023-10-30T13:59:36.184161+0000","last_deep_scrub":"56'874","last_deep_scrub_stamp":"2023-10-30T13:59:36.184161+0000","last_clean_scrub_stamp":"2023-10-30T13:59:36.184161+0000","objects_scrubbed":147,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:44:59.440315+0000","scrub_duration":0.16968080599999999,"objects_trimmed":0,"snaptrim_duration":0.0029726169999999999,"stat_sum":{"num_bytes":19889578,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":112,"num_read_kb":12306,"num_write":688,"num_write_kb":31224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"56'485","reported_seq":567,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.778391+0000","last_change":"2023-10-30T13:59:15.778438+0000","last_active":"2023-10-30T13:59:15.778391+0000","last_peered":"2023-10-30T13:59:15.778391+0000","last_clean":"2023-10-30T13:59:15.778391+0000","last_became_active":"2023-10-30T13:35:07.284239+0000","last_became_peered":"2023-10-30T13:35:07.284239+0000","last_unstale":"2023-10-30T13:59:15.778391+0000","last_undegraded":"2023-10-30T13:59:15.778391+0000","last_fullsized":"2023-10-30T13:59:15.778391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'485","last_scrub_stamp":"2023-10-30T13:59:15.778326+0000","last_deep_scrub":"56'485","last_deep_scrub_stamp":"2023-10-30T13:59:15.778326+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.778326+0000","objects_scrubbed":8,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:34.958856+0000","scrub_duration":0.04450407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":53262,"num_write":506,"num_write_kb":12694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35586,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"56'849","reported_seq":3070,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:44.494907+0000","last_change":"2023-10-30T13:59:44.494969+0000","last_active":"2023-10-30T13:59:44.494907+0000","last_peered":"2023-10-30T13:59:44.494907+0000","last_clean":"2023-10-30T13:59:44.494907+0000","last_became_active":"2023-10-30T13:35:07.755017+0000","last_became_peered":"2023-10-30T13:35:07.755017+0000","last_unstale":"2023-10-30T13:59:44.494907+0000","last_undegraded":"2023-10-30T13:59:44.494907+0000","last_fullsized":"2023-10-30T13:59:44.494907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'849","last_scrub_stamp":"2023-10-30T13:59:44.494821+0000","last_deep_scrub":"56'849","last_deep_scrub_stamp":"2023-10-30T13:59:44.494821+0000","last_clean_scrub_stamp":"2023-10-30T13:59:44.494821+0000","objects_scrubbed":158,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:02.881643+0000","scrub_duration":0.38383446399999999,"objects_trimmed":0,"snaptrim_duration":0.002600925,"stat_sum":{"num_bytes":9733947,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":111,"num_read_kb":4112,"num_write":647,"num_write_kb":24811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"56'1293","reported_seq":1390,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:25.628077+0000","last_change":"2023-10-30T13:59:25.628077+0000","last_active":"2023-10-30T13:59:25.628077+0000","last_peered":"2023-10-30T13:59:25.628077+0000","last_clean":"2023-10-30T13:59:25.628077+0000","last_became_active":"2023-10-30T13:35:07.300044+0000","last_became_peered":"2023-10-30T13:35:07.300044+0000","last_unstale":"2023-10-30T13:59:25.628077+0000","last_undegraded":"2023-10-30T13:59:25.628077+0000","last_fullsized":"2023-10-30T13:59:25.628077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1293","last_scrub_stamp":"2023-10-30T13:59:25.627729+0000","last_deep_scrub":"56'1293","last_deep_scrub_stamp":"2023-10-30T13:59:25.627729+0000","last_clean_scrub_stamp":"2023-10-30T13:59:25.627729+0000","objects_scrubbed":8,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:46:32.130653+0000","scrub_duration":0.082082413000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":81960,"num_write":1252,"num_write_kb":21130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27956,"num_omap_keys":18,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"56'873","reported_seq":3091,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:35.136342+0000","last_change":"2023-10-30T13:59:35.136398+0000","last_active":"2023-10-30T13:59:35.136342+0000","last_peered":"2023-10-30T13:59:35.136342+0000","last_clean":"2023-10-30T13:59:35.136342+0000","last_became_active":"2023-10-30T13:35:07.779492+0000","last_became_peered":"2023-10-30T13:35:07.779492+0000","last_unstale":"2023-10-30T13:59:35.136342+0000","last_undegraded":"2023-10-30T13:59:35.136342+0000","last_fullsized":"2023-10-30T13:59:35.136342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'873","last_scrub_stamp":"2023-10-30T13:59:35.136239+0000","last_deep_scrub":"56'873","last_deep_scrub_stamp":"2023-10-30T13:59:35.136239+0000","last_clean_scrub_stamp":"2023-10-30T13:59:35.136239+0000","objects_scrubbed":158,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:12:25.717527+0000","scrub_duration":0.34861871900000002,"objects_trimmed":0,"snaptrim_duration":0.0051869250000000002,"stat_sum":{"num_bytes":18626842,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":5015,"num_write":667,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"56'799","reported_seq":3030,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:43.912906+0000","last_change":"2023-10-30T13:59:43.912906+0000","last_active":"2023-10-30T13:59:43.912906+0000","last_peered":"2023-10-30T13:59:43.912906+0000","last_clean":"2023-10-30T13:59:43.912906+0000","last_became_active":"2023-10-30T13:35:07.799527+0000","last_became_peered":"2023-10-30T13:35:07.799527+0000","last_unstale":"2023-10-30T13:59:43.912906+0000","last_undegraded":"2023-10-30T13:59:43.912906+0000","last_fullsized":"2023-10-30T13:59:43.912906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'799","last_scrub_stamp":"2023-10-30T13:59:43.912716+0000","last_deep_scrub":"56'799","last_deep_scrub_stamp":"2023-10-30T13:59:43.912716+0000","last_clean_scrub_stamp":"2023-10-30T13:59:43.912716+0000","objects_scrubbed":142,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:53.615287+0000","scrub_duration":0.26467791000000002,"objects_trimmed":0,"snaptrim_duration":0.0061609869999999997,"stat_sum":{"num_bytes":14245271,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":123,"num_read_kb":12313,"num_write":620,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"56'235","reported_seq":317,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:24.601686+0000","last_change":"2023-10-30T13:59:24.601686+0000","last_active":"2023-10-30T13:59:24.601686+0000","last_peered":"2023-10-30T13:59:24.601686+0000","last_clean":"2023-10-30T13:59:24.601686+0000","last_became_active":"2023-10-30T13:35:07.299078+0000","last_became_peered":"2023-10-30T13:35:07.299078+0000","last_unstale":"2023-10-30T13:59:24.601686+0000","last_undegraded":"2023-10-30T13:59:24.601686+0000","last_fullsized":"2023-10-30T13:59:24.601686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'235","last_scrub_stamp":"2023-10-30T13:59:24.601285+0000","last_deep_scrub":"56'235","last_deep_scrub_stamp":"2023-10-30T13:59:24.601285+0000","last_clean_scrub_stamp":"2023-10-30T13:59:24.601285+0000","objects_scrubbed":4,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:28.578041+0000","scrub_duration":0.016429274000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":24606,"num_write":206,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35337,"num_omap_keys":63,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"56'894","reported_seq":3116,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:46.931391+0000","last_change":"2023-10-30T13:59:46.931391+0000","last_active":"2023-10-30T13:59:46.931391+0000","last_peered":"2023-10-30T13:59:46.931391+0000","last_clean":"2023-10-30T13:59:46.931391+0000","last_became_active":"2023-10-30T13:35:07.789635+0000","last_became_peered":"2023-10-30T13:35:07.789635+0000","last_unstale":"2023-10-30T13:59:46.931391+0000","last_undegraded":"2023-10-30T13:59:46.931391+0000","last_fullsized":"2023-10-30T13:59:46.931391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'894","last_scrub_stamp":"2023-10-30T13:59:46.931198+0000","last_deep_scrub":"56'894","last_deep_scrub_stamp":"2023-10-30T13:59:46.931198+0000","last_clean_scrub_stamp":"2023-10-30T13:59:46.931198+0000","objects_scrubbed":181,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:39.907025+0000","scrub_duration":0.31800989200000002,"objects_trimmed":0,"snaptrim_duration":0.0061457359999999997,"stat_sum":{"num_bytes":11313833,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":112,"num_read_kb":16,"num_write":670,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"56'957","reported_seq":1061,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:30.546377+0000","last_change":"2023-10-30T13:59:30.546377+0000","last_active":"2023-10-30T13:59:30.546377+0000","last_peered":"2023-10-30T13:59:30.546377+0000","last_clean":"2023-10-30T13:59:30.546377+0000","last_became_active":"2023-10-30T13:35:07.300760+0000","last_became_peered":"2023-10-30T13:35:07.300760+0000","last_unstale":"2023-10-30T13:59:30.546377+0000","last_undegraded":"2023-10-30T13:59:30.546377+0000","last_fullsized":"2023-10-30T13:59:30.546377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'957","last_scrub_stamp":"2023-10-30T13:59:30.546025+0000","last_deep_scrub":"56'957","last_deep_scrub_stamp":"2023-10-30T13:59:30.546025+0000","last_clean_scrub_stamp":"2023-10-30T13:59:30.546025+0000","objects_scrubbed":7,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:04.224187+0000","scrub_duration":0.051378422999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":103644,"num_write":908,"num_write_kb":16901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17921,"num_omap_keys":34,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"56'829","reported_seq":3062,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:26.753762+0000","last_change":"2023-10-30T13:59:26.753762+0000","last_active":"2023-10-30T13:59:26.753762+0000","last_peered":"2023-10-30T13:59:26.753762+0000","last_clean":"2023-10-30T13:59:26.753762+0000","last_became_active":"2023-10-30T13:35:07.789767+0000","last_became_peered":"2023-10-30T13:35:07.789767+0000","last_unstale":"2023-10-30T13:59:26.753762+0000","last_undegraded":"2023-10-30T13:59:26.753762+0000","last_fullsized":"2023-10-30T13:59:26.753762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'829","last_scrub_stamp":"2023-10-30T13:59:26.753633+0000","last_deep_scrub":"56'829","last_deep_scrub_stamp":"2023-10-30T13:59:26.753633+0000","last_clean_scrub_stamp":"2023-10-30T13:59:26.753633+0000","objects_scrubbed":184,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:01:43.957619+0000","scrub_duration":0.27683662399999998,"objects_trimmed":0,"snaptrim_duration":0.0029730300000000002,"stat_sum":{"num_bytes":19886092,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":121,"num_read_kb":4113,"num_write":647,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"56'590","reported_seq":679,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:13.115956+0000","last_change":"2023-10-30T13:59:13.116019+0000","last_active":"2023-10-30T13:59:13.115956+0000","last_peered":"2023-10-30T13:59:13.115956+0000","last_clean":"2023-10-30T13:59:13.115956+0000","last_became_active":"2023-10-30T13:35:07.284299+0000","last_became_peered":"2023-10-30T13:35:07.284299+0000","last_unstale":"2023-10-30T13:59:13.115956+0000","last_undegraded":"2023-10-30T13:59:13.115956+0000","last_fullsized":"2023-10-30T13:59:13.115956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'590","last_scrub_stamp":"2023-10-30T13:59:13.115857+0000","last_deep_scrub":"56'590","last_deep_scrub_stamp":"2023-10-30T13:59:13.115857+0000","last_clean_scrub_stamp":"2023-10-30T13:59:13.115857+0000","objects_scrubbed":4,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:55.093851+0000","scrub_duration":0.036041838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":72355,"num_write":540,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14742,"num_omap_keys":27,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"56'816","reported_seq":3025,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:45.340376+0000","last_change":"2023-10-30T13:59:45.340430+0000","last_active":"2023-10-30T13:59:45.340376+0000","last_peered":"2023-10-30T13:59:45.340376+0000","last_clean":"2023-10-30T13:59:45.340376+0000","last_became_active":"2023-10-30T13:35:07.782019+0000","last_became_peered":"2023-10-30T13:35:07.782019+0000","last_unstale":"2023-10-30T13:59:45.340376+0000","last_undegraded":"2023-10-30T13:59:45.340376+0000","last_fullsized":"2023-10-30T13:59:45.340376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'816","last_scrub_stamp":"2023-10-30T13:59:45.340302+0000","last_deep_scrub":"56'816","last_deep_scrub_stamp":"2023-10-30T13:59:45.340302+0000","last_clean_scrub_stamp":"2023-10-30T13:59:45.340302+0000","objects_scrubbed":156,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:39.933055+0000","scrub_duration":0.26337302099999998,"objects_trimmed":0,"snaptrim_duration":0.0026777319999999999,"stat_sum":{"num_bytes":7974576,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":98,"num_read_kb":4107,"num_write":628,"num_write_kb":17606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"56'364","reported_seq":469,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-10-30T13:59:15.646200+0000","last_change":"2023-10-30T13:59:15.646200+0000","last_active":"2023-10-30T13:59:15.646200+0000","last_peered":"2023-10-30T13:59:15.646200+0000","last_clean":"2023-10-30T13:59:15.646200+0000","last_became_active":"2023-10-30T13:35:07.293391+0000","last_became_peered":"2023-10-30T13:35:07.293391+0000","last_unstale":"2023-10-30T13:59:15.646200+0000","last_undegraded":"2023-10-30T13:59:15.646200+0000","last_fullsized":"2023-10-30T13:59:15.646200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'364","last_scrub_stamp":"2023-10-30T13:59:15.645995+0000","last_deep_scrub":"56'364","last_deep_scrub_stamp":"2023-10-30T13:59:15.645995+0000","last_clean_scrub_stamp":"2023-10-30T13:59:15.645995+0000","objects_scrubbed":11,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:40.607663+0000","scrub_duration":0.031506254999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4418509,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":57427,"num_write":339,"num_write_kb":8706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35486,"num_omap_keys":15,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":925185184,"num_objects":10324,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":7268,"num_read_kb":400185,"num_write":41527,"num_write_kb":1950152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1902518272,"data_stored":1848830272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53316,"ondisk_log_size":53316,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":260427066,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":1553,"num_read_kb":1442867,"num_write":16575,"num_write_kb":377808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":116,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":782235,"num_omap_keys":1350,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":521019392,"data_stored":520854132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17547,"ondisk_log_size":17547,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509748,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433140,"kb_used_data":333932,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93938700,"statfs":{"total":96636764160,"available":96193228800,"internally_reserved":0,"allocated":341946368,"data_stored":333745462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":101579211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.84299999999999997,"15min":0.81000000000000005},"min":{"1min":0.51700000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.353,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.2749999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.94899999999999995,"15min":0.91000000000000003},"min":{"1min":0.52700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2709999999999999,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.82099999999999995,"15min":0.90900000000000003},"min":{"1min":0.503,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.04,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":1.097},{"interface":"front","average":{"1min":1.002,"5min":0.97899999999999998,"15min":0.92500000000000004},"min":{"1min":0.58799999999999997,"5min":0.45200000000000001,"15min":0.33400000000000002},"max":{"1min":1.357,"5min":1.577,"15min":1.625},"last":0.81200000000000006}]},{"osd":2,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.018,"15min":0.97899999999999998},"min":{"1min":0.622,"5min":0.51300000000000001,"15min":0.32000000000000001},"max":{"1min":1.4410000000000001,"5min":1.468,"15min":1.468},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.94899999999999995,"15min":0.96099999999999997},"min":{"1min":0.65000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.6279999999999999,"5min":3.7810000000000001,"15min":4.1120000000000001},"last":0.92800000000000005}]},{"osd":3,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.1120000000000001,"15min":1.0900000000000001},"min":{"1min":0.78600000000000003,"5min":0.55300000000000005,"15min":0.45900000000000002},"max":{"1min":1.6659999999999999,"5min":1.7669999999999999,"15min":4.2069999999999999},"last":1.036},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.605,"15min":1.1950000000000001},"min":{"1min":0.50600000000000001,"5min":0.47199999999999998,"15min":0.44400000000000001},"max":{"1min":1.3220000000000001,"5min":59.844999999999999,"15min":59.844999999999999},"last":0.98899999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.163,"15min":1.1080000000000001},"min":{"1min":0.98099999999999998,"5min":0.623,"15min":0.38100000000000001},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":3.1059999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.3440000000000001,"15min":1.298},"min":{"1min":1.1930000000000001,"5min":0.754,"15min":0.42399999999999999},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":3.198},"last":1.5269999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.012,"15min":1.0860000000000001},"min":{"1min":0.66200000000000003,"5min":0.53100000000000003,"15min":0.40300000000000002},"max":{"1min":1.758,"5min":3.9510000000000001,"15min":4.2759999999999998},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.2809999999999999,"15min":1.258},"min":{"1min":1.0680000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":3.4380000000000002},"last":1.4430000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92700000000000005,"15min":1.0760000000000001},"min":{"1min":0.754,"5min":0.48999999999999999,"15min":0.48399999999999999},"max":{"1min":1.351,"5min":1.429,"15min":3.9630000000000001},"last":1.135},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.133,"15min":1.2090000000000001},"min":{"1min":0.75,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.9019999999999999,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":1.2310000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509747,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429880,"kb_used_data":318192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93941960,"statfs":{"total":96636764160,"available":96196567040,"internally_reserved":0,"allocated":325828608,"data_stored":317592434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":114358728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47799999999999998,"15min":0.52100000000000002},"min":{"1min":0.27100000000000002,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.74099999999999999,"5min":0.78600000000000003,"15min":9.3320000000000007},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.0349999999999999,"15min":0.72899999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.308},"max":{"1min":0.65800000000000003,"5min":38.082999999999998,"15min":38.082999999999998},"last":0.44}]},{"osd":1,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.69599999999999995,"15min":0.61399999999999999},"min":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.30099999999999999},"max":{"1min":0.67700000000000005,"5min":11.599,"15min":11.599},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.70599999999999996,"15min":0.66000000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.72199999999999998,"5min":15.715999999999999,"15min":15.715999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.73199999999999998,"15min":0.67300000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.88600000000000001,"5min":11.646000000000001,"15min":12.351000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.76200000000000001,"15min":0.65400000000000003},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":15.641999999999999,"15min":15.641999999999999},"last":0.48899999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.012,"15min":0.92300000000000004},"min":{"1min":0.59799999999999998,"5min":0.55400000000000005,"15min":0.51600000000000001},"max":{"1min":1.0620000000000001,"5min":15.759,"15min":15.759},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.752,"5min":1.268,"15min":1.054},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.296,"5min":38.137999999999998,"15min":38.137999999999998},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.4059999999999999,"15min":1.1000000000000001},"min":{"1min":0.626,"5min":0.59299999999999997,"15min":0.52600000000000002},"max":{"1min":1.0469999999999999,"5min":40.808,"15min":40.808},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.032,"15min":0.95899999999999996},"min":{"1min":0.56100000000000005,"5min":0.55800000000000005,"15min":0.55300000000000005},"max":{"1min":1.127,"5min":16.498000000000001,"15min":19.835999999999999},"last":0.90100000000000002}]},{"osd":6,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.96999999999999997,"15min":0.92000000000000004},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.53500000000000003},"max":{"1min":1.085,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.024,"15min":0.96299999999999997},"min":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":1.1870000000000001,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1020000000000001,"15min":0.96499999999999997},"min":{"1min":0.66000000000000003,"5min":0.60599999999999998,"15min":0.59999999999999998},"max":{"1min":1.1240000000000001,"5min":16.082999999999998,"15min":16.082999999999998},"last":0.877},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.569,"15min":1.113},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.2370000000000001,"5min":52.902999999999999,"15min":52.902999999999999},"last":0.85899999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258464,"kb_used_data":187096,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94113376,"statfs":{"total":96636764160,"available":96372097024,"internally_reserved":0,"allocated":191586304,"data_stored":186614303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.29399999999999998,"5min":0.251,"15min":0.246},"max":{"1min":2.242,"5min":3.407,"15min":3.8969999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.9430000000000001,"5min":0.85899999999999999,"15min":0.67300000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33300000000000002},"max":{"1min":23.335999999999999,"5min":23.335999999999999,"15min":23.335999999999999},"last":0.69099999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.56299999999999994,"15min":0.62},"min":{"1min":0.33200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.5919999999999996,"5min":5.5750000000000002,"15min":36.109000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.337,"5min":0.73099999999999998,"15min":0.627},"min":{"1min":0.311,"5min":0.311,"15min":0.28599999999999998},"max":{"1min":13.584,"5min":13.584,"15min":13.584},"last":0.60999999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.65300000000000002,"15min":0.70199999999999996},"min":{"1min":0.437,"5min":0.35899999999999999,"15min":0.29199999999999998},"max":{"1min":2.339,"5min":5.532,"15min":35.890999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.431,"5min":0.83099999999999996,"15min":0.75600000000000001},"min":{"1min":0.47699999999999998,"5min":0.42899999999999999,"15min":0.371},"max":{"1min":13.706,"5min":13.706,"15min":22.846},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":3.8690000000000002,"5min":1.446,"15min":1.0429999999999999},"min":{"1min":0.66200000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":48.710999999999999,"5min":48.710999999999999,"15min":48.710999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.86399999999999999,"15min":0.85499999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.302,"5min":5.6059999999999999,"15min":7.8220000000000001},"last":0.84699999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":2.504,"5min":1.1919999999999999,"15min":0.98799999999999999},"min":{"1min":0.58499999999999996,"5min":0.442,"15min":0.442},"max":{"1min":26.788,"5min":26.788,"15min":26.788},"last":0.873},{"interface":"front","average":{"1min":1.71,"5min":1.0580000000000001,"15min":0.92500000000000004},"min":{"1min":0.66900000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":13.654999999999999,"5min":13.654999999999999,"15min":13.654999999999999},"last":1.0629999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.222,"15min":0.96299999999999997},"min":{"1min":0.62,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.379,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.105,"15min":0.97699999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56599999999999995},"max":{"1min":1.6220000000000001,"5min":15.975,"15min":15.975},"last":1.0429999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:59:34 2023","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.087,"15min":0.96399999999999997},"min":{"1min":0.70499999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":13.747999999999999,"5min":13.747999999999999,"15min":13.747999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.738,"5min":1.0860000000000001,"15min":0.97199999999999998},"min":{"1min":0.61399999999999999,"5min":0.53900000000000003,"15min":0.53700000000000003},"max":{"1min":13.779999999999999,"5min":13.779999999999999,"15min":13.779999999999999},"last":1.099}]}]},{"osd":1,"up_from":15,"seq":64424509746,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357220,"kb_used_data":245596,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94014620,"statfs":{"total":96636764160,"available":96270970880,"internally_reserved":0,"allocated":251490304,"data_stored":246176700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114293190},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.51000000000000001,"15min":0.46000000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.7469999999999999,"5min":4.7469999999999999,"15min":4.7469999999999999},"last":0.441},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":6.577,"5min":6.577,"15min":22.149000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":1.905,"5min":0.76900000000000002,"15min":0.56399999999999995},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":25.263999999999999,"5min":25.263999999999999,"15min":25.263999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.66600000000000004,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":6.056,"5min":6.056,"15min":6.056},"last":0.5}]},{"osd":3,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.64100000000000001,"15min":0.78100000000000003},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":54.530000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.72399999999999998,"15min":0.71499999999999997},"min":{"1min":0.437,"5min":0.40100000000000002,"15min":0.34100000000000003},"max":{"1min":7.1070000000000002,"5min":7.1070000000000002,"15min":25.457000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":0.97299999999999998,"15min":0.873},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":14.019,"5min":14.019,"15min":14.019},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.1319999999999999,"5min":0.88,"15min":0.81999999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":4.71,"5min":4.71,"15min":4.71},"last":0.83899999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":0.95299999999999996,"15min":0.91300000000000003},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.505},"max":{"1min":11.82,"5min":11.82,"15min":11.82},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.4119999999999999,"5min":0.95499999999999996,"15min":0.879},"min":{"1min":0.63300000000000001,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":10.568,"5min":10.568,"15min":10.568},"last":0.84999999999999998}]},{"osd":6,"last update":"Mon Oct 30 13:59:36 2023","interfaces":[{"interface":"back","average":{"1min":2.9420000000000002,"5min":1.2350000000000001,"15min":0.97799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.497},"max":{"1min":38.530999999999999,"5min":38.530999999999999,"15min":38.530999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.7869999999999999,"5min":1.0249999999999999,"15min":0.91200000000000003},"min":{"1min":0.64100000000000001,"5min":0.56299999999999994,"15min":0.47999999999999998},"max":{"1min":14.568,"5min":14.568,"15min":14.568},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:59:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.0960000000000001,"15min":1.0089999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.506,"5min":12.221,"15min":12.221},"last":0.752},{"interface":"front","average":{"1min":1.044,"5min":1.0629999999999999,"15min":0.95399999999999996},"min":{"1min":0.66700000000000004,"5min":0.61699999999999999,"15min":0.57099999999999995},"max":{"1min":3.7360000000000002,"5min":12.153,"15min":12.153},"last":0.92100000000000004}]}]},{"osd":6,"up_from":14,"seq":60129542452,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332636,"kb_used_data":247380,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94039204,"statfs":{"total":96636764160,"available":96296144896,"internally_reserved":0,"allocated":253317120,"data_stored":248156410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87292359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.1299999999999999,"15min":1.0249999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.41999999999999998},"max":{"1min":1.1879999999999999,"5min":19.741,"15min":19.741},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.1359999999999999,"15min":1.002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.34899999999999998},"max":{"1min":1.742,"5min":13.863,"15min":13.863},"last":1.3740000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.92500000000000004,"15min":0.86799999999999999},"min":{"1min":0.52400000000000002,"5min":0.496,"15min":0.48299999999999998},"max":{"1min":1.1839999999999999,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.179,"15min":1.0369999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.4279999999999999,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":1.234}]},{"osd":2,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96299999999999997,"15min":0.97999999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.7130000000000001,"5min":3.5329999999999999,"15min":3.7709999999999999},"last":1.111},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0760000000000001,"15min":1.079},"min":{"1min":0.60199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7549999999999999,"5min":3.407,"15min":4.6840000000000002},"last":1.3160000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.036,"15min":1.0189999999999999},"min":{"1min":0.64300000000000002,"5min":0.52800000000000002,"15min":0.46700000000000003},"max":{"1min":2.0059999999999998,"5min":3.7999999999999998,"15min":4.7629999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.224,"5min":1.1870000000000001,"15min":1.109},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.909,"5min":3.968,"15min":3.968},"last":1.425}]},{"osd":4,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1739999999999999,"15min":1.202},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5309999999999999,"5min":3.8900000000000001,"15min":16.204000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2250000000000001,"15min":1.266},"min":{"1min":0.70599999999999996,"5min":0.622,"15min":0.42699999999999999},"max":{"1min":2.0899999999999999,"5min":19.398,"15min":43.198},"last":1.476}]},{"osd":5,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96399999999999997,"15min":1.0409999999999999},"min":{"1min":0.51200000000000001,"5min":0.47699999999999998,"15min":0.41699999999999998},"max":{"1min":1.484,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.3129999999999999,"15min":1.3069999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.58599999999999997},"max":{"1min":2.1819999999999999,"5min":3.7269999999999999,"15min":3.802},"last":1.524}]},{"osd":7,"last update":"Mon Oct 30 13:59:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.8340000000000001,"15min":1.5900000000000001},"min":{"1min":0.65200000000000002,"5min":0.58899999999999997,"15min":0.55400000000000005},"max":{"1min":1.6559999999999999,"5min":70.335999999999999,"15min":73.228999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.661,"15min":1.629},"min":{"1min":0.625,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6100000000000001,"5min":42.164999999999999,"15min":73.289000000000001},"last":1.1539999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542450,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425100,"kb_used_data":335620,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93946740,"statfs":{"total":96636764160,"available":96201461760,"internally_reserved":0,"allocated":343674880,"data_stored":335552063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.88300000000000001,"15min":0.872},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.40400000000000003},"max":{"1min":1.7350000000000001,"5min":3.9159999999999999,"15min":5.0599999999999996},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":0.86699999999999999,"15min":0.876},"min":{"1min":0.47699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.95499999999999996,"15min":0.876},"min":{"1min":0.52300000000000002,"5min":0.40600000000000003,"15min":0.33600000000000002},"max":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"last":1.276},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.85499999999999998,"15min":0.90200000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.5300000000000002,"5min":4.5300000000000002,"15min":4.5300000000000002},"last":0.72699999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.997,"15min":0.96899999999999997},"min":{"1min":0.59499999999999997,"5min":0.55000000000000004,"15min":0.51100000000000001},"max":{"1min":5.109,"5min":5.109,"15min":5.3710000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.361,"5min":1.1639999999999999,"15min":1.0780000000000001},"min":{"1min":0.70099999999999996,"5min":0.63800000000000001,"15min":0.53200000000000003},"max":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.96999999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1200000000000001,"15min":1.0760000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.2709999999999999,"5min":5.2709999999999999,"15min":5.476},"last":0.875},{"interface":"front","average":{"1min":1.163,"5min":1.014,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.6950000000000003,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":1.2010000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.0009999999999999,"15min":1.004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":5.0229999999999997,"5min":5.0229999999999997,"15min":5.0229999999999997},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.319,"5min":1.109,"15min":1.123},"min":{"1min":0.752,"5min":0.49199999999999999,"15min":0.41799999999999998},"max":{"1min":4.843,"5min":4.843,"15min":4.843},"last":0.91900000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2250000000000001,"15min":1.1930000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.6029999999999998},"last":1.127},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.1819999999999999,"15min":1.1879999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41899999999999998},"max":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":1.325}]},{"osd":7,"last update":"Mon Oct 30 13:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":1.24,"15min":1.2629999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.45100000000000001},"max":{"1min":10.884,"5min":10.884,"15min":10.884},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.272,"15min":1.3},"min":{"1min":0.5,"5min":0.5,"15min":0.48499999999999999},"max":{"1min":5.335,"5min":5.335,"15min":5.335},"last":1.1659999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542450,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":430672,"kb_used_data":337032,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93941168,"statfs":{"total":96636764160,"available":96195756032,"internally_reserved":0,"allocated":345120768,"data_stored":336789947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.872},"min":{"1min":0.45000000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.3159999999999998,"5min":8.7680000000000007,"15min":8.7680000000000007},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.329,"5min":0.999,"15min":0.89600000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":10.52,"5min":10.52,"15min":10.52},"last":0.69799999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.82299999999999995,"15min":0.88900000000000001},"min":{"1min":0.51500000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":1.236,"15min":5.8010000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.016,"5min":0.86499999999999999,"15min":0.874},"min":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.77900000000000003}]},{"osd":2,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.55400000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":3.323},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0329999999999999,"15min":0.98199999999999998},"min":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.0549999999999997,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":0.89000000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.105,"15min":1.0209999999999999},"min":{"1min":0.69299999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.268,"5min":8.0570000000000004,"15min":8.0570000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.0529999999999999,"15min":0.995},"min":{"1min":0.66800000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1459999999999999,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:59:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":1.1599999999999999,"15min":1.103},"min":{"1min":0.54200000000000004,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":12.798,"5min":12.798,"15min":12.798},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.032,"5min":1.1459999999999999,"15min":1.169},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5229999999999999,"5min":4.0609999999999999,"15min":8.5399999999999991},"last":1.202}]},{"osd":6,"last update":"Mon Oct 30 13:59:19 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.2010000000000001,"15min":1.0900000000000001},"min":{"1min":0.56000000000000005,"5min":0.503,"15min":0.45100000000000001},"max":{"1min":1.3680000000000001,"5min":16.989999999999998,"15min":16.989999999999998},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1890000000000001,"15min":1.1599999999999999},"min":{"1min":0.81599999999999995,"5min":0.41299999999999998,"15min":0.36699999999999999},"max":{"1min":1.4690000000000001,"5min":4.4530000000000003,"15min":4.4530000000000003},"last":1.294}]},{"osd":7,"last update":"Mon Oct 30 13:59:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.194,"15min":1.1739999999999999},"min":{"1min":0.69999999999999996,"5min":0.45100000000000001,"15min":0.42199999999999999},"max":{"1min":1.6799999999999999,"5min":2.3220000000000001,"15min":3.4510000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.3109999999999999,"15min":1.3120000000000001},"min":{"1min":0.754,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9259999999999999,"5min":2.3570000000000002,"15min":3.5470000000000002},"last":1.3440000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542452,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449200,"kb_used_data":368040,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93922640,"statfs":{"total":96636764160,"available":96176783360,"internally_reserved":0,"allocated":376872960,"data_stored":368821573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.99199999999999999,"15min":0.64000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.63700000000000001,"5min":45.311,"15min":45.311},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.52200000000000002,"15min":0.59499999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27200000000000002},"max":{"1min":0.79400000000000004,"5min":2.2410000000000001,"15min":27.616},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.29099999999999998},"max":{"1min":0.70799999999999996,"5min":0.88100000000000001,"15min":2.5449999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.61499999999999999,"15min":0.60299999999999998},"min":{"1min":0.442,"5min":0.40899999999999997,"15min":0.35199999999999998},"max":{"1min":0.74299999999999999,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.67400000000000004,"15min":0.56000000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.29299999999999998},"max":{"1min":0.77100000000000002,"5min":12.368,"15min":12.368},"last":0.432},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.75700000000000001,"15min":0.67200000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82199999999999995,"5min":12.442,"15min":12.442},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.59599999999999997,"5min":0.54000000000000004,"15min":0.41599999999999998},"max":{"1min":1.196,"5min":2.6949999999999998,"15min":9.4979999999999993},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.86199999999999999,"15min":0.86699999999999999},"min":{"1min":0.65700000000000003,"5min":0.60199999999999998,"15min":0.46700000000000003},"max":{"1min":1.1699999999999999,"5min":2.6440000000000001,"15min":9.5429999999999993},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89100000000000001,"15min":0.88500000000000001},"min":{"1min":0.74399999999999999,"5min":0.47099999999999997,"15min":0.317},"max":{"1min":1.1339999999999999,"5min":1.1359999999999999,"15min":11.026},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.3700000000000001,"15min":1.0289999999999999},"min":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.46999999999999997},"max":{"1min":1.1120000000000001,"5min":44.225999999999999,"15min":44.225999999999999},"last":0.93400000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.89000000000000001,"15min":0.878},"min":{"1min":0.68200000000000005,"5min":0.63,"15min":0.48799999999999999},"max":{"1min":1.234,"5min":2.5169999999999999,"15min":2.9300000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.86499999999999999,"15min":0.879},"min":{"1min":0.64000000000000001,"5min":0.60499999999999998,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":1.236,"15min":11.087999999999999},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.91100000000000003,"15min":0.877},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.50900000000000001},"max":{"1min":1.2529999999999999,"5min":5.6849999999999996,"15min":5.6849999999999996},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.92400000000000004,"15min":0.93100000000000005},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.57199999999999995},"max":{"1min":1.0860000000000001,"5min":2.6030000000000002,"15min":9.5890000000000004},"last":0.86599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50573312,"data_stored":50555931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46346240,"data_stored":46320604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75603968,"data_stored":75575369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":71393280,"data_stored":71379650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67129344,"data_stored":67110274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":75489280,"data_stored":75469249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":67350528,"data_stored":67333247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325066752,"data_stored":317352305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183717888,"data_stored":178743923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144601088,"data_stored":139970730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":249585664,"data_stored":241694096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":273088512,"data_stored":265087328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":275906560,"data_stored":268118820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":177188864,"data_stored":172364192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":273362944,"data_stored":265498878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:00:12.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T14:00:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.911+0000 7fedbb165640 1 Processor -- start 2023-10-30T14:00:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedbb165640 1 -- start start 2023-10-30T14:00:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedbb165640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fedb40c1020 0x7fedb4072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedbb165640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedbb165640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb41581f0 con 0x7fedb4151a20 2023-10-30T14:00:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedbb165640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb41576d0 con 0x7fedb40c1020 2023-10-30T14:00:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedbb165640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb4158680 con 0x7fedb40725c0 2023-10-30T14:00:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fedb40c1020 0x7fedb4072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb3fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fedb40c1020 0x7fedb4072080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52518/0 (socket says 172.21.15.31:52518) 2023-10-30T14:00:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb8f01640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fedb4151a20 0x7fedb40cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54090/0 (socket says 172.21.15.31:54090) 2023-10-30T14:00:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb3fff640 1 -- 172.21.15.31:0/637658700 learned_addr learned my addr 172.21.15.31:0/637658700 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb9702640 1 --2- 172.21.15.31:0/637658700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb3fff640 1 -- 172.21.15.31:0/637658700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 msgr2=0x7fedb40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb3fff640 1 --2- 172.21.15.31:0/637658700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb40c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb3fff640 1 -- 172.21.15.31:0/637658700 >> v1:172.21.15.31:6789/0 conn(0x7fedb4151a20 legacy=0x7fedb40cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb3fff640 1 -- 172.21.15.31:0/637658700 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb4151e00 con 0x7fedb40c1020 2023-10-30T14:00:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.915+0000 7fedb37fe640 1 -- 172.21.15.31:0/637658700 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328845721 0 0) 0x7fedb41581f0 con 0x7fedb4151a20 2023-10-30T14:00:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedb3fff640 1 --2- 172.21.15.31:0/637658700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb4072080 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7feda00098a0 tx=0x7feda001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8474c7a20c14311 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedb37fe640 1 -- 172.21.15.31:0/637658700 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00a8a10 con 0x7fedb40c1020 2023-10-30T14:00:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedb37fe640 1 -- 172.21.15.31:0/637658700 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda00a8b70 con 0x7fedb40c1020 2023-10-30T14:00:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedb37fe640 1 -- 172.21.15.31:0/637658700 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00b0440 con 0x7fedb40c1020 2023-10-30T14:00:12.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 -- 172.21.15.31:0/637658700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 msgr2=0x7fedb4072080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 --2- 172.21.15.31:0/637658700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb4072080 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7feda00098a0 tx=0x7feda001ed10 comp rx=0 tx=0).stop 2023-10-30T14:00:12.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 -- 172.21.15.31:0/637658700 shutdown_connections 2023-10-30T14:00:12.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 --2- 172.21.15.31:0/637658700 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb40c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 --2- 172.21.15.31:0/637658700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb4072080 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 -- 172.21.15.31:0/637658700 >> 172.21.15.31:0/637658700 conn(0x7fedb406da40 msgr2=0x7fedb406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:12.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 -- 172.21.15.31:0/637658700 shutdown_connections 2023-10-30T14:00:12.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 -- 172.21.15.31:0/637658700 wait complete. 2023-10-30T14:00:12.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 Processor -- start 2023-10-30T14:00:12.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.919+0000 7fedbb165640 1 -- start start 2023-10-30T14:00:12.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb41f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb40c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedb4151a20 0x7fedb414d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40c4550 con 0x7fedb40c1020 2023-10-30T14:00:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40ae6f0 con 0x7fedb4151a20 2023-10-30T14:00:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40c3f90 con 0x7fedb40725c0 2023-10-30T14:00:12.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb40c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb40c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52524/0 (socket says 172.21.15.31:52524) 2023-10-30T14:00:12.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 -- 172.21.15.31:0/1901578580 learned_addr learned my addr 172.21.15.31:0/1901578580 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:12.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb8f01640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb41f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb9702640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedb4151a20 0x7fedb414d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 -- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 msgr2=0x7fedb41f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb41f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 -- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedb4151a20 msgr2=0x7fedb414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:12.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedb4151a20 0x7fedb414d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:12.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 -- 172.21.15.31:0/1901578580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00094d0 con 0x7fedb40c1020 2023-10-30T14:00:12.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb3fff640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb40c9eb0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7feda00b3040 tx=0x7feda001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:12.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00be070 con 0x7fedb40c1020 2023-10-30T14:00:12.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda00ba400 con 0x7fedb40c1020 2023-10-30T14:00:12.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00ba7d0 con 0x7fedb40c1020 2023-10-30T14:00:12.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb40c4270 con 0x7fedb40c1020 2023-10-30T14:00:12.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb41f2490 con 0x7fedb40c1020 2023-10-30T14:00:12.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb404e170 con 0x7fedb40c1020 2023-10-30T14:00:12.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.923+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feda00c3410 con 0x7fedb40c1020 2023-10-30T14:00:12.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.927+0000 7fedb17fa640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed980664a0 0x7fed98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:12.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.927+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7feda0121d90 con 0x7fedb40c1020 2023-10-30T14:00:12.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.927+0000 7fedb8f01640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed980664a0 0x7fed98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:12.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.927+0000 7fedb8f01640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed980664a0 0x7fed98068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feda8000f70 tx=0x7feda8007b60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:12.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:12.931+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feda00eede0 con 0x7fedb40c1020 2023-10-30T14:00:13.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.087+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fedb41f2680 con 0x7fedb40c1020 2023-10-30T14:00:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.091+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7feda00b50c0 con 0x7fedb40c1020 2023-10-30T14:00:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.091+0000 7fedb17fa640 1 -- 172.21.15.31:0/1901578580 <== mon.0 v2:172.21.15.31:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7feda00eede0 con 0x7fedb40c1020 2023-10-30T14:00:13.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed980664a0 msgr2=0x7fed98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed980664a0 0x7fed98068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feda8000f70 tx=0x7feda8007b60 comp rx=0 tx=0).stop 2023-10-30T14:00:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 msgr2=0x7fedb40c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb40c9eb0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7feda00b3040 tx=0x7feda001bab0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 shutdown_connections 2023-10-30T14:00:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fed980664a0 0x7fed98068960 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedb4151a20 0x7fedb414d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedb40c1020 0x7fedb40c9eb0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 --2- 172.21.15.31:0/1901578580 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fedb40725c0 0x7fedb41f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 >> 172.21.15.31:0/1901578580 conn(0x7fedb406da40 msgr2=0x7fedb40c2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:13.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 shutdown_connections 2023-10-30T14:00:13.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.095+0000 7fedbb165640 1 -- 172.21.15.31:0/1901578580 wait complete. 2023-10-30T14:00:13.117 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-30T14:00:13.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:13.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89ef3e640 1 Processor -- start 2023-10-30T14:00:13.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89ef3e640 1 -- start start 2023-10-30T14:00:13.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89ef3e640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe8980c4a10 0x7fe89815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89ef3e640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fe8980c42a0 0x7fe8980ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89ef3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe898159f20 con 0x7fe898153c80 2023-10-30T14:00:13.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89ef3e640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89815dec0 con 0x7fe8980c4a10 2023-10-30T14:00:13.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89ef3e640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8981599a0 con 0x7fe8980c42a0 2023-10-30T14:00:13.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe897fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe8980c4a10 0x7fe89815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe897fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fe8980c4a10 0x7fe89815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52526/0 (socket says 172.21.15.31:52526) 2023-10-30T14:00:13.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe897fff640 1 -- 172.21.15.31:0/2973977064 learned_addr learned my addr 172.21.15.31:0/2973977064 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:13.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe89d4db640 1 --2- 172.21.15.31:0/2973977064 >> v2:172.21.15.33:3300/0 conn(0x7fe8980c42a0 0x7fe8980ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe897fff640 1 -- 172.21.15.31:0/2973977064 >> v2:172.21.15.33:3300/0 conn(0x7fe8980c42a0 msgr2=0x7fe8980ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:13.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe897fff640 1 --2- 172.21.15.31:0/2973977064 >> v2:172.21.15.33:3300/0 conn(0x7fe8980c42a0 0x7fe8980ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe897fff640 1 -- 172.21.15.31:0/2973977064 >> v1:172.21.15.33:6789/0 conn(0x7fe898153c80 legacy=0x7fe89815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.259+0000 7fe897fff640 1 -- 172.21.15.31:0/2973977064 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89815d620 con 0x7fe8980c4a10 2023-10-30T14:00:13.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe897fff640 1 --2- 172.21.15.31:0/2973977064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe89815cf90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe8980b1d40 tx=0x7fe88401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc048fa92ed91824 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:13.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe8977fe640 1 -- 172.21.15.31:0/2973977064 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8840a89f0 con 0x7fe8980c4a10 2023-10-30T14:00:13.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe8977fe640 1 -- 172.21.15.31:0/2973977064 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe8840a8b50 con 0x7fe8980c4a10 2023-10-30T14:00:13.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe8977fe640 1 -- 172.21.15.31:0/2973977064 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8840b0440 con 0x7fe8980c4a10 2023-10-30T14:00:13.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 -- 172.21.15.31:0/2973977064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 msgr2=0x7fe89815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/2973977064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe89815cf90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe8980b1d40 tx=0x7fe88401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 -- 172.21.15.31:0/2973977064 shutdown_connections 2023-10-30T14:00:13.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/2973977064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe89815cf90 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/2973977064 >> v2:172.21.15.33:3300/0 conn(0x7fe8980c42a0 0x7fe8980ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 -- 172.21.15.31:0/2973977064 >> 172.21.15.31:0/2973977064 conn(0x7fe8980715c0 msgr2=0x7fe8980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:13.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 -- 172.21.15.31:0/2973977064 shutdown_connections 2023-10-30T14:00:13.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.263+0000 7fe89ef3e640 1 -- 172.21.15.31:0/2973977064 wait complete. 2023-10-30T14:00:13.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 Processor -- start 2023-10-30T14:00:13.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 -- start start 2023-10-30T14:00:13.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe8980c42a0 0x7fe8980c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe8980b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe898153c80 0x7fe898149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89815b7d0 con 0x7fe8980c4a10 2023-10-30T14:00:13.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89815bc60 con 0x7fe898153c80 2023-10-30T14:00:13.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89815ac00 con 0x7fe8980c42a0 2023-10-30T14:00:13.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe8980b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe8980b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52532/0 (socket says 172.21.15.31:52532) 2023-10-30T14:00:13.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 -- 172.21.15.31:0/666319341 learned_addr learned my addr 172.21.15.31:0/666319341 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:13.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89d4db640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe898153c80 0x7fe898149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ccda640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe8980c42a0 0x7fe8980c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 -- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe8980c42a0 msgr2=0x7fe8980c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe8980c42a0 0x7fe8980c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 -- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe898153c80 msgr2=0x7fe898149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe898153c80 0x7fe898149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 -- 172.21.15.31:0/666319341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840094d0 con 0x7fe8980c4a10 2023-10-30T14:00:13.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89d4db640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe898153c80 0x7fe898149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:13.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe897fff640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe8980b2980 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe88401b900 tx=0x7fe88401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:13.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe8957fa640 1 -- 172.21.15.31:0/666319341 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8840a89f0 con 0x7fe8980c4a10 2023-10-30T14:00:13.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe8957fa640 1 -- 172.21.15.31:0/666319341 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe8840bb480 con 0x7fe8980c4a10 2023-10-30T14:00:13.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8981501c0 con 0x7fe8980c4a10 2023-10-30T14:00:13.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe8957fa640 1 -- 172.21.15.31:0/666319341 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8840bb870 con 0x7fe8980c4a10 2023-10-30T14:00:13.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89815b180 con 0x7fe8980c4a10 2023-10-30T14:00:13.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.267+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89815b940 con 0x7fe8980c4a10 2023-10-30T14:00:13.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.271+0000 7fe8957fa640 1 -- 172.21.15.31:0/666319341 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8840bb9d0 con 0x7fe8980c4a10 2023-10-30T14:00:13.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.271+0000 7fe8957fa640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.271+0000 7fe8957fa640 1 -- 172.21.15.31:0/666319341 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fe884122ae0 con 0x7fe8980c4a10 2023-10-30T14:00:13.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.271+0000 7fe89ccda640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.271+0000 7fe89ccda640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000f70 tx=0x7fe88c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:13.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.275+0000 7fe8957fa640 1 -- 172.21.15.31:0/666319341 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8840efa10 con 0x7fe8980c4a10 2023-10-30T14:00:13.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.463+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe89804e170 con 0x7fe8980c4a10 2023-10-30T14:00:13.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.463+0000 7fe8957fa640 1 -- 172.21.15.31:0/666319341 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fe8840b5020 con 0x7fe8980c4a10 2023-10-30T14:00:13.468 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T14:00:13.469 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/1696211446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":1696211446},{"type":"v1","addr":"172.21.15.33:6836","nonce":1696211446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:11.104156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:13.469 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 95 2023-10-30T14:00:13.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe87c0664a0 msgr2=0x7fe87c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000f70 tx=0x7fe88c009040 comp rx=0 tx=0).stop 2023-10-30T14:00:13.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 msgr2=0x7fe8980b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe8980b2980 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe88401b900 tx=0x7fe88401ba90 comp rx=0 tx=0).stop 2023-10-30T14:00:13.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 shutdown_connections 2023-10-30T14:00:13.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe898153c80 0x7fe898149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8980c4a10 0x7fe8980b2980 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 --2- 172.21.15.31:0/666319341 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fe8980c42a0 0x7fe8980c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 >> 172.21.15.31:0/666319341 conn(0x7fe8980715c0 msgr2=0x7fe898159220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:13.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 shutdown_connections 2023-10-30T14:00:13.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.467+0000 7fe89ef3e640 1 -- 172.21.15.31:0/666319341 wait complete. 2023-10-30T14:00:13.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdded0f640 1 Processor -- start 2023-10-30T14:00:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdded0f640 1 -- start start 2023-10-30T14:00:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdded0f640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbdd80ccf60 0x7fbdd80cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdded0f640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fbdd80c2500 0x7fbdd80c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdded0f640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd8151aa0 con 0x7fbdd81516c0 2023-10-30T14:00:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdded0f640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd80c74a0 con 0x7fbdd80ccf60 2023-10-30T14:00:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdded0f640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd80c2e70 con 0x7fbdd80c2500 2023-10-30T14:00:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdd7fff640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fbdd80ccf60 0x7fbdd80cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbddcaab640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fbdd81516c0 0x7fbdd80c7d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54094/0 (socket says 172.21.15.31:54094) 2023-10-30T14:00:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbddcaab640 1 -- 172.21.15.31:0/1522738431 learned_addr learned my addr 172.21.15.31:0/1522738431 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T14:00:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbddd2ac640 1 --2- 172.21.15.31:0/1522738431 >> v2:172.21.15.33:3300/0 conn(0x7fbdd80c2500 0x7fbdd80c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdd7fff640 1 -- 172.21.15.31:0/1522738431 >> v2:172.21.15.33:3300/0 conn(0x7fbdd80c2500 msgr2=0x7fbdd80c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdd7fff640 1 --2- 172.21.15.31:0/1522738431 >> v2:172.21.15.33:3300/0 conn(0x7fbdd80c2500 0x7fbdd80c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdd7fff640 1 -- 172.21.15.31:0/1522738431 >> v1:172.21.15.31:6789/0 conn(0x7fbdd81516c0 legacy=0x7fbdd80c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdd77fe640 1 -- 172.21.15.31:0/1522738431 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538615778 0 0) 0x7fbdd8151aa0 con 0x7fbdd81516c0 2023-10-30T14:00:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.631+0000 7fbdd7fff640 1 -- 172.21.15.31:0/1522738431 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8158af0 con 0x7fbdd80ccf60 2023-10-30T14:00:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdd7fff640 1 --2- 172.21.15.31:0/1522738431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cd3a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80b3200 tx=0x7fbdc401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fffb2bdb86157cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdd77fe640 1 -- 172.21.15.31:0/1522738431 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40ab600 con 0x7fbdd80ccf60 2023-10-30T14:00:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdd77fe640 1 -- 172.21.15.31:0/1522738431 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbdc40a7050 con 0x7fbdd80ccf60 2023-10-30T14:00:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdd77fe640 1 -- 172.21.15.31:0/1522738431 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40ab9a0 con 0x7fbdd80ccf60 2023-10-30T14:00:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 -- 172.21.15.31:0/1522738431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 msgr2=0x7fbdd80cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1522738431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cd3a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80b3200 tx=0x7fbdc401c460 comp rx=0 tx=0).stop 2023-10-30T14:00:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 -- 172.21.15.31:0/1522738431 shutdown_connections 2023-10-30T14:00:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1522738431 >> v2:172.21.15.33:3300/0 conn(0x7fbdd80c2500 0x7fbdd80c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1522738431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cd3a0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 -- 172.21.15.31:0/1522738431 >> 172.21.15.31:0/1522738431 conn(0x7fbdd8071fb0 msgr2=0x7fbdd80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 -- 172.21.15.31:0/1522738431 shutdown_connections 2023-10-30T14:00:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 -- 172.21.15.31:0/1522738431 wait complete. 2023-10-30T14:00:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 Processor -- start 2023-10-30T14:00:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.635+0000 7fbdded0f640 1 -- start start 2023-10-30T14:00:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdd80c2500 0x7fbdd8146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdd81516c0 0x7fbdd80cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd8151d30 con 0x7fbdd80ccf60 2023-10-30T14:00:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd81f07d0 con 0x7fbdd80c2500 2023-10-30T14:00:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd81f0270 con 0x7fbdd81516c0 2023-10-30T14:00:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52544/0 (socket says 172.21.15.31:52544) 2023-10-30T14:00:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 -- 172.21.15.31:0/1090274038 learned_addr learned my addr 172.21.15.31:0/1090274038 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbddcaab640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdd80c2500 0x7fbdd8146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbddd2ac640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdd81516c0 0x7fbdd80cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 -- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdd81516c0 msgr2=0x7fbdd80cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdd81516c0 0x7fbdd80cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 -- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdd80c2500 msgr2=0x7fbdd8146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdd80c2500 0x7fbdd8146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 -- 172.21.15.31:0/1090274038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc4009420 con 0x7fbdd80ccf60 2023-10-30T14:00:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbddcaab640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdd80c2500 0x7fbdd8146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd7fff640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cadb0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40ae040 tx=0x7fbdc4019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd57fa640 1 -- 172.21.15.31:0/1090274038 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40b4840 con 0x7fbdd80ccf60 2023-10-30T14:00:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd57fa640 1 -- 172.21.15.31:0/1090274038 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbdc40abe90 con 0x7fbdd80ccf60 2023-10-30T14:00:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd81f04f0 con 0x7fbdd80ccf60 2023-10-30T14:00:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdd57fa640 1 -- 172.21.15.31:0/1090274038 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40b4be0 con 0x7fbdd80ccf60 2023-10-30T14:00:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd80c96c0 con 0x7fbdd80ccf60 2023-10-30T14:00:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.639+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd804e170 con 0x7fbdd80ccf60 2023-10-30T14:00:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.643+0000 7fbdd57fa640 1 -- 172.21.15.31:0/1090274038 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbdc40b2040 con 0x7fbdd80ccf60 2023-10-30T14:00:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.643+0000 7fbdd57fa640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdbc0664a0 0x7fbdbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.643+0000 7fbdd57fa640 1 -- 172.21.15.31:0/1090274038 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fbdc411e650 con 0x7fbdd80ccf60 2023-10-30T14:00:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.643+0000 7fbddcaab640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdbc0664a0 0x7fbdbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.643+0000 7fbddcaab640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdbc0664a0 0x7fbdbc068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc000f70 tx=0x7fbdcc007cb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.647+0000 7fbdd57fa640 1 -- 172.21.15.31:0/1090274038 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbdc40eb5a0 con 0x7fbdd80ccf60 2023-10-30T14:00:13.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.835+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdd8154f20 con 0x7fbdd80ccf60 2023-10-30T14:00:13.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.835+0000 7fbdd57fa640 1 -- 172.21.15.31:0/1090274038 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+6493 (secure 0 0 0) 0x7fbdc40b0020 con 0x7fbdd80ccf60 2023-10-30T14:00:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T14:00:13.840 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/818491485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":818491485},{"type":"v1","addr":"172.21.15.31:6833","nonce":818491485}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3532148251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3532148251},{"type":"v1","addr":"172.21.15.31:6835","nonce":3532148251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/331154977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":331154977},{"type":"v1","addr":"172.21.15.33:6839","nonce":331154977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":5308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/2118974990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6835","nonce":2118974990},{"type":"v1","addr":"172.21.15.33:6837","nonce":2118974990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6836/1696211446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":1696211446},{"type":"v1","addr":"172.21.15.33:6836","nonce":1696211446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:35:07.654733+0000","modified":"2023-10-30T13:58:11.104156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6549},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6549":{"gid":6549,"name":"a","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.31:6837/1195492284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1195492284},{"type":"v1","addr":"172.21.15.31:6837","nonce":1195492284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:13.841 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 95 2023-10-30T14:00:13.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdbc0664a0 msgr2=0x7fbdbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdbc0664a0 0x7fbdbc068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc000f70 tx=0x7fbdcc007cb0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 msgr2=0x7fbdd80cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:13.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cadb0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40ae040 tx=0x7fbdc4019200 comp rx=0 tx=0).stop 2023-10-30T14:00:13.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 shutdown_connections 2023-10-30T14:00:13.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fbdbc0664a0 0x7fbdbc068960 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fbdd81516c0 0x7fbdd80cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdd80ccf60 0x7fbdd80cadb0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 --2- 172.21.15.31:0/1090274038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdd80c2500 0x7fbdd8146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:13.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 >> 172.21.15.31:0/1090274038 conn(0x7fbdd8071fb0 msgr2=0x7fbdd8156a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:13.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 shutdown_connections 2023-10-30T14:00:13.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.839+0000 7fbdded0f640 1 -- 172.21.15.31:0/1090274038 wait complete. 2023-10-30T14:00:13.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:00:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:13.999+0000 7f352286b640 1 Processor -- start 2023-10-30T14:00:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f352286b640 1 -- start start 2023-10-30T14:00:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f352286b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 0x7f351c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f352286b640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f351c0c9b40 0x7f351c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f352286b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c15cda0 con 0x7f351c0c5f80 2023-10-30T14:00:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f352286b640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351c15d990 con 0x7f351c0c9b40 2023-10-30T14:00:14.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f352286b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c153f60 con 0x7f351c0c5ba0 2023-10-30T14:00:14.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f351c0c5f80 0x7f351c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54100/0 (socket says 172.21.15.31:54100) 2023-10-30T14:00:14.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 -- 172.21.15.31:0/1128253034 learned_addr learned my addr 172.21.15.31:0/1128253034 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T14:00:14.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f3520e08640 1 --2- 172.21.15.31:0/1128253034 >> v2:172.21.15.33:3300/0 conn(0x7f351c0c9b40 0x7f351c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351bfff640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 0x7f351c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886860142 0 0) 0x7f351c15cda0 con 0x7f351c0c5f80 2023-10-30T14:00:14.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3504003530 con 0x7f351c0c5f80 2023-10-30T14:00:14.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1198147909 0 0) 0x7f3504003530 con 0x7f351c0c5f80 2023-10-30T14:00:14.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 msgr2=0x7f351c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:14.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 0x7f351c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 >> v2:172.21.15.33:3300/0 conn(0x7f351c0c9b40 msgr2=0x7f351c1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:14.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 --2- 172.21.15.31:0/1128253034 >> v2:172.21.15.33:3300/0 conn(0x7f351c0c9b40 0x7f351c1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c06f490 con 0x7f351c0c5f80 2023-10-30T14:00:14.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f3510001680 con 0x7f351c0c5f80 2023-10-30T14:00:14.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f3520e08640 1 --2- 172.21.15.31:0/1128253034 >> v2:172.21.15.33:3300/0 conn(0x7f351c0c9b40 0x7f351c1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:00:14.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 >> v1:172.21.15.31:6789/0 conn(0x7f351c0c5f80 legacy=0x7f351c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35040039b0 0x7f3504003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040047d0 0x7f3504004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 0x7f3504005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c15d990 con 0x7f35040047d0 2023-10-30T14:00:14.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c153f60 con 0x7f35040039b0 2023-10-30T14:00:14.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350400b990 con 0x7f351c0c5ba0 2023-10-30T14:00:14.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040047d0 0x7f3504004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351bfff640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35040039b0 0x7f3504003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f3520e08640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 0x7f3504005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 -- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 msgr2=0x7f3504005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:14.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 0x7f3504005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 -- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35040039b0 msgr2=0x7f3504003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:14.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35040039b0 0x7f3504003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f351b7fe640 1 -- 172.21.15.31:0/1128253034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504003530 con 0x7f35040047d0 2023-10-30T14:00:14.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.003+0000 7f3520e08640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f351c0c5ba0 0x7f3504005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:14.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f351b7fe640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040047d0 0x7f3504004c50 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f351000c190 tx=0x7f351001f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163fadc08452139e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35100a8e70 con 0x7f35040047d0 2023-10-30T14:00:14.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f35100058e0 con 0x7f35040047d0 2023-10-30T14:00:14.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f351affd640 1 -- 172.21.15.31:0/1128253034 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510005cf0 con 0x7f35040047d0 2023-10-30T14:00:14.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f352286b640 1 -- 172.21.15.31:0/1128253034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040047d0 msgr2=0x7f3504004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f352286b640 1 --2- 172.21.15.31:0/1128253034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040047d0 0x7f3504004c50 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f351000c190 tx=0x7f351001f350 comp rx=0 tx=0).stop 2023-10-30T14:00:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f3520e08640 1 -- 172.21.15.31:0/1128253034 reap_dead start 2023-10-30T14:00:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f352286b640 1 -- 172.21.15.31:0/1128253034 shutdown_connections 2023-10-30T14:00:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f352286b640 1 -- 172.21.15.31:0/1128253034 >> 172.21.15.31:0/1128253034 conn(0x7f351c06afa0 msgr2=0x7f351c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f352286b640 1 -- 172.21.15.31:0/1128253034 shutdown_connections 2023-10-30T14:00:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.007+0000 7f352286b640 1 -- 172.21.15.31:0/1128253034 wait complete. 2023-10-30T14:00:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 Processor -- start 2023-10-30T14:00:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 -- start start 2023-10-30T14:00:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35040047d0 0x7f351c0c1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f351c1565c0 0x7f351c0c2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c153270 0x7f351c15a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c1ef030 con 0x7f351c1565c0 2023-10-30T14:00:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c1ef240 con 0x7f351c153270 2023-10-30T14:00:14.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c1ef520 con 0x7f35040047d0 2023-10-30T14:00:14.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f351c1565c0 0x7f351c0c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f351c1565c0 0x7f351c0c2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52572/0 (socket says 172.21.15.31:52572) 2023-10-30T14:00:14.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 -- 172.21.15.31:0/1587982226 learned_addr learned my addr 172.21.15.31:0/1587982226 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f3520e08640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c153270 0x7f351c15a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351bfff640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35040047d0 0x7f351c0c1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 -- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35040047d0 msgr2=0x7f351c0c1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35040047d0 0x7f351c0c1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 -- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c153270 msgr2=0x7f351c15a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c153270 0x7f351c15a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 -- 172.21.15.31:0/1587982226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c1ef800 con 0x7f351c1565c0 2023-10-30T14:00:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f351b7fe640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f351c1565c0 0x7f351c0c2340 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f351001f180 tx=0x7f3504005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f3518ff9640 1 -- 172.21.15.31:0/1587982226 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35100b7070 con 0x7f351c1565c0 2023-10-30T14:00:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f3518ff9640 1 -- 172.21.15.31:0/1587982226 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3510000ee0 con 0x7f351c1565c0 2023-10-30T14:00:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f3518ff9640 1 -- 172.21.15.31:0/1587982226 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510001660 con 0x7f351c1565c0 2023-10-30T14:00:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c1efae0 con 0x7f351c1565c0 2023-10-30T14:00:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.011+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c15ae10 con 0x7f351c1565c0 2023-10-30T14:00:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.015+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c1ef660 con 0x7f351c1565c0 2023-10-30T14:00:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.015+0000 7f3518ff9640 1 -- 172.21.15.31:0/1587982226 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3510007030 con 0x7f351c1565c0 2023-10-30T14:00:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.015+0000 7f3518ff9640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35080664a0 0x7f3508068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.015+0000 7f3518ff9640 1 -- 172.21.15.31:0/1587982226 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f3510120f80 con 0x7f351c1565c0 2023-10-30T14:00:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.015+0000 7f351bfff640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35080664a0 0x7f3508068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.019+0000 7f351bfff640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35080664a0 0x7f3508068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f350c009a80 tx=0x7f350c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.019+0000 7f3518ff9640 1 -- 172.21.15.31:0/1587982226 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f351c1ef660 con 0x7f351c1565c0 2023-10-30T14:00:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f351c1ef660 con 0x7f351c1565c0 2023-10-30T14:00:14.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f3518ff9640 1 -- 172.21.15.31:0/1587982226 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18184 (secure 0 0 0) 0x7f351c1ef660 con 0x7f351c1565c0 2023-10-30T14:00:14.182 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-30T14:00:14.183 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":56,"fsid":"62b651f4-6cfc-4dcb-91a0-e9335b7700dd","created":"2023-10-30T13:34:42.004900+0000","modified":"2023-10-30T13:57:42.156993+0000","last_up_change":"2023-10-30T13:34:53.554510+0000","last_in_change":"2023-10-30T13:34:44.552560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:34:53.247214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:35:05.304460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:35:06.652927+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1902efe2-909b-4286-9f44-68398744f1b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6801","nonce":1469638853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6805","nonce":1469638853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6813","nonce":1469638853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":1469638853},{"type":"v1","addr":"172.21.15.31:6809","nonce":1469638853}]},"public_addr":"172.21.15.31:6801/1469638853","cluster_addr":"172.21.15.31:6805/1469638853","heartbeat_back_addr":"172.21.15.31:6813/1469638853","heartbeat_front_addr":"172.21.15.31:6809/1469638853","state":["exists","up"]},{"osd":1,"uuid":"c659a563-8574-4193-ae70-39bed11809f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6804","nonce":1190064951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6808","nonce":1190064951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6815","nonce":1190064951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1190064951},{"type":"v1","addr":"172.21.15.31:6812","nonce":1190064951}]},"public_addr":"172.21.15.31:6804/1190064951","cluster_addr":"172.21.15.31:6808/1190064951","heartbeat_back_addr":"172.21.15.31:6815/1190064951","heartbeat_front_addr":"172.21.15.31:6812/1190064951","state":["exists","up"]},{"osd":2,"uuid":"2c5a63c6-dd27-4bfd-b9b4-af64e974503e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6825","nonce":3739146666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6827","nonce":3739146666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6831","nonce":3739146666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3739146666},{"type":"v1","addr":"172.21.15.31:6829","nonce":3739146666}]},"public_addr":"172.21.15.31:6825/3739146666","cluster_addr":"172.21.15.31:6827/3739146666","heartbeat_back_addr":"172.21.15.31:6831/3739146666","heartbeat_front_addr":"172.21.15.31:6829/3739146666","state":["exists","up"]},{"osd":3,"uuid":"66544431-3c45-45bb-9505-2469b825ccc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6817","nonce":2725375815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6819","nonce":2725375815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6823","nonce":2725375815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2725375815},{"type":"v1","addr":"172.21.15.31:6821","nonce":2725375815}]},"public_addr":"172.21.15.31:6817/2725375815","cluster_addr":"172.21.15.31:6819/2725375815","heartbeat_back_addr":"172.21.15.31:6823/2725375815","heartbeat_front_addr":"172.21.15.31:6821/2725375815","state":["exists","up"]},{"osd":4,"uuid":"8fa6bf36-d2cf-4588-b248-7ee20a44b370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6801","nonce":465067075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6803","nonce":465067075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6807","nonce":465067075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":465067075},{"type":"v1","addr":"172.21.15.33:6805","nonce":465067075}]},"public_addr":"172.21.15.33:6801/465067075","cluster_addr":"172.21.15.33:6803/465067075","heartbeat_back_addr":"172.21.15.33:6807/465067075","heartbeat_front_addr":"172.21.15.33:6805/465067075","state":["exists","up"]},{"osd":5,"uuid":"716c46a6-ea0f-4a1d-9fc2-8b8ac2b02d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6817","nonce":3902046602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6819","nonce":3902046602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6823","nonce":3902046602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3902046602},{"type":"v1","addr":"172.21.15.33:6821","nonce":3902046602}]},"public_addr":"172.21.15.33:6817/3902046602","cluster_addr":"172.21.15.33:6819/3902046602","heartbeat_back_addr":"172.21.15.33:6823/3902046602","heartbeat_front_addr":"172.21.15.33:6821/3902046602","state":["exists","up"]},{"osd":6,"uuid":"e95e1fe0-72b5-4773-8689-99d6417fa36f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6809","nonce":931136877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6811","nonce":931136877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6815","nonce":931136877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":931136877},{"type":"v1","addr":"172.21.15.33:6813","nonce":931136877}]},"public_addr":"172.21.15.33:6809/931136877","cluster_addr":"172.21.15.33:6811/931136877","heartbeat_back_addr":"172.21.15.33:6815/931136877","heartbeat_front_addr":"172.21.15.33:6813/931136877","state":["exists","up"]},{"osd":7,"uuid":"0727ffd4-34a3-48d6-8431-37963964af96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6825","nonce":3463894958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6827","nonce":3463894958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6831","nonce":3463894958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3463894958},{"type":"v1","addr":"172.21.15.33:6829","nonce":3463894958}]},"public_addr":"172.21.15.33:6825/3463894958","cluster_addr":"172.21.15.33:6827/3463894958","heartbeat_back_addr":"172.21.15.33:6831/3463894958","heartbeat_front_addr":"172.21.15.33:6829/3463894958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.107781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.396071+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.444331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.258389+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.081619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.085569+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.106266+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:34:50.141318+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6836/2267972067":"2023-10-31T13:56:02.130126+0000","172.21.15.33:6836/3928996203":"2023-10-31T13:53:47.073133+0000","172.21.15.33:6834/3928996203":"2023-10-31T13:53:47.073133+0000","172.21.15.31:6837/327248983":"2023-10-31T13:51:22.020336+0000","172.21.15.31:6836/327248983":"2023-10-31T13:51:22.020336+0000","172.21.15.33:6834/2281022644":"2023-10-31T13:57:42.156901+0000","172.21.15.31:6837/2267972067":"2023-10-31T13:56:02.130126+0000","172.21.15.33:6836/2545654386":"2023-10-31T13:36:26.753730+0000","172.21.15.31:6836/1956809670":"2023-10-31T13:39:06.794112+0000","172.21.15.33:6836/15857146":"2023-10-31T13:49:36.991529+0000","172.21.15.31:6837/1956809670":"2023-10-31T13:39:06.794112+0000","172.21.15.33:6835/3402459191":"2023-10-31T13:40:56.840815+0000","172.21.15.33:6836/2281022644":"2023-10-31T13:57:42.156901+0000","172.21.15.33:6836/210349556":"2023-10-31T13:44:26.910290+0000","172.21.15.31:6836/349618961":"2023-10-31T13:47:11.952274+0000","172.21.15.33:6838/2419808617":"2023-10-31T13:43:31.895625+0000","172.21.15.33:6834/2545654386":"2023-10-31T13:36:26.753730+0000","172.21.15.33:6834/15857146":"2023-10-31T13:49:36.991529+0000","172.21.15.33:6839/2419808617":"2023-10-31T13:43:31.895625+0000","172.21.15.33:6837/3402459191":"2023-10-31T13:40:56.840815+0000","172.21.15.33:6834/210349556":"2023-10-31T13:44:26.910290+0000","172.21.15.31:6837/349618961":"2023-10-31T13:47:11.952274+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:00:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35080664a0 msgr2=0x7f3508068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35080664a0 0x7f3508068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f350c009a80 tx=0x7f350c009190 comp rx=0 tx=0).stop 2023-10-30T14:00:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f351c1565c0 msgr2=0x7f351c0c2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f351c1565c0 0x7f351c0c2340 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f351001f180 tx=0x7f3504005800 comp rx=0 tx=0).stop 2023-10-30T14:00:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 shutdown_connections 2023-10-30T14:00:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f35080664a0 0x7f3508068960 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c153270 0x7f351c15a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f351c1565c0 0x7f351c0c2340 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 --2- 172.21.15.31:0/1587982226 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f35040047d0 0x7f351c0c1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.179+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 >> 172.21.15.31:0/1587982226 conn(0x7f351c06afa0 msgr2=0x7f351c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.183+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 shutdown_connections 2023-10-30T14:00:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.183+0000 7f352286b640 1 -- 172.21.15.31:0/1587982226 wait complete. 2023-10-30T14:00:14.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-30T14:00:14.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.347+0000 7fca707c4640 1 Processor -- start 2023-10-30T14:00:14.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.347+0000 7fca707c4640 1 -- start start 2023-10-30T14:00:14.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca707c4640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7fca68065ba0 0x7fca6815a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca707c4640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6806e390 con 0x7fca68153c10 2023-10-30T14:00:14.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca707c4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680bec10 con 0x7fca68071340 2023-10-30T14:00:14.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca707c4640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca68072390 con 0x7fca68065ba0 2023-10-30T14:00:14.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6e560640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7fca68153c10 0x7fca68070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:54102/0 (socket says 172.21.15.31:54102) 2023-10-30T14:00:14.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6e560640 1 -- 172.21.15.31:0/1088603092 learned_addr learned my addr 172.21.15.31:0/1088603092 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T14:00:14.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6ed61640 1 --2- 172.21.15.31:0/1088603092 >> v2:172.21.15.31:3300/0 conn(0x7fca68065ba0 0x7fca6815a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6d55e640 1 -- 172.21.15.31:0/1088603092 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444941768 0 0) 0x7fca6806e390 con 0x7fca68153c10 2023-10-30T14:00:14.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6d55e640 1 -- 172.21.15.31:0/1088603092 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54003530 con 0x7fca68153c10 2023-10-30T14:00:14.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6ed61640 1 -- 172.21.15.31:0/1088603092 >> v1:172.21.15.33:6789/0 conn(0x7fca68071340 legacy=0x7fca68159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6ed61640 1 -- 172.21.15.31:0/1088603092 >> v1:172.21.15.31:6789/0 conn(0x7fca68153c10 legacy=0x7fca68070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6ed61640 1 -- 172.21.15.31:0/1088603092 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6806e5e0 con 0x7fca68065ba0 2023-10-30T14:00:14.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6ed61640 1 --2- 172.21.15.31:0/1088603092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68065ba0 0x7fca6815a080 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fca60009d90 tx=0x7fca6001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57292afd431e24f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6d55e640 1 -- 172.21.15.31:0/1088603092 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca600a6660 con 0x7fca68065ba0 2023-10-30T14:00:14.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6d55e640 1 -- 172.21.15.31:0/1088603092 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fca600a67c0 con 0x7fca68065ba0 2023-10-30T14:00:14.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.351+0000 7fca6d55e640 1 -- 172.21.15.31:0/1088603092 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca600a6b70 con 0x7fca68065ba0 2023-10-30T14:00:14.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- 172.21.15.31:0/1088603092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68065ba0 msgr2=0x7fca6815a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 --2- 172.21.15.31:0/1088603092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68065ba0 0x7fca6815a080 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fca60009d90 tx=0x7fca6001c8c0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- 172.21.15.31:0/1088603092 shutdown_connections 2023-10-30T14:00:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 --2- 172.21.15.31:0/1088603092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68065ba0 0x7fca6815a080 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- 172.21.15.31:0/1088603092 >> 172.21.15.31:0/1088603092 conn(0x7fca680c7f60 msgr2=0x7fca680c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- 172.21.15.31:0/1088603092 shutdown_connections 2023-10-30T14:00:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- 172.21.15.31:0/1088603092 wait complete. 2023-10-30T14:00:14.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 Processor -- start 2023-10-30T14:00:14.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- start start 2023-10-30T14:00:14.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca68065ba0 0x7fca681406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68071340 0x7fca68141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca68153c10 0x7fca681416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6814cc00 con 0x7fca68071340 2023-10-30T14:00:14.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca68143ea0 con 0x7fca68153c10 2023-10-30T14:00:14.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca707c4640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca681443d0 con 0x7fca68065ba0 2023-10-30T14:00:14.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca6dd5f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68071340 0x7fca68141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca6dd5f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68071340 0x7fca68141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52586/0 (socket says 172.21.15.31:52586) 2023-10-30T14:00:14.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca6dd5f640 1 -- 172.21.15.31:0/3354280895 learned_addr learned my addr 172.21.15.31:0/3354280895 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:14.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca6ed61640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca68153c10 0x7fca681416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca6e560640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca68065ba0 0x7fca681406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca6dd5f640 1 -- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca68065ba0 msgr2=0x7fca681406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.355+0000 7fca6dd5f640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca68065ba0 0x7fca681406d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca6dd5f640 1 -- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca68153c10 msgr2=0x7fca681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca6dd5f640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca68153c10 0x7fca681416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca6dd5f640 1 -- 172.21.15.31:0/3354280895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca60009a70 con 0x7fca68071340 2023-10-30T14:00:14.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca6dd5f640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68071340 0x7fca68141190 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fca5800c500 tx=0x7fca5800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca5effd640 1 -- 172.21.15.31:0/3354280895 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58005220 con 0x7fca68071340 2023-10-30T14:00:14.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca5effd640 1 -- 172.21.15.31:0/3354280895 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fca58005380 con 0x7fca68071340 2023-10-30T14:00:14.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca68141c10 con 0x7fca68071340 2023-10-30T14:00:14.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca5effd640 1 -- 172.21.15.31:0/3354280895 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58007290 con 0x7fca68071340 2023-10-30T14:00:14.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6814e050 con 0x7fca68071340 2023-10-30T14:00:14.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6804e170 con 0x7fca68071340 2023-10-30T14:00:14.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca5effd640 1 -- 172.21.15.31:0/3354280895 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca580073f0 con 0x7fca68071340 2023-10-30T14:00:14.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca5effd640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca4c0664a0 0x7fca4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca5effd640 1 -- 172.21.15.31:0/3354280895 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fca58083c90 con 0x7fca68071340 2023-10-30T14:00:14.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.359+0000 7fca6e560640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca4c0664a0 0x7fca4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.363+0000 7fca6e560640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca4c0664a0 0x7fca4c068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fca640011f0 tx=0x7fca64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.363+0000 7fca5effd640 1 -- 172.21.15.31:0/3354280895 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca58050c60 con 0x7fca68071340 2023-10-30T14:00:14.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.555+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fca680c5e00 con 0x7fca68071340 2023-10-30T14:00:14.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.559+0000 7fca5effd640 1 -- 172.21.15.31:0/3354280895 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) v1 ==== 161+0+0 (secure 0 0 0) 0x7fca58054280 con 0x7fca68071340 2023-10-30T14:00:14.562 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-30T14:00:14.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.559+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca4c0664a0 msgr2=0x7fca4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.559+0000 7fca707c4640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca4c0664a0 0x7fca4c068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fca640011f0 tx=0x7fca64009040 comp rx=0 tx=0).stop 2023-10-30T14:00:14.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.559+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68071340 msgr2=0x7fca68141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.559+0000 7fca707c4640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68071340 0x7fca68141190 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fca5800c500 tx=0x7fca5800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 shutdown_connections 2023-10-30T14:00:14.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fca4c0664a0 0x7fca4c068960 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca68153c10 0x7fca681416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca68071340 0x7fca68141190 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 --2- 172.21.15.31:0/3354280895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fca68065ba0 0x7fca681406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 >> 172.21.15.31:0/3354280895 conn(0x7fca680c7f60 msgr2=0x7fca680c9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:14.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 shutdown_connections 2023-10-30T14:00:14.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.563+0000 7fca707c4640 1 -- 172.21.15.31:0/3354280895 wait complete. 2023-10-30T14:00:14.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-30T14:00:14.640 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2023-10-30T14:00:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d27b5640 1 Processor -- start 2023-10-30T14:00:14.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d27b5640 1 -- start start 2023-10-30T14:00:14.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d27b5640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7f01cc153a10 0x7f01cc0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d27b5640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc15d0a0 con 0x7f01cc0718d0 2023-10-30T14:00:14.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d27b5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc15d380 con 0x7f01cc071cb0 2023-10-30T14:00:14.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d27b5640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01cc0cca10 con 0x7f01cc153a10 2023-10-30T14:00:14.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d17b3640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f01cc0718d0 0x7f01cc15a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51124/0 (socket says 172.21.15.31:51124) 2023-10-30T14:00:14.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d17b3640 1 -- 172.21.15.31:0/998327895 learned_addr learned my addr 172.21.15.31:0/998327895 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T14:00:14.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.731+0000 7f01d1fb4640 1 --2- 172.21.15.31:0/998327895 >> v2:172.21.15.33:3300/0 conn(0x7f01cc153a10 0x7f01cc0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675899830 0 0) 0x7f01cc15d0a0 con 0x7f01cc0718d0 2023-10-30T14:00:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 --> v1:172.21.15.31:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b8003530 con 0x7f01cc0718d0 2023-10-30T14:00:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 <== mon.0 v1:172.21.15.31:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1333860304 0 0) 0x7f01b8003530 con 0x7f01cc0718d0 2023-10-30T14:00:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 >> v2:172.21.15.33:3300/0 conn(0x7f01cc153a10 msgr2=0x7f01cc0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 --2- 172.21.15.31:0/998327895 >> v2:172.21.15.33:3300/0 conn(0x7f01cc153a10 0x7f01cc0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 >> v1:172.21.15.33:6789/0 conn(0x7f01cc071cb0 legacy=0x7f01cc153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 --> v1:172.21.15.31:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc15d660 con 0x7f01cc0718d0 2023-10-30T14:00:14.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 <== mon.0 v1:172.21.15.31:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4101561608 0 0) 0x7f01c8001680 con 0x7f01cc0718d0 2023-10-30T14:00:14.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 >> v1:172.21.15.31:6789/0 conn(0x7f01cc0718d0 legacy=0x7f01cc15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8003980 0x7f01b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01b80047d0 0x7f01b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b8005190 0x7f01b800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0cca10 con 0x7f01b8005190 2023-10-30T14:00:14.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b800c040 con 0x7f01b8003980 2023-10-30T14:00:14.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b800c200 con 0x7f01b80047d0 2023-10-30T14:00:14.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d1fb4640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b8005190 0x7f01b800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d17b3640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8003980 0x7f01b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d0fb2640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01b80047d0 0x7f01b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d1fb4640 1 -- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01b80047d0 msgr2=0x7f01b8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:14.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d1fb4640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01b80047d0 0x7f01b8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d1fb4640 1 -- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8003980 msgr2=0x7f01b8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:14.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d1fb4640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8003980 0x7f01b8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d1fb4640 1 -- 172.21.15.31:0/998327895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8003530 con 0x7f01b8005190 2023-10-30T14:00:14.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d17b3640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8003980 0x7f01b8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:14.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.735+0000 7f01d1fb4640 1 -- 172.21.15.31:0/998327895 reap_dead start 2023-10-30T14:00:14.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d1fb4640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b8005190 0x7f01b800b990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0c9f50 tx=0x7f01c401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f38eb4666d9da07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c40a5c50 con 0x7f01b8005190 2023-10-30T14:00:14.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f01c400f040 con 0x7f01b8005190 2023-10-30T14:00:14.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01c3fff640 1 -- 172.21.15.31:0/998327895 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c40ad440 con 0x7f01b8005190 2023-10-30T14:00:14.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- 172.21.15.31:0/998327895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b8005190 msgr2=0x7f01b800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b8005190 0x7f01b800b990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0c9f50 tx=0x7f01c401cc40 comp rx=0 tx=0).stop 2023-10-30T14:00:14.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- 172.21.15.31:0/998327895 shutdown_connections 2023-10-30T14:00:14.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 --2- 172.21.15.31:0/998327895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b8005190 0x7f01b800b990 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- 172.21.15.31:0/998327895 >> 172.21.15.31:0/998327895 conn(0x7f01cc06ab00 msgr2=0x7f01cc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:14.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- 172.21.15.31:0/998327895 shutdown_connections 2023-10-30T14:00:14.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- 172.21.15.31:0/998327895 wait complete. 2023-10-30T14:00:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 Processor -- start 2023-10-30T14:00:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- start start 2023-10-30T14:00:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8005190 0x7f01cc149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc151e30 0x7f01cc1eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc159360 0x7f01cc0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1ee650 con 0x7f01cc151e30 2023-10-30T14:00:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc157620 con 0x7f01b8005190 2023-10-30T14:00:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.739+0000 7f01d27b5640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1ee340 con 0x7f01cc159360 2023-10-30T14:00:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc151e30 0x7f01cc1eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc151e30 0x7f01cc1eea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43454/0 (socket says 172.21.15.31:43454) 2023-10-30T14:00:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 -- 172.21.15.31:0/3844460747 learned_addr learned my addr 172.21.15.31:0/3844460747 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d17b3640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8005190 0x7f01cc149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d1fb4640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc159360 0x7f01cc0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 -- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc159360 msgr2=0x7f01cc0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc159360 0x7f01cc0b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 -- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8005190 msgr2=0x7f01cc149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8005190 0x7f01cc149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 -- 172.21.15.31:0/3844460747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc0b4080 con 0x7f01cc151e30 2023-10-30T14:00:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d17b3640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8005190 0x7f01cc149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d0fb2640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc151e30 0x7f01cc1eea70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00e4d0 tx=0x7f01bc00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01c1ffb640 1 -- 172.21.15.31:0/3844460747 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0052e0 con 0x7f01cc151e30 2023-10-30T14:00:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01c1ffb640 1 -- 172.21.15.31:0/3844460747 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f01bc005440 con 0x7f01cc151e30 2023-10-30T14:00:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01cc0b42b0 con 0x7f01cc151e30 2023-10-30T14:00:14.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01c1ffb640 1 -- 172.21.15.31:0/3844460747 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc003840 con 0x7f01cc151e30 2023-10-30T14:00:14.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc152650 con 0x7f01cc151e30 2023-10-30T14:00:14.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc1ee790 con 0x7f01cc151e30 2023-10-30T14:00:14.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01c1ffb640 1 -- 172.21.15.31:0/3844460747 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01bc0039a0 con 0x7f01cc151e30 2023-10-30T14:00:14.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.743+0000 7f01c1ffb640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:14.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.747+0000 7f01c1ffb640 1 -- 172.21.15.31:0/3844460747 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f01bc084220 con 0x7f01cc151e30 2023-10-30T14:00:14.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.747+0000 7f01d17b3640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:14.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.747+0000 7f01d17b3640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f01c80019e0 tx=0x7f01c8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:14.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.747+0000 7f01c1ffb640 1 -- 172.21.15.31:0/3844460747 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01cc1ee790 con 0x7f01cc151e30 2023-10-30T14:00:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:14.943+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f01cc1ee790 con 0x7f01cc151e30 2023-10-30T14:00:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.559+0000 7f01c1ffb640 1 -- 172.21.15.31:0/3844460747 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v97) v1 ==== 104+0+0 (secure 0 0 0) 0x7f01cc1ee790 con 0x7f01cc151e30 2023-10-30T14:00:15.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 msgr2=0x7f01b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f01c80019e0 tx=0x7f01c8009040 comp rx=0 tx=0).stop 2023-10-30T14:00:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc151e30 msgr2=0x7f01cc1eea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc151e30 0x7f01cc1eea70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00e4d0 tx=0x7f01bc00e990 comp rx=0 tx=0).stop 2023-10-30T14:00:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 shutdown_connections 2023-10-30T14:00:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f01b00664a0 0x7f01b0068960 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f01cc159360 0x7f01cc0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc151e30 0x7f01cc1eea70 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 --2- 172.21.15.31:0/3844460747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01b8005190 0x7f01cc149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 >> 172.21.15.31:0/3844460747 conn(0x7f01cc06ab00 msgr2=0x7f01cc0cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:15.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 shutdown_connections 2023-10-30T14:00:15.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.563+0000 7f01d27b5640 1 -- 172.21.15.31:0/3844460747 wait complete. 2023-10-30T14:00:15.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-30T14:00:15.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd96a5b8640 1 Processor -- start 2023-10-30T14:00:15.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd96a5b8640 1 -- start start 2023-10-30T14:00:15.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd96a5b8640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd96406fb20 0x7fd9640c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:15.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd96a5b8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd964153bd0 0x7fd9640c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:15.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd96a5b8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96415ded0 con 0x7fd96406ff00 2023-10-30T14:00:15.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd96a5b8640 1 -- --> v2:172.21.15.33:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96406ee00 con 0x7fd96406fb20 2023-10-30T14:00:15.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd96a5b8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96406b130 con 0x7fd964153bd0 2023-10-30T14:00:15.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd963fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd96406fb20 0x7fd9640c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:15.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd968b55640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd964153bd0 0x7fd9640c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:15.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd963fff640 1 --2- >> v2:172.21.15.33:3300/0 conn(0x7fd96406fb20 0x7fd9640c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.31:37860/0 (socket says 172.21.15.31:37860) 2023-10-30T14:00:15.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.735+0000 7fd963fff640 1 -- 172.21.15.31:0/457351510 learned_addr learned my addr 172.21.15.31:0/457351510 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:15.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd968b55640 1 -- 172.21.15.31:0/457351510 >> v2:172.21.15.33:3300/0 conn(0x7fd96406fb20 msgr2=0x7fd9640c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:15.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd968b55640 1 --2- 172.21.15.31:0/457351510 >> v2:172.21.15.33:3300/0 conn(0x7fd96406fb20 0x7fd9640c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd968b55640 1 -- 172.21.15.31:0/457351510 >> v1:172.21.15.33:6789/0 conn(0x7fd96406ff00 legacy=0x7fd9641534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:15.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd968b55640 1 -- 172.21.15.31:0/457351510 --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96406cff0 con 0x7fd964153bd0 2023-10-30T14:00:15.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd963fff640 1 --2- 172.21.15.31:0/457351510 >> v2:172.21.15.33:3300/0 conn(0x7fd96406fb20 0x7fd9640c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:00:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd968b55640 1 --2- 172.21.15.31:0/457351510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd964153bd0 0x7fd9640c20f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009d90 tx=0x7fd95c01c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c26be2f3691102 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd962ffd640 1 -- 172.21.15.31:0/457351510 <== mon.2 v2:172.21.15.33:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0a5780 con 0x7fd964153bd0 2023-10-30T14:00:15.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd962ffd640 1 -- 172.21.15.31:0/457351510 <== mon.2 v2:172.21.15.33:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd95c0a58e0 con 0x7fd964153bd0 2023-10-30T14:00:15.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.739+0000 7fd962ffd640 1 -- 172.21.15.31:0/457351510 <== mon.2 v2:172.21.15.33:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0a5c90 con 0x7fd964153bd0 2023-10-30T14:00:15.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- 172.21.15.31:0/457351510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd964153bd0 msgr2=0x7fd9640c20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:15.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/457351510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd964153bd0 0x7fd9640c20f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009d90 tx=0x7fd95c01c8c0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- 172.21.15.31:0/457351510 shutdown_connections 2023-10-30T14:00:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/457351510 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd964153bd0 0x7fd9640c20f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/457351510 >> v2:172.21.15.33:3300/0 conn(0x7fd96406fb20 0x7fd9640c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- 172.21.15.31:0/457351510 >> 172.21.15.31:0/457351510 conn(0x7fd9640cd010 msgr2=0x7fd9640cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- 172.21.15.31:0/457351510 shutdown_connections 2023-10-30T14:00:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- 172.21.15.31:0/457351510 wait complete. 2023-10-30T14:00:15.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 Processor -- start 2023-10-30T14:00:15.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- start start 2023-10-30T14:00:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd96406fb20 0x7fd9641523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd96406ff00 0x7fd9641f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd964153bd0 0x7fd9641f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9641f6420 con 0x7fd96406ff00 2023-10-30T14:00:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9641f55c0 con 0x7fd964153bd0 2023-10-30T14:00:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd96a5b8640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9641f58a0 con 0x7fd96406fb20 2023-10-30T14:00:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd96406ff00 0x7fd9641f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd96406ff00 0x7fd9641f2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43456/0 (socket says 172.21.15.31:43456) 2023-10-30T14:00:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd9637fe640 1 -- 172.21.15.31:0/3299185632 learned_addr learned my addr 172.21.15.31:0/3299185632 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd963fff640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd96406fb20 0x7fd9641523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.743+0000 7fd968b55640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd964153bd0 0x7fd9641f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd9637fe640 1 -- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd96406fb20 msgr2=0x7fd9641523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd9637fe640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd96406fb20 0x7fd9641523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd9637fe640 1 -- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd964153bd0 msgr2=0x7fd9641f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd9637fe640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd964153bd0 0x7fd9641f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:15.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd9637fe640 1 -- 172.21.15.31:0/3299185632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540094d0 con 0x7fd96406ff00 2023-10-30T14:00:15.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd9637fe640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd96406ff00 0x7fd9641f2350 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd95800c430 tx=0x7fd95800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:15.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd960ff9640 1 -- 172.21.15.31:0/3299185632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9580052d0 con 0x7fd96406ff00 2023-10-30T14:00:15.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd960ff9640 1 -- 172.21.15.31:0/3299185632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd958005430 con 0x7fd96406ff00 2023-10-30T14:00:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c009a70 con 0x7fd96406ff00 2023-10-30T14:00:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd960ff9640 1 -- 172.21.15.31:0/3299185632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958007280 con 0x7fd96406ff00 2023-10-30T14:00:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9641f5e60 con 0x7fd96406ff00 2023-10-30T14:00:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96404e170 con 0x7fd96406ff00 2023-10-30T14:00:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd960ff9640 1 -- 172.21.15.31:0/3299185632 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9580073e0 con 0x7fd96406ff00 2023-10-30T14:00:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd960ff9640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9500664a0 0x7fd950068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.747+0000 7fd960ff9640 1 -- 172.21.15.31:0/3299185632 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fd9580837c0 con 0x7fd96406ff00 2023-10-30T14:00:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.751+0000 7fd963fff640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9500664a0 0x7fd950068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.751+0000 7fd963fff640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9500664a0 0x7fd950068960 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd954006cb0 tx=0x7fd954009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.751+0000 7fd960ff9640 1 -- 172.21.15.31:0/3299185632 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd958050720 con 0x7fd96406ff00 2023-10-30T14:00:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:15.911+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd964150b70 con 0x7fd96406ff00 2023-10-30T14:00:16.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.575+0000 7fd960ff9640 1 -- 172.21.15.31:0/3299185632 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd958053d40 con 0x7fd96406ff00 2023-10-30T14:00:16.577 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2023-10-30T14:00:16.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9500664a0 msgr2=0x7fd950068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9500664a0 0x7fd950068960 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd954006cb0 tx=0x7fd954009240 comp rx=0 tx=0).stop 2023-10-30T14:00:16.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd96406ff00 msgr2=0x7fd9641f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd96406ff00 0x7fd9641f2350 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd95800c430 tx=0x7fd95800c8f0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 shutdown_connections 2023-10-30T14:00:16.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7fd9500664a0 0x7fd950068960 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd964153bd0 0x7fd9641f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd96406ff00 0x7fd9641f2350 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 --2- 172.21.15.31:0/3299185632 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7fd96406fb20 0x7fd9641523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 >> 172.21.15.31:0/3299185632 conn(0x7fd9640cd010 msgr2=0x7fd96406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:16.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 shutdown_connections 2023-10-30T14:00:16.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.579+0000 7fd96a5b8640 1 -- 172.21.15.31:0/3299185632 wait complete. 2023-10-30T14:00:16.625 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-30T14:00:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d6186b640 1 Processor -- start 2023-10-30T14:00:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d6186b640 1 -- start start 2023-10-30T14:00:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d6186b640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8d5c14e470 0x7f8d5c06e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d6186b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c0c8a00 0x7f8d5c0c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d6186b640 1 -- --> v1:172.21.15.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c1580a0 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d6186b640 1 -- --> v2:172.21.15.31:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c157e90 con 0x7f8d5c14e470 2023-10-30T14:00:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d6186b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c06ed90 con 0x7f8d5c0c8a00 2023-10-30T14:00:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d5affd640 1 --2- >> v2:172.21.15.31:3300/0 conn(0x7f8d5c14e470 0x7f8d5c06e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d53fff640 1 --1- >> v1:172.21.15.31:6789/0 conn(0x7f8d5c0c1eb0 0x7f8d5c157780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.31:6789/0 says I am v1:172.21.15.31:51140/0 (socket says 172.21.15.31:51140) 2023-10-30T14:00:16.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.775+0000 7f8d53fff640 1 -- 172.21.15.31:0/4108935069 learned_addr learned my addr 172.21.15.31:0/4108935069 (peer_addr_for_me v1:172.21.15.31:0/0) 2023-10-30T14:00:16.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5b7fe640 1 --2- 172.21.15.31:0/4108935069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c0c8a00 0x7f8d5c0c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5affd640 1 -- 172.21.15.31:0/4108935069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c0c8a00 msgr2=0x7f8d5c0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:16.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5affd640 1 --2- 172.21.15.31:0/4108935069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c0c8a00 0x7f8d5c0c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5affd640 1 -- 172.21.15.31:0/4108935069 >> v1:172.21.15.31:6789/0 conn(0x7f8d5c0c1eb0 legacy=0x7f8d5c157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5a7fc640 1 -- 172.21.15.31:0/4108935069 <== mon.0 v1:172.21.15.31:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110195169 0 0) 0x7f8d5c1580a0 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5affd640 1 -- 172.21.15.31:0/4108935069 --> v2:172.21.15.31:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c158380 con 0x7f8d5c14e470 2023-10-30T14:00:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5b7fe640 1 --2- 172.21.15.31:0/4108935069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c0c8a00 0x7f8d5c0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:16.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5affd640 1 --2- 172.21.15.31:0/4108935069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c14e470 0x7f8d5c06e850 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0097c0 tx=0x7f8d4c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d8f4191da10f0b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5a7fc640 1 -- 172.21.15.31:0/4108935069 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c0ad660 con 0x7f8d5c14e470 2023-10-30T14:00:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5a7fc640 1 -- 172.21.15.31:0/4108935069 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8d4c0ad7c0 con 0x7f8d5c14e470 2023-10-30T14:00:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.779+0000 7f8d5a7fc640 1 -- 172.21.15.31:0/4108935069 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c0adb70 con 0x7f8d5c14e470 2023-10-30T14:00:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- 172.21.15.31:0/4108935069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c14e470 msgr2=0x7f8d5c06e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 --2- 172.21.15.31:0/4108935069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c14e470 0x7f8d5c06e850 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0097c0 tx=0x7f8d4c01c450 comp rx=0 tx=0).stop 2023-10-30T14:00:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- 172.21.15.31:0/4108935069 shutdown_connections 2023-10-30T14:00:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 --2- 172.21.15.31:0/4108935069 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c0c8a00 0x7f8d5c0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 --2- 172.21.15.31:0/4108935069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c14e470 0x7f8d5c06e850 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- 172.21.15.31:0/4108935069 >> 172.21.15.31:0/4108935069 conn(0x7f8d5c0cd010 msgr2=0x7f8d5c0cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:16.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- 172.21.15.31:0/4108935069 shutdown_connections 2023-10-30T14:00:16.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- 172.21.15.31:0/4108935069 wait complete. 2023-10-30T14:00:16.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 Processor -- start 2023-10-30T14:00:16.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- start start 2023-10-30T14:00:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c0c1eb0 0x7f8d5c0cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d5c0c8a00 0x7f8d5c1540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 --2- >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c14e470 0x7f8d5c154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1522e0 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1f0d50 con 0x7f8d5c0c8a00 2023-10-30T14:00:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d6186b640 1 -- --> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c14d6d0 con 0x7f8d5c14e470 2023-10-30T14:00:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c0c1eb0 0x7f8d5c0cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c0c1eb0 0x7f8d5c0cb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43472/0 (socket says 172.21.15.31:43472) 2023-10-30T14:00:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 -- 172.21.15.31:0/384411257 learned_addr learned my addr 172.21.15.31:0/384411257 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-10-30T14:00:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5b7fe640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c14e470 0x7f8d5c154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d53fff640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d5c0c8a00 0x7f8d5c1540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 -- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c14e470 msgr2=0x7f8d5c154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c14e470 0x7f8d5c154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 -- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d5c0c8a00 msgr2=0x7f8d5c1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d5c0c8a00 0x7f8d5c1540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 -- 172.21.15.31:0/384411257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c009420 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d53fff640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d5c0c8a00 0x7f8d5c1540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:00:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.783+0000 7f8d5affd640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c0c1eb0 0x7f8d5c0cb050 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c01a080 tx=0x7f8d4c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:16.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d58ff9640 1 -- 172.21.15.31:0/384411257 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c0b6610 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d58ff9640 1 -- 172.21.15.31:0/384411257 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8d4c0b6770 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d58ff9640 1 -- 172.21.15.31:0/384411257 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c0b6b40 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c1f0140 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c1f0700 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c04e170 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d58ff9640 1 -- 172.21.15.31:0/384411257 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d4c0b4040 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d58ff9640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d3c0664d0 0x7f8d3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d58ff9640 1 -- 172.21.15.31:0/384411257 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6448+0+0 (secure 0 0 0) 0x7f8d4c11e7a0 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.787+0000 7f8d53fff640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d3c0664d0 0x7f8d3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.791+0000 7f8d53fff640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d3c0664d0 0x7f8d3c068990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0c7da0 tx=0x7f8d44009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:16.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.791+0000 7f8d58ff9640 1 -- 172.21.15.31:0/384411257 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d4c0eb8e0 con 0x7f8d5c0c1eb0 2023-10-30T14:00:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:16.959+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8d5c0c7420 con 0x7f8d5c0c1eb0 2023-10-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d58ff9640 1 -- 172.21.15.31:0/384411257 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8d4c0b20d0 con 0x7f8d5c0c1eb0 2023-10-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2023-10-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d3c0664d0 msgr2=0x7f8d3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d3c0664d0 0x7f8d3c068990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0c7da0 tx=0x7f8d44009040 comp rx=0 tx=0).stop 2023-10-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c0c1eb0 msgr2=0x7f8d5c0cb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c0c1eb0 0x7f8d5c0cb050 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c01a080 tx=0x7f8d4c0191f0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 shutdown_connections 2023-10-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:6832/2139658642,v1:172.21.15.33:6833/2139658642] conn(0x7f8d3c0664d0 0x7f8d3c068990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3301/0,v1:172.21.15.33:6790/0] conn(0x7f8d5c14e470 0x7f8d5c154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d5c0c8a00 0x7f8d5c1540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 --2- 172.21.15.31:0/384411257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5c0c1eb0 0x7f8d5c0cb050 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 >> 172.21.15.31:0/384411257 conn(0x7f8d5c0cd010 msgr2=0x7f8d5c070050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 shutdown_connections 2023-10-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-30T14:00:17.579+0000 7f8d6186b640 1 -- 172.21.15.31:0/384411257 wait complete. 2023-10-30T14:00:17.617 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T14:00:17.617 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:00:17.618 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.636 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:00:17.637 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:00:17.637 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.697 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:00:17.697 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:00:17.698 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.752 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:00:17.753 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T14:00:17.753 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.765 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T14:00:17.766 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T14:00:17.766 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.822 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T14:00:17.822 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:00:17.823 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.877 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:00:17.878 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T14:00:17.878 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T14:00:17.878 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.964 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T14:00:17.965 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T14:00:17.965 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:21.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-30T14:00:24.067 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T14:00:24.067 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T14:00:24.068 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:24.147 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T14:00:24.148 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T14:00:24.148 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:27.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-10-30T14:00:27.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-30T14:00:29.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:29.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:30.250 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T14:00:30.251 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T14:00:30.252 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:30.319 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T14:00:30.319 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T14:00:30.320 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:33.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-30T14:00:33.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-10-30T14:00:36.421 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T14:00:36.422 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T14:00:36.422 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:36.502 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T14:00:36.503 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T14:00:36.503 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:36.592 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T14:00:36.592 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T14:00:36.593 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T14:00:36.593 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:36.650 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T14:00:36.650 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T14:00:36.650 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:36.680 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T14:00:36.680 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T14:00:36.681 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T14:00:36.681 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:36.702 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T14:00:36.703 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T14:00:36.703 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:36.735 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T14:00:36.735 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T14:00:36.735 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:36.796 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T14:00:36.797 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T14:00:36.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-30T14:00:36.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T14:00:36.842 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T14:00:37.161 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T14:00:37.161 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T14:00:37.375 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T14:00:37.375 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T14:00:37.699 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T14:00:37.700 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T14:00:37.887 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T14:00:37.887 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T14:00:38.151 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T14:00:38.151 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T14:00:38.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-30T14:00:38.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T14:00:38.334 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T14:00:38.516 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T14:00:38.516 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T14:00:38.688 INFO:tasks.ceph:Archiving mon data... 2023-10-30T14:00:38.690 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/data/mon.a.tgz 2023-10-30T14:00:38.690 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-10-30T14:00:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.VRiaOxP7Vw 2023-10-30T14:00:38.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VRiaOxP7Vw 2023-10-30T14:00:38.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.VRiaOxP7Vw 2023-10-30T14:00:38.914 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.VRiaOxP7Vw is 352KB 2023-10-30T14:00:38.978 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.VRiaOxP7Vw 2023-10-30T14:00:38.987 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/data/mon.b.tgz 2023-10-30T14:00:38.988 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-10-30T14:00:38.994 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.V8sVCxeKtg 2023-10-30T14:00:38.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.V8sVCxeKtg 2023-10-30T14:00:39.120 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.V8sVCxeKtg 2023-10-30T14:00:39.183 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.V8sVCxeKtg is 357KB 2023-10-30T14:00:39.200 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.V8sVCxeKtg 2023-10-30T14:00:39.206 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/data/mon.c.tgz 2023-10-30T14:00:39.206 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-10-30T14:00:39.254 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.jy84IOj8tO 2023-10-30T14:00:39.255 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.jy84IOj8tO 2023-10-30T14:00:39.388 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.jy84IOj8tO 2023-10-30T14:00:39.451 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.jy84IOj8tO is 357KB 2023-10-30T14:00:39.468 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.jy84IOj8tO 2023-10-30T14:00:39.474 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T14:00:39.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T14:00:39.478 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T14:00:39.532 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T14:00:39.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T14:00:39.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T14:00:39.578 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T14:00:39.625 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T14:00:39.629 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T14:00:39.631 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/remote/smithi031/crash 2023-10-30T14:00:39.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T14:00:39.654 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/remote/smithi033/crash 2023-10-30T14:00:39.655 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T14:00:39.666 INFO:tasks.ceph:Shutting down logrotate 2023-10-30T14:00:39.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:39.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:39.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:39.779 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:39.807 INFO:tasks.ceph:Compressing logs... 2023-10-30T14:00:39.807 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T14:00:39.827 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T14:00:39.844 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54659.log 2023-10-30T14:00:39.845 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51153.log 2023-10-30T14:00:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51904.log 2023-10-30T14:00:39.846 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54659.log.gz 2023-10-30T14:00:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48942.log 2023-10-30T14:00:39.847 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51153.log.gz 2023-10-30T14:00:39.848 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48887.log 2023-10-30T14:00:39.848 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51904.log.gz 2023-10-30T14:00:39.849 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45517.log 2023-10-30T14:00:39.849 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48942.log.gz 2023-10-30T14:00:39.850 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54277.log 2023-10-30T14:00:39.850 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48887.log.gz 2023-10-30T14:00:39.851 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54945.log 2023-10-30T14:00:39.852 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45517.log.gz 2023-10-30T14:00:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49549.log 2023-10-30T14:00:39.853 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54277.log.gz 2023-10-30T14:00:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49997.log 2023-10-30T14:00:39.854 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54945.log.gz 2023-10-30T14:00:39.855 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44198.log 2023-10-30T14:00:39.855 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49549.log.gz 2023-10-30T14:00:39.855 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55614.log 2023-10-30T14:00:39.856 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49997.log.gz 2023-10-30T14:00:39.856 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49471.log 2023-10-30T14:00:39.857 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44198.log.gz 2023-10-30T14:00:39.857 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53875.log 2023-10-30T14:00:39.857 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55614.log.gz 2023-10-30T14:00:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53528.log 2023-10-30T14:00:39.858 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49471.log.gz 2023-10-30T14:00:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53875.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T14:00:39.859 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53875.log.gz 2023-10-30T14:00:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53796.log 2023-10-30T14:00:39.860 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53528.log.gz 2023-10-30T14:00:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53822.log 2023-10-30T14:00:39.862 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43983.log 2023-10-30T14:00:39.862 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53796.log.gz 2023-10-30T14:00:39.863 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45387.log 2023-10-30T14:00:39.863 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53822.log.gz 2023-10-30T14:00:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44722.log 2023-10-30T14:00:39.864 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43983.log.gz 2023-10-30T14:00:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45387.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45387.log.gz 2023-10-30T14:00:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53986.log 2023-10-30T14:00:39.866 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54383.log 2023-10-30T14:00:39.867 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44722.log.gz 2023-10-30T14:00:39.867 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53986.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53986.log.gz 2023-10-30T14:00:39.868 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50958.log 2023-10-30T14:00:39.868 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54383.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54383.log.gz 2023-10-30T14:00:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54225.log 2023-10-30T14:00:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50958.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50958.log.gz 2023-10-30T14:00:39.869 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45725.log 2023-10-30T14:00:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54225.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54225.log.gz 2023-10-30T14:00:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43978.log 2023-10-30T14:00:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45725.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45725.log.gz 2023-10-30T14:00:39.872 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44384.log 2023-10-30T14:00:39.872 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43978.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43978.log.gz 2023-10-30T14:00:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54303.log 2023-10-30T14:00:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44384.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44384.log.gz 2023-10-30T14:00:39.874 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50075.log 2023-10-30T14:00:39.875 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54303.log: 95.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T14:00:39.875 INFO:teuthology.orchestra.run.smithi031.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54303.log.gz 2023-10-30T14:00:39.875 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54012.log 2023-10-30T14:00:39.876 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47353.log 2023-10-30T14:00:39.876 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50075.log.gz 2023-10-30T14:00:39.877 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54012.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54012.log.gz 2023-10-30T14:00:39.877 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T14:00:39.878 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47353.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47353.log.gz 2023-10-30T14:00:39.878 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49124.log 2023-10-30T14:00:39.879 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43890.log 2023-10-30T14:00:39.879 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38764.log 2023-10-30T14:00:39.880 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49124.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49124.log.gz 2023-10-30T14:00:39.880 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50171.log 2023-10-30T14:00:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51045.log 2023-10-30T14:00:39.882 INFO:teuthology.orchestra.run.smithi031.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43890.log.gz 2023-10-30T14:00:39.883 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50171.log: gzip -5 88.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.50171.log.gz -- 2023-10-30T14:00:39.883 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.51179.log 2023-10-30T14:00:39.884 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38884.log 2023-10-30T14:00:39.884 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38740.log 2023-10-30T14:00:39.885 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38884.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T14:00:39.885 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38740.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38884.log.gz 2023-10-30T14:00:39.885 INFO:teuthology.orchestra.run.smithi033.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38764.log.gz 2023-10-30T14:00:39.885 INFO:teuthology.orchestra.run.smithi033.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38740.log.gz 2023-10-30T14:00:39.886 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T14:00:39.886 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38836.log 2023-10-30T14:00:39.886 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T14:00:39.887 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38860.log 2023-10-30T14:00:39.887 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.5.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38836.log.gz 2023-10-30T14:00:39.887 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38908.log 2023-10-30T14:00:39.887 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38812.log 2023-10-30T14:00:39.887 INFO:teuthology.orchestra.run.smithi033.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38860.log.gz 2023-10-30T14:00:39.888 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.38908.log: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T14:00:39.891 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38812.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-30T14:00:39.892 INFO:teuthology.orchestra.run.smithi033.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38908.log.gz 2023-10-30T14:00:39.892 INFO:teuthology.orchestra.run.smithi033.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38812.log.gz 2023-10-30T14:00:39.892 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-mgr.x.log: -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T14:00:39.892 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T14:00:39.893 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38788.log 2023-10-30T14:00:39.893 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T14:00:39.893 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38788.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38788.log.gz 2023-10-30T14:00:39.893 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-30T14:00:39.896 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.audit.log: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T14:00:39.896 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-30T14:00:39.896 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mds.d.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T14:00:39.897 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T14:00:39.899 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47535.log 2023-10-30T14:00:39.899 INFO:teuthology.orchestra.run.smithi031.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51045.log.gz 2023-10-30T14:00:39.899 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51179.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.51179.log.gz 2023-10-30T14:00:39.899 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47665.log 2023-10-30T14:00:39.900 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47535.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.54501.log -- replaced with /var/log/ceph/ceph-client.admin.47535.log.gz 2023-10-30T14:00:39.900 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-30T14:00:39.900 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47665.log: gzip -5 --verbose 88.8% -- -- replaced with /var/log/ceph/ceph-client.admin.47665.log.gz 2023-10-30T14:00:39.900 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.50513.log 2023-10-30T14:00:39.901 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54501.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54501.log.gz 2023-10-30T14:00:39.901 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45335.log 2023-10-30T14:00:39.901 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50513.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.50513.log.gz 2023-10-30T14:00:39.901 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54553.log 2023-10-30T14:00:39.902 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45335.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45335.log.gz 2023-10-30T14:00:39.902 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2023-10-30T14:00:39.902 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53580.log 2023-10-30T14:00:39.902 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54553.log.gz 2023-10-30T14:00:39.902 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47509.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2023-10-30T14:00:39.903 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50023.log 2023-10-30T14:00:39.903 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2023-10-30T14:00:39.903 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53580.log.gz 2023-10-30T14:00:39.904 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50023.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50023.log.gz 2023-10-30T14:00:39.904 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45673.log 2023-10-30T14:00:39.904 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50932.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz 2023-10-30T14:00:39.904 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55238.log 2023-10-30T14:00:39.905 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47483.log 2023-10-30T14:00:39.905 INFO:teuthology.orchestra.run.smithi031.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45673.log.gz 2023-10-30T14:00:39.905 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48861.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55238.log.gz 2023-10-30T14:00:39.905 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-30T14:00:39.906 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47483.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47483.log.gz 2023-10-30T14:00:39.906 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56101.log 2023-10-30T14:00:39.906 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45699.log 2023-10-30T14:00:39.906 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48861.log.gz 2023-10-30T14:00:39.907 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T14:00:39.910 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56101.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56101.log.gz 2023-10-30T14:00:39.911 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50698.log 2023-10-30T14:00:39.911 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56361.log 2023-10-30T14:00:39.911 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45699.log.gz 2023-10-30T14:00:39.911 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43303.log 2023-10-30T14:00:39.912 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50698.log.gz 2023-10-30T14:00:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56361.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56361.log.gz 2023-10-30T14:00:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44696.log 2023-10-30T14:00:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43303.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43303.log.gz -5 2023-10-30T14:00:39.913 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51337.log 2023-10-30T14:00:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49231.log 2023-10-30T14:00:39.913 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44696.log.gz 2023-10-30T14:00:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48968.log 2023-10-30T14:00:39.914 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51337.log.gz 2023-10-30T14:00:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43522.log 2023-10-30T14:00:39.914 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49231.log.gz 2023-10-30T14:00:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48968.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48968.log.gz 2023-10-30T14:00:39.915 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49390.log 2023-10-30T14:00:39.915 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55430.log 2023-10-30T14:00:39.915 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43522.log.gz 2023-10-30T14:00:39.915 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53849.log 2023-10-30T14:00:39.916 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49390.log.gz 2023-10-30T14:00:39.916 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44644.log 2023-10-30T14:00:39.916 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55430.log.gz 2023-10-30T14:00:39.919 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53849.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53849.log.gz 2023-10-30T14:00:39.919 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43996.log 2023-10-30T14:00:39.919 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44984.log 2023-10-30T14:00:39.919 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44644.log.gz 2023-10-30T14:00:39.920 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43996.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43996.log.gz -5 2023-10-30T14:00:39.920 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44184.log 2023-10-30T14:00:39.920 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54711.log 2023-10-30T14:00:39.920 INFO:teuthology.orchestra.run.smithi031.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44984.log.gz 2023-10-30T14:00:39.921 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44184.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44184.log.gz 2023-10-30T14:00:39.921 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56309.log 2023-10-30T14:00:39.921 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54711.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54711.log.gz 2023-10-30T14:00:39.921 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47247.log 2023-10-30T14:00:39.922 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49150.log 2023-10-30T14:00:39.922 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56309.log.gz 2023-10-30T14:00:39.922 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47247.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T14:00:39.922 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47247.log.gz 2023-10-30T14:00:39.924 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52699.log 2023-10-30T14:00:39.925 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49150.log.gz 2023-10-30T14:00:39.925 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49575.log 2023-10-30T14:00:39.925 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55867.log 2023-10-30T14:00:39.925 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52699.log.gz 2023-10-30T14:00:39.926 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53502.log 2023-10-30T14:00:39.926 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49575.log.gz 2023-10-30T14:00:39.926 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55867.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55867.log.gz 2023-10-30T14:00:39.926 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52168.log 2023-10-30T14:00:39.928 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56257.log 2023-10-30T14:00:39.928 INFO:teuthology.orchestra.run.smithi031.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53502.log.gz 2023-10-30T14:00:39.928 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52168.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52168.log.gz 2023-10-30T14:00:39.929 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48747.log 2023-10-30T14:00:39.929 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49204.log 2023-10-30T14:00:39.929 INFO:teuthology.orchestra.run.smithi031.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56257.log.gz 2023-10-30T14:00:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48747.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48747.log.gz 2023-10-30T14:00:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44826.log 2023-10-30T14:00:39.931 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49204.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49204.log.gz 2023-10-30T14:00:39.931 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52998.log 2023-10-30T14:00:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44826.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44826.log.gz 2023-10-30T14:00:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52460.log 2023-10-30T14:00:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52998.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52998.log.gz 2023-10-30T14:00:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55378.log 2023-10-30T14:00:39.934 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52460.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52460.log.gz 2023-10-30T14:00:39.934 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T14:00:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55378.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55378.log.gz 2023-10-30T14:00:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44396.log 2023-10-30T14:00:39.936 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56127.log 2023-10-30T14:00:39.937 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44396.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44396.log.gz 2023-10-30T14:00:39.937 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44904.log 2023-10-30T14:00:39.938 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56127.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56127.log.gz 2023-10-30T14:00:39.939 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52116.log 2023-10-30T14:00:39.940 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46391.log 2023-10-30T14:00:39.940 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44904.log.gz 2023-10-30T14:00:39.941 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52116.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52116.log.gz 2023-10-30T14:00:39.941 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50356.log 2023-10-30T14:00:39.942 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46292.log 2023-10-30T14:00:39.943 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46391.log.gz 2023-10-30T14:00:39.943 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50356.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50356.log.gz 2023-10-30T14:00:39.944 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50854.log 2023-10-30T14:00:39.945 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.46292.log: --verbose -- /var/log/ceph/ceph-client.admin.49338.log 2023-10-30T14:00:39.945 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46292.log.gz 2023-10-30T14:00:39.946 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50854.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50854.log.gz 2023-10-30T14:00:39.946 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46038.log 2023-10-30T14:00:39.947 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44930.log 2023-10-30T14:00:39.948 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49338.log.gz 2023-10-30T14:00:39.948 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46038.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46038.log.gz 2023-10-30T14:00:39.949 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55893.log 2023-10-30T14:00:39.949 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49176.log 2023-10-30T14:00:39.950 INFO:teuthology.orchestra.run.smithi031.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44930.log.gz 2023-10-30T14:00:39.950 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55893.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55893.log.gz 2023-10-30T14:00:39.951 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56387.log 2023-10-30T14:00:39.952 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48773.log 2023-10-30T14:00:39.952 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49176.log.gz 2023-10-30T14:00:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56387.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56387.log.gz 2023-10-30T14:00:39.953 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43656.log 2023-10-30T14:00:39.954 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52246.log 2023-10-30T14:00:39.955 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48773.log.gz 2023-10-30T14:00:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43656.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43656.log.gz 2023-10-30T14:00:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54038.log 2023-10-30T14:00:39.957 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48669.log 2023-10-30T14:00:39.957 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52246.log.gz 2023-10-30T14:00:39.958 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54038.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54038.log.gz 2023-10-30T14:00:39.958 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49707.log 2023-10-30T14:00:39.959 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52673.log 2023-10-30T14:00:39.960 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48669.log.gz 2023-10-30T14:00:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49707.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49707.log.gz 2023-10-30T14:00:39.961 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45979.log 2023-10-30T14:00:39.962 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43942.log 2023-10-30T14:00:39.962 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52673.log.gz 2023-10-30T14:00:39.963 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45979.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45979.log.gz 2023-10-30T14:00:39.963 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56231.log 2023-10-30T14:00:39.964 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53475.log 2023-10-30T14:00:39.965 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43942.log.gz 2023-10-30T14:00:39.965 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56231.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56231.log.gz 2023-10-30T14:00:39.966 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52918.log 2023-10-30T14:00:39.966 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52541.log 2023-10-30T14:00:39.967 INFO:teuthology.orchestra.run.smithi031.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53475.log.gz 2023-10-30T14:00:39.968 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52918.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52918.log.gz 2023-10-30T14:00:39.968 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43630.log 2023-10-30T14:00:39.969 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45621.log 2023-10-30T14:00:39.969 INFO:teuthology.orchestra.run.smithi031.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52541.log.gz 2023-10-30T14:00:39.970 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43630.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43630.log.gz 2023-10-30T14:00:39.970 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53632.log 2023-10-30T14:00:39.971 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49601.log 2023-10-30T14:00:39.972 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45621.log.gz 2023-10-30T14:00:39.973 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53632.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53632.log.gz 2023-10-30T14:00:39.973 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2023-10-30T14:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55484.log 2023-10-30T14:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49601.log.gz 2023-10-30T14:00:39.975 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44398.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2023-10-30T14:00:39.975 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53213.log 2023-10-30T14:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54685.log 2023-10-30T14:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55484.log.gz 2023-10-30T14:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53213.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53213.log.gz 2023-10-30T14:00:39.978 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48913.log 2023-10-30T14:00:39.979 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2023-10-30T14:00:39.980 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54685.log.gz 2023-10-30T14:00:39.980 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48913.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48913.log.gz 2023-10-30T14:00:39.981 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44203.log 2023-10-30T14:00:39.981 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45843.log 2023-10-30T14:00:39.982 INFO:teuthology.orchestra.run.smithi031.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2023-10-30T14:00:39.982 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44203.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44203.log.gz 2023-10-30T14:00:39.983 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45791.log 2023-10-30T14:00:39.984 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50644.log 2023-10-30T14:00:39.984 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45791.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45843.log.gz 2023-10-30T14:00:39.985 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45791.log.gz 2023-10-30T14:00:39.985 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50984.log 2023-10-30T14:00:39.986 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53927.log 2023-10-30T14:00:39.987 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50644.log.gz 2023-10-30T14:00:39.987 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50984.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50984.log.gz 2023-10-30T14:00:39.988 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56470.log 2023-10-30T14:00:39.989 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49838.log 2023-10-30T14:00:39.989 INFO:teuthology.orchestra.run.smithi031.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53927.log.gz 2023-10-30T14:00:39.990 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56470.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56470.log.gz 2023-10-30T14:00:39.990 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50224.log 2023-10-30T14:00:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56153.log 2023-10-30T14:00:39.992 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49838.log.gz 2023-10-30T14:00:39.993 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50224.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50224.log.gz 2023-10-30T14:00:39.993 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53130.log 2023-10-30T14:00:39.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49416.log 2023-10-30T14:00:39.994 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56153.log.gz 2023-10-30T14:00:39.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53130.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53130.log.gz 2023-10-30T14:00:39.995 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55716.log 2023-10-30T14:00:39.996 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53395.log 2023-10-30T14:00:39.997 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49416.log.gz 2023-10-30T14:00:39.997 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55716.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55716.log.gz 2023-10-30T14:00:39.998 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45288.log 2023-10-30T14:00:39.999 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43993.log 2023-10-30T14:00:40.000 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53395.log.gz 2023-10-30T14:00:40.000 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.45288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45288.log.gz 2023-10-30T14:00:40.000 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51231.log 2023-10-30T14:00:40.001 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43993.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43993.log.gz 2023-10-30T14:00:40.002 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55456.log 2023-10-30T14:00:40.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51231.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51231.log.gz 2023-10-30T14:00:40.004 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54841.log 2023-10-30T14:00:40.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47273.log 2023-10-30T14:00:40.005 INFO:teuthology.orchestra.run.smithi031.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55456.log.gz 2023-10-30T14:00:40.007 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53658.log 2023-10-30T14:00:40.007 INFO:teuthology.orchestra.run.smithi031.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54841.log.gz 2023-10-30T14:00:40.008 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49098.log 2023-10-30T14:00:40.009 INFO:teuthology.orchestra.run.smithi031.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47273.log.gz 2023-10-30T14:00:40.009 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53658.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53658.log.gz 2023-10-30T14:00:40.010 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51283.log 2023-10-30T14:00:40.011 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45361.log 2023-10-30T14:00:40.011 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49098.log.gz 2023-10-30T14:00:40.012 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51283.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51283.log.gz 2023-10-30T14:00:40.013 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51878.log 2023-10-30T14:00:40.014 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45361.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45361.log.gz 2023-10-30T14:00:40.014 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52759.log 2023-10-30T14:00:40.015 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51878.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51878.log.gz 2023-10-30T14:00:40.016 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54067.log 2023-10-30T14:00:40.017 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52759.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52759.log.gz 2023-10-30T14:00:40.018 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46501.log 2023-10-30T14:00:40.018 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43838.log 2023-10-30T14:00:40.019 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54067.log.gz 2023-10-30T14:00:40.020 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46501.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46501.log.gz 2023-10-30T14:00:40.020 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52142.log 2023-10-30T14:00:40.021 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43838.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43838.log.gz 2023-10-30T14:00:40.022 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52220.log 2023-10-30T14:00:40.022 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52142.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52142.log.gz 2023-10-30T14:00:40.023 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T14:00:40.024 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46120.log 2023-10-30T14:00:40.025 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52220.log.gz 2023-10-30T14:00:40.026 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49444.log 2023-10-30T14:00:40.027 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44878.log 2023-10-30T14:00:40.027 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46120.log.gz 2023-10-30T14:00:40.029 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49890.log 2023-10-30T14:00:40.029 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49444.log.gz 2023-10-30T14:00:40.030 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55768.log 2023-10-30T14:00:40.030 INFO:teuthology.orchestra.run.smithi031.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44878.log.gz 2023-10-30T14:00:40.032 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54579.log 2023-10-30T14:00:40.032 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49890.log.gz 2023-10-30T14:00:40.033 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49760.log 2023-10-30T14:00:40.033 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55768.log.gz 2023-10-30T14:00:40.034 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55186.log 2023-10-30T14:00:40.035 INFO:teuthology.orchestra.run.smithi031.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54579.log.gz 2023-10-30T14:00:40.036 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43916.log 2023-10-30T14:00:40.036 INFO:teuthology.orchestra.run.smithi031.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49760.log.gz 2023-10-30T14:00:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55186.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.52647.log 2023-10-30T14:00:40.037 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55186.log.gz 2023-10-30T14:00:40.039 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50592.log 2023-10-30T14:00:40.039 INFO:teuthology.orchestra.run.smithi031.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43916.log.gz 2023-10-30T14:00:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49864.log 2023-10-30T14:00:40.040 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52647.log.gz 2023-10-30T14:00:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45439.log 2023-10-30T14:00:40.042 INFO:teuthology.orchestra.run.smithi031.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50592.log.gz 2023-10-30T14:00:40.043 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48721.log 2023-10-30T14:00:40.043 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49864.log.gz 2023-10-30T14:00:40.045 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45439.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.55510.log -- replaced with /var/log/ceph/ceph-client.admin.45439.log.gz 2023-10-30T14:00:40.045 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-30T14:00:40.046 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43267.log 2023-10-30T14:00:40.046 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48721.log.gz 2023-10-30T14:00:40.048 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52506.log 2023-10-30T14:00:40.048 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55510.log.gz 2023-10-30T14:00:40.049 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56049.log 2023-10-30T14:00:40.049 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43267.log.gz 2023-10-30T14:00:40.051 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44852.log 2023-10-30T14:00:40.051 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52506.log.gz 2023-10-30T14:00:40.052 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53449.log 2023-10-30T14:00:40.052 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56049.log.gz 2023-10-30T14:00:40.053 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54893.log 2023-10-30T14:00:40.053 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44852.log.gz 2023-10-30T14:00:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52090.log 2023-10-30T14:00:40.055 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53449.log.gz 2023-10-30T14:00:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44188.log 2023-10-30T14:00:40.057 INFO:teuthology.orchestra.run.smithi031.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54893.log.gz 2023-10-30T14:00:40.058 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55271.log 2023-10-30T14:00:40.058 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52090.log.gz 2023-10-30T14:00:40.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52970.log 2023-10-30T14:00:40.059 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44188.log.gz 2023-10-30T14:00:40.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44774.log 2023-10-30T14:00:40.061 INFO:teuthology.orchestra.run.smithi031.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55271.log.gz 2023-10-30T14:00:40.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45751.log 2023-10-30T14:00:40.062 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52970.log.gz 2023-10-30T14:00:40.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46324.log 2023-10-30T14:00:40.064 INFO:teuthology.orchestra.run.smithi031.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44774.log.gz 2023-10-30T14:00:40.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55562.log 2023-10-30T14:00:40.065 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45751.log.gz 2023-10-30T14:00:40.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55667.log 2023-10-30T14:00:40.067 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46324.log.gz 2023-10-30T14:00:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55536.log 2023-10-30T14:00:40.068 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55562.log.gz 2023-10-30T14:00:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52036.log 2023-10-30T14:00:40.069 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55667.log.gz 2023-10-30T14:00:40.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55536.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55536.log.gz 2023-10-30T14:00:40.071 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50539.log 2023-10-30T14:00:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53024.log 2023-10-30T14:00:40.073 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52036.log.gz 2023-10-30T14:00:40.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53317.log 2023-10-30T14:00:40.074 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50539.log.gz 2023-10-30T14:00:40.075 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49681.log 2023-10-30T14:00:40.076 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53024.log.gz 2023-10-30T14:00:40.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48994.log 2023-10-30T14:00:40.077 INFO:teuthology.orchestra.run.smithi031.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53317.log.gz 2023-10-30T14:00:40.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53606.log 2023-10-30T14:00:40.079 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49681.log.gz 2023-10-30T14:00:40.079 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54867.log 2023-10-30T14:00:40.080 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48994.log.gz 2023-10-30T14:00:40.081 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53606.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.53606.log.gz 2023-10-30T14:00:40.081 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T14:00:40.083 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53369.log 2023-10-30T14:00:40.083 INFO:teuthology.orchestra.run.smithi031.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54867.log.gz 2023-10-30T14:00:40.084 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50618.log 2023-10-30T14:00:40.086 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43708.log 2023-10-30T14:00:40.086 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53369.log.gz 2023-10-30T14:00:40.087 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54449.log 2023-10-30T14:00:40.087 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50618.log.gz 2023-10-30T14:00:40.089 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43708.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43708.log.gz 2023-10-30T14:00:40.089 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44618.log 2023-10-30T14:00:40.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54449.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54449.log.gz 2023-10-30T14:00:40.090 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52300.log 2023-10-30T14:00:40.092 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49072.log 2023-10-30T14:00:40.092 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44618.log.gz 2023-10-30T14:00:40.093 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2023-10-30T14:00:40.094 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52300.log.gz 2023-10-30T14:00:40.095 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49072.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49072.log.gz 2023-10-30T14:00:40.095 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49046.log 2023-10-30T14:00:40.097 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44202.log 2023-10-30T14:00:40.097 INFO:teuthology.orchestra.run.smithi031.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2023-10-30T14:00:40.098 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49046.log: gzip -5 89.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49046.log.gz /var/log/ceph/ceph-client.admin.49523.log 2023-10-30T14:00:40.098 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-30T14:00:40.100 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37975.log 2023-10-30T14:00:40.100 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44202.log.gz 2023-10-30T14:00:40.101 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49786.log 88.9% 2023-10-30T14:00:40.101 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49523.log.gz 2023-10-30T14:00:40.103 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50802.log 2023-10-30T14:00:40.103 INFO:teuthology.orchestra.run.smithi031.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.37975.log.gz 2023-10-30T14:00:40.104 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54973.log 2023-10-30T14:00:40.104 INFO:teuthology.orchestra.run.smithi031.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49786.log.gz 2023-10-30T14:00:40.105 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50802.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50802.log.gz 2023-10-30T14:00:40.106 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54973.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54973.log.gz 2023-10-30T14:00:40.107 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43576.log 2023-10-30T14:00:40.109 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54475.log 2023-10-30T14:00:40.110 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43576.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43576.log.gz 2023-10-30T14:00:40.111 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53343.log 2023-10-30T14:00:40.111 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54475.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54475.log.gz 2023-10-30T14:00:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43812.log 2023-10-30T14:00:40.113 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53343.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53343.log.gz 2023-10-30T14:00:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50408.log 2023-10-30T14:00:40.115 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49627.log 2023-10-30T14:00:40.115 INFO:teuthology.orchestra.run.smithi031.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43812.log.gz 2023-10-30T14:00:40.117 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51071.log 2023-10-30T14:00:40.117 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50408.log.gz 2023-10-30T14:00:40.118 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49627.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49627.log.gz 2023-10-30T14:00:40.119 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51071.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51071.log.gz 2023-10-30T14:00:40.120 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44190.log 2023-10-30T14:00:40.121 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54633.log 2023-10-30T14:00:40.123 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55794.log 2023-10-30T14:00:40.123 INFO:teuthology.orchestra.run.smithi031.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44190.log.gz 2023-10-30T14:00:40.125 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54633.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54633.log.gz 2023-10-30T14:00:40.126 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55794.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55794.log.gz 2023-10-30T14:00:40.126 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44592.log 2023-10-30T14:00:40.128 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33122.log 2023-10-30T14:00:40.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44592.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44592.log.gz 2023-10-30T14:00:40.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.33122.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33122.log.gz 2023-10-30T14:00:40.131 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55742.log 2023-10-30T14:00:40.132 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51984.log 2023-10-30T14:00:40.134 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45110.log 2023-10-30T14:00:40.134 INFO:teuthology.orchestra.run.smithi031.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55742.log.gz 2023-10-30T14:00:40.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54605.log 2023-10-30T14:00:40.136 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51984.log.gz 2023-10-30T14:00:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53423.log 2023-10-30T14:00:40.137 INFO:teuthology.orchestra.run.smithi031.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.45110.log.gz 2023-10-30T14:00:40.138 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54605.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54605.log.gz 2023-10-30T14:00:40.139 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53423.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53423.log.gz 2023-10-30T14:00:40.140 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52405.log 2023-10-30T14:00:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44410.log 2023-10-30T14:00:40.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55352.log 88.6% 2023-10-30T14:00:40.143 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52405.log.gz 2023-10-30T14:00:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44410.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44410.log.gz 2023-10-30T14:00:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56624.log 2023-10-30T14:00:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55352.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55352.log.gz 2023-10-30T14:00:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51205.log 2023-10-30T14:00:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43995.log 2023-10-30T14:00:40.148 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56624.log.gz 2023-10-30T14:00:40.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51205.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51205.log.gz 2023-10-30T14:00:40.151 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43995.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43995.log.gz 2023-10-30T14:00:40.151 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52272.log 2023-10-30T14:00:40.153 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47457.log 2023-10-30T14:00:40.154 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44956.log 2023-10-30T14:00:40.155 INFO:teuthology.orchestra.run.smithi031.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52272.log.gz 2023-10-30T14:00:40.156 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45595.log 2023-10-30T14:00:40.156 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47457.log.gz 2023-10-30T14:00:40.157 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44956.log: gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.52064.log 2023-10-30T14:00:40.158 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44956.log.gz 2023-10-30T14:00:40.159 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50776.log 2023-10-30T14:00:40.159 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45595.log.gz 2023-10-30T14:00:40.160 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52064.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52064.log.gz 2023-10-30T14:00:40.161 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49286.log 2023-10-30T14:00:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50776.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50776.log.gz 2023-10-30T14:00:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53050.log 2023-10-30T14:00:40.164 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43682.log 2023-10-30T14:00:40.164 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49286.log.gz 2023-10-30T14:00:40.165 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53050.log.gz 2023-10-30T14:00:40.165 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54527.log 2023-10-30T14:00:40.166 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43682.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43682.log.gz 2023-10-30T14:00:40.167 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51309.log 2023-10-30T14:00:40.169 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54527.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54527.log.gzgzip 2023-10-30T14:00:40.169 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54789.log 2023-10-30T14:00:40.170 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53104.log 2023-10-30T14:00:40.170 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51309.log.gz 2023-10-30T14:00:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54789.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 87.9% 2023-10-30T14:00:40.172 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54789.log.gz 2023-10-30T14:00:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53104.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53104.log.gz -5 2023-10-30T14:00:40.173 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49734.log 2023-10-30T14:00:40.174 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50101.log 2023-10-30T14:00:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49734.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49734.log.gz 2023-10-30T14:00:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50101.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50101.log.gz 2023-10-30T14:00:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53291.log 2023-10-30T14:00:40.179 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T14:00:40.179 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T14:00:40.180 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48800.log 2023-10-30T14:00:40.181 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53291.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53291.log.gz 2023-10-30T14:00:40.181 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52431.log 2023-10-30T14:00:40.183 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56572.log 2023-10-30T14:00:40.183 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48800.log.gz 2023-10-30T14:00:40.184 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52431.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52431.log.gz 2023-10-30T14:00:40.185 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56572.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56572.log.gz 2023-10-30T14:00:40.186 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45817.log 2023-10-30T14:00:40.187 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53161.log 2023-10-30T14:00:40.189 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45279.log 2023-10-30T14:00:40.189 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45817.log.gz 2023-10-30T14:00:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53161.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53161.log.gz 2023-10-30T14:00:40.191 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.45279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45279.log.gz 2023-10-30T14:00:40.192 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50724.log 2023-10-30T14:00:40.193 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53239.log 2023-10-30T14:00:40.195 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45569.log 2023-10-30T14:00:40.195 INFO:teuthology.orchestra.run.smithi031.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50724.log.gz 2023-10-30T14:00:40.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53239.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53239.log.gz 2023-10-30T14:00:40.198 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45569.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45569.log.gz 2023-10-30T14:00:40.198 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56179.log 2023-10-30T14:00:40.199 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43980.log 2023-10-30T14:00:40.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55029.log 2023-10-30T14:00:40.202 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56179.log.gz 2023-10-30T14:00:40.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43980.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43980.log.gz 2023-10-30T14:00:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55029.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55029.log.gz 2023-10-30T14:00:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51100.log 2023-10-30T14:00:40.206 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56520.log 2023-10-30T14:00:40.208 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52866.log 2023-10-30T14:00:40.208 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51100.log.gz 2023-10-30T14:00:40.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56520.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56520.log.gz 2023-10-30T14:00:40.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52866.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52866.log.gz 2023-10-30T14:00:40.211 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47299.log 2023-10-30T14:00:40.213 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49943.log 2023-10-30T14:00:40.214 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47299.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47299.log.gz 2023-10-30T14:00:40.214 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2023-10-30T14:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49943.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49943.log.gz 2023-10-30T14:00:40.216 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55997.log 2023-10-30T14:00:40.217 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54119.log 2023-10-30T14:00:40.218 INFO:teuthology.orchestra.run.smithi031.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2023-10-30T14:00:40.219 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55945.log 89.1% 2023-10-30T14:00:40.219 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55997.log.gz 2023-10-30T14:00:40.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54119.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54119.log.gz 2023-10-30T14:00:40.221 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51127.log 2023-10-30T14:00:40.221 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55945.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55945.log.gz 2023-10-30T14:00:40.222 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44670.log 2023-10-30T14:00:40.224 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49812.log 2023-10-30T14:00:40.224 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51127.log.gz 2023-10-30T14:00:40.225 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56335.log 2023-10-30T14:00:40.226 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44670.log.gz 2023-10-30T14:00:40.227 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55919.log 2023-10-30T14:00:40.227 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49812.log.gz 2023-10-30T14:00:40.228 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44402.log 2023-10-30T14:00:40.229 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56335.log.gz 2023-10-30T14:00:40.230 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56442.log 2023-10-30T14:00:40.230 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55919.log.gz 2023-10-30T14:00:40.231 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45413.log 2023-10-30T14:00:40.232 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44402.log.gz 2023-10-30T14:00:40.233 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45465.log 2023-10-30T14:00:40.233 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56442.log.gz 2023-10-30T14:00:40.234 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45413.log: gzip -5 87.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45413.log.gz /var/log/ceph/ceph-client.admin.44748.log 2023-10-30T14:00:40.234 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-30T14:00:40.236 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46067.log 2023-10-30T14:00:40.236 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45465.log.gz 2023-10-30T14:00:40.237 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47587.log 2023-10-30T14:00:40.237 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44748.log.gz 2023-10-30T14:00:40.239 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52726.log 2023-10-30T14:00:40.239 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46067.log.gz 2023-10-30T14:00:40.240 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49917.log 2023-10-30T14:00:40.240 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47587.log.gz 2023-10-30T14:00:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55079.log 2023-10-30T14:00:40.242 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52726.log.gz 2023-10-30T14:00:40.243 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49917.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49917.log.gz 2023-10-30T14:00:40.244 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55079.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55079.log.gz 2023-10-30T14:00:40.245 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44404.log 2023-10-30T14:00:40.246 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53901.log 2023-10-30T14:00:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51930.log 2023-10-30T14:00:40.248 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44404.log.gz 2023-10-30T14:00:40.249 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53901.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53901.log.gz 2023-10-30T14:00:40.250 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51930.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51930.log.gz 2023-10-30T14:00:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45925.log 2023-10-30T14:00:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45647.log 2023-10-30T14:00:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51257.log 2023-10-30T14:00:40.254 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45925.log.gz 2023-10-30T14:00:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45647.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45647.log.gz 2023-10-30T14:00:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51257.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51257.log.gz 2023-10-30T14:00:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54173.log 2023-10-30T14:00:40.259 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38087.log 2023-10-30T14:00:40.261 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53554.log 2023-10-30T14:00:40.261 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54173.log.gz 2023-10-30T14:00:40.262 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55588.log 2023-10-30T14:00:40.262 INFO:teuthology.orchestra.run.smithi031.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38087.log.gz 2023-10-30T14:00:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54199.log 2023-10-30T14:00:40.264 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53554.log.gz 2023-10-30T14:00:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54737.log 2023-10-30T14:00:40.265 INFO:teuthology.orchestra.run.smithi031.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55588.log.gz 2023-10-30T14:00:40.267 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45010.log 2023-10-30T14:00:40.267 INFO:teuthology.orchestra.run.smithi031.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54199.log.gz 2023-10-30T14:00:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46012.log 2023-10-30T14:00:40.268 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54737.log.gz 2023-10-30T14:00:40.270 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55323.log 2023-10-30T14:00:40.270 INFO:teuthology.orchestra.run.smithi031.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45010.log.gz 2023-10-30T14:00:40.271 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46012.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46012.log.gz 2023-10-30T14:00:40.272 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44204.log 2023-10-30T14:00:40.273 INFO:teuthology.orchestra.run.smithi031.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55323.log.gz 2023-10-30T14:00:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43444.log 2023-10-30T14:00:40.276 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54999.log 2023-10-30T14:00:40.276 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44204.log.gz 2023-10-30T14:00:40.277 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45951.log 2023-10-30T14:00:40.278 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43444.log.gz 2023-10-30T14:00:40.278 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54999.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54999.log.gz 2023-10-30T14:00:40.280 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45951.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45951.log.gz 2023-10-30T14:00:40.280 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48643.log 2023-10-30T14:00:40.282 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50276.log 2023-10-30T14:00:40.283 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50302.log 2023-10-30T14:00:40.283 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48643.log.gz 2023-10-30T14:00:40.285 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53265.log 2023-10-30T14:00:40.285 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50276.log.gz 2023-10-30T14:00:40.286 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50302.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50302.log.gz 2023-10-30T14:00:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53265.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53265.log.gz 2023-10-30T14:00:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52567.log 2023-10-30T14:00:40.289 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56415.log 2023-10-30T14:00:40.291 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53187.log 2023-10-30T14:00:40.291 INFO:teuthology.orchestra.run.smithi031.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52567.log.gz 2023-10-30T14:00:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56415.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T14:00:40.293 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56415.log.gz 2023-10-30T14:00:40.294 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53187.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-30T14:00:40.294 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53187.log.gz 2023-10-30T14:00:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55159.log 2023-10-30T14:00:40.298 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mds.c.log: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T14:00:40.298 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52811.log 2023-10-30T14:00:40.301 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55159.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55159.log.gz 2023-10-30T14:00:40.301 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52811.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52811.log.gz 2023-10-30T14:00:40.302 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.53689.log 2023-10-30T14:00:40.303 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54329.log 2023-10-30T14:00:40.303 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53689.log: 94.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-30T14:00:40.304 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53689.log.gz 2023-10-30T14:00:40.304 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50906.log 2023-10-30T14:00:40.306 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52892.log 2023-10-30T14:00:40.306 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54329.log.gz 2023-10-30T14:00:40.307 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46199.log 2023-10-30T14:00:40.307 INFO:teuthology.orchestra.run.smithi031.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50906.log.gz 2023-10-30T14:00:40.309 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52892.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52892.log.gz 2023-10-30T14:00:40.310 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46199.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46199.log.gz 2023-10-30T14:00:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-30T14:00:40.312 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44196.log 2023-10-30T14:00:40.314 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2023-10-30T14:00:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44196.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44196.log.gz 2023-10-30T14:00:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52326.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2023-10-30T14:00:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50461.log 2023-10-30T14:00:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43496.log 2023-10-30T14:00:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50461.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50461.log.gz 2023-10-30T14:00:40.321 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56494.log 2023-10-30T14:00:40.322 INFO:teuthology.orchestra.run.smithi031.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43496.log.gz 2023-10-30T14:00:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48828.log 2023-10-30T14:00:40.324 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56494.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56494.log.gz 2023-10-30T14:00:40.325 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48828.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48828.log.gz 2023-10-30T14:00:40.326 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43989.log 2023-10-30T14:00:40.327 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56695.log 2023-10-30T14:00:40.329 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43989.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43989.log.gz 2023-10-30T14:00:40.330 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56695.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56695.log.gz 2023-10-30T14:00:40.331 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53770.log 2023-10-30T14:00:40.332 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50127.log 2023-10-30T14:00:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53770.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53770.log.gz 2023-10-30T14:00:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50127.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50127.log.gz 2023-10-30T14:00:40.336 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53076.log 2023-10-30T14:00:40.338 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46094.log 2023-10-30T14:00:40.339 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53076.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53076.log.gz 2023-10-30T14:00:40.340 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46094.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46094.log.gz 2023-10-30T14:00:40.340 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50828.log 2023-10-30T14:00:40.342 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45869.log 2023-10-30T14:00:40.344 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50828.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50828.log.gz 2023-10-30T14:00:40.345 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45869.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45869.log.gz 2023-10-30T14:00:40.345 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53744.log 2023-10-30T14:00:40.347 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45491.log 2023-10-30T14:00:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53744.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53744.log.gz 2023-10-30T14:00:40.350 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45491.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2023-10-30T14:00:40.350 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45899.log 2023-10-30T14:00:40.352 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55105.log 2023-10-30T14:00:40.353 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45899.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45899.log.gz 2023-10-30T14:00:40.355 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55105.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55105.log.gz 2023-10-30T14:00:40.355 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56598.log 2023-10-30T14:00:40.357 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47561.log 2023-10-30T14:00:40.358 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56598.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56598.log.gz 2023-10-30T14:00:40.360 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47561.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47561.log.gz 2023-10-30T14:00:40.360 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49497.log 2023-10-30T14:00:40.362 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44411.log 2023-10-30T14:00:40.363 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49497.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49497.log.gz 2023-10-30T14:00:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52785.log 2023-10-30T14:00:40.365 INFO:teuthology.orchestra.run.smithi031.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44411.log.gz 2023-10-30T14:00:40.366 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43760.log 2023-10-30T14:00:40.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52785.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52785.log.gz 2023-10-30T14:00:40.369 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43760.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43760.log.gz 2023-10-30T14:00:40.369 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50670.log 2023-10-30T14:00:40.371 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53953.log 2023-10-30T14:00:40.372 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50670.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50670.log.gz 2023-10-30T14:00:40.374 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53953.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53953.log.gz 2023-10-30T14:00:40.374 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52194.log 2023-10-30T14:00:40.376 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52944.log 2023-10-30T14:00:40.377 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52194.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52194.log.gz 2023-10-30T14:00:40.378 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52944.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52944.log.gz 2023-10-30T14:00:40.379 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45308.log 2023-10-30T14:00:40.380 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54919.log 2023-10-30T14:00:40.382 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45308.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45308.log.gz 2023-10-30T14:00:40.383 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54919.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54919.log.gz 2023-10-30T14:00:40.384 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49655.log 2023-10-30T14:00:40.385 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43412.log 2023-10-30T14:00:40.387 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49655.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49655.log.gz 2023-10-30T14:00:40.388 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43412.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43412.log.gz 2023-10-30T14:00:40.389 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56023.log 2023-10-30T14:00:40.390 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56283.log 2023-10-30T14:00:40.392 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56023.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56023.log.gz 2023-10-30T14:00:40.393 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56283.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56283.log.gz 2023-10-30T14:00:40.393 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52619.log 2023-10-30T14:00:40.395 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47693.log 2023-10-30T14:00:40.396 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52619.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52619.log.gz 2023-10-30T14:00:40.398 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47693.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47693.log.gz 2023-10-30T14:00:40.398 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44412.log 2023-10-30T14:00:40.400 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56669.log 2023-10-30T14:00:40.401 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44412.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44412.log.gz 2023-10-30T14:00:40.403 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56669.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56669.log.gz 2023-10-30T14:00:40.403 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52352.log 2023-10-30T14:00:40.405 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49020.log 2023-10-30T14:00:40.406 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52352.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52352.log.gz 2023-10-30T14:00:40.408 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49020.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49020.log.gz 2023-10-30T14:00:40.408 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44800.log 2023-10-30T14:00:40.411 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44800.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44800.log.gz 2023-10-30T14:00:40.411 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55212.log 2023-10-30T14:00:40.413 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54147.log 2023-10-30T14:00:40.414 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55212.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55212.log.gz 2023-10-30T14:00:40.416 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54147.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54147.log.gz 2023-10-30T14:00:40.416 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2023-10-30T14:00:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56546.log 2023-10-30T14:00:40.419 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49260.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2023-10-30T14:00:40.420 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56546.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56546.log.gz 2023-10-30T14:00:40.421 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50566.log 2023-10-30T14:00:40.423 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52010.log 2023-10-30T14:00:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50566.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50566.log.gz 2023-10-30T14:00:40.425 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52010.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52010.log.gz 2023-10-30T14:00:40.426 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52379.log 2023-10-30T14:00:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46172.log 2023-10-30T14:00:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52379.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52379.log.gz 2023-10-30T14:00:40.430 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46172.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46172.log.gz 2023-10-30T14:00:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47327.log 2023-10-30T14:00:40.433 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45060.log 2023-10-30T14:00:40.434 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47327.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47327.log.gz 2023-10-30T14:00:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45060.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45060.log.gz 2023-10-30T14:00:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48695.log 2023-10-30T14:00:40.438 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47613.log 2023-10-30T14:00:40.440 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48695.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48695.log.gz 2023-10-30T14:00:40.440 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50328.log 2023-10-30T14:00:40.443 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47613.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47613.log.gz 2023-10-30T14:00:40.443 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55404.log 2023-10-30T14:00:40.443 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50328.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50328.log.gz 2023-10-30T14:00:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56205.log 2023-10-30T14:00:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55404.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55404.log.gz 2023-10-30T14:00:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2023-10-30T14:00:40.448 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56205.log.gz 2023-10-30T14:00:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45165.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2023-10-30T14:00:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50382.log 2023-10-30T14:00:40.452 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55133.log 2023-10-30T14:00:40.454 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50382.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50382.log.gz 2023-10-30T14:00:40.455 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55133.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55133.log.gz 2023-10-30T14:00:40.455 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47639.log 2023-10-30T14:00:40.457 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55820.log 2023-10-30T14:00:40.459 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47639.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47639.log.gz 2023-10-30T14:00:40.460 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55820.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55820.log.gz 2023-10-30T14:00:40.460 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55640.log 2023-10-30T14:00:40.462 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49364.log 2023-10-30T14:00:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55640.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55640.log.gz 2023-10-30T14:00:40.465 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49364.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49364.log.gz 2023-10-30T14:00:40.466 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54093.log 2023-10-30T14:00:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43991.log 2023-10-30T14:00:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54093.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54093.log.gz 2023-10-30T14:00:40.470 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43991.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43991.log.gz 2023-10-30T14:00:40.471 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46146.log 2023-10-30T14:00:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54251.log 2023-10-30T14:00:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46146.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46146.log.gz 2023-10-30T14:00:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54251.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54251.log.gz 2023-10-30T14:00:40.476 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47719.log 2023-10-30T14:00:40.477 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50750.log 2023-10-30T14:00:40.479 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47719.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47719.log.gz 2023-10-30T14:00:40.480 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50750.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50750.log.gz 2023-10-30T14:00:40.481 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47405.log 2023-10-30T14:00:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43864.log 2023-10-30T14:00:40.483 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47405.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47405.log.gz 2023-10-30T14:00:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43864.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43864.log.gz 2023-10-30T14:00:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50250.log 2023-10-30T14:00:40.487 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47379.log 2023-10-30T14:00:40.488 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50250.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50250.log.gz 2023-10-30T14:00:40.489 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47379.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47379.log.gz 2023-10-30T14:00:40.490 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50434.log 2023-10-30T14:00:40.492 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43209.log 2023-10-30T14:00:40.493 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50434.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50434.log.gz 2023-10-30T14:00:40.494 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43209.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43209.log.gz 2023-10-30T14:00:40.495 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43548.log 2023-10-30T14:00:40.497 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43386.log 2023-10-30T14:00:40.498 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43548.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43548.log.gz 2023-10-30T14:00:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43386.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43386.log.gz 2023-10-30T14:00:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-30T14:00:40.502 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43786.log 2023-10-30T14:00:40.505 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.43786.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43786.log.gz 2023-10-30T14:00:40.506 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55297.log 2023-10-30T14:00:40.509 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55297.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55297.log.gz 2023-10-30T14:00:40.509 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38211.log 2023-10-30T14:00:40.510 INFO:teuthology.orchestra.run.smithi031.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-30T14:00:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38211.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38211.log.gz 2023-10-30T14:00:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54815.log 2023-10-30T14:00:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50049.log 2023-10-30T14:00:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54815.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54815.log.gz 2023-10-30T14:00:40.516 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49971.log 2023-10-30T14:00:40.517 INFO:teuthology.orchestra.run.smithi031.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50049.log.gz 2023-10-30T14:00:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55971.log 2023-10-30T14:00:40.520 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49971.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49971.log.gz 2023-10-30T14:00:40.521 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55971.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55971.log.gz 2023-10-30T14:00:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50880.log 2023-10-30T14:00:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52593.log 2023-10-30T14:00:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50880.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50880.log.gz 2023-10-30T14:00:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52593.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52593.log.gz 2023-10-30T14:00:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45543.log 2023-10-30T14:00:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43604.log 2023-10-30T14:00:40.530 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45543.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45543.log.gz 2023-10-30T14:00:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43604.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43604.log.gz 2023-10-30T14:00:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47431.log 2023-10-30T14:00:40.533 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50487.log 2023-10-30T14:00:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47431.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47431.log.gz 2023-10-30T14:00:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50487.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50487.log.gz 2023-10-30T14:00:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43734.log 2023-10-30T14:00:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54763.log 2023-10-30T14:00:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43734.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43734.log.gz 2023-10-30T14:00:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54763.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54763.log.gz 2023-10-30T14:00:40.541 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54357.log 2023-10-30T14:00:40.543 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56075.log 2023-10-30T14:00:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54357.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54357.log.gz 2023-10-30T14:00:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56075.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56075.log.gz 2023-10-30T14:00:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49312.log 2023-10-30T14:00:40.548 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50197.log 2023-10-30T14:00:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49312.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49312.log.gz 2023-10-30T14:00:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50197.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50197.log.gz 2023-10-30T14:00:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52840.log 2023-10-30T14:00:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43470.log 2023-10-30T14:00:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52840.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52840.log.gz 2023-10-30T14:00:40.555 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43470.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43470.log.gz 2023-10-30T14:00:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47749.log 2023-10-30T14:00:40.559 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47749.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47749.log.gz 2023-10-30T14:00:40.582 INFO:teuthology.orchestra.run.smithi033.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T14:00:40.613 INFO:teuthology.orchestra.run.smithi033.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T14:00:41.036 INFO:teuthology.orchestra.run.smithi031.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T14:00:42.500 INFO:teuthology.orchestra.run.smithi033.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-30T14:00:44.177 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T14:00:44.299 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T14:00:44.369 INFO:teuthology.orchestra.run.smithi033.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T14:00:44.558 INFO:teuthology.orchestra.run.smithi033.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T14:00:44.753 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T14:00:44.914 INFO:teuthology.orchestra.run.smithi033.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T14:00:44.982 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T14:00:45.632 INFO:teuthology.orchestra.run.smithi033.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T14:00:49.612 INFO:teuthology.orchestra.run.smithi033.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-30T14:01:23.667 INFO:teuthology.orchestra.run.smithi033.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-30T14:01:23.668 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-10-30T14:01:23.668 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m43.814s 2023-10-30T14:01:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:user 1m14.533s 2023-10-30T14:01:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m2.254s 2023-10-30T14:01:26.367 INFO:teuthology.orchestra.run.smithi031.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-30T14:01:26.368 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-30T14:01:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m46.538s 2023-10-30T14:01:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:user 1m1.074s 2023-10-30T14:01:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m5.620s 2023-10-30T14:01:26.370 INFO:tasks.ceph:Archiving logs... 2023-10-30T14:01:26.371 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/remote/smithi031/log 2023-10-30T14:01:26.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T14:01:32.505 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/remote/smithi033/log 2023-10-30T14:01:32.506 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T14:01:39.459 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T14:01:39.468 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T14:01:39.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T14:01:39.473 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T14:01:39.544 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T14:01:39.554 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T14:01:39.555 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:01:39.559 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:01:39.571 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:01:39.571 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-10-30T14:01:39.571 INFO:teuthology.orchestra.run.smithi033.stdout:*hv01.front.sepi 44.4.53.2 2 u 59 64 377 0.098 -4.621 1.844 2023-10-30T14:01:39.572 INFO:teuthology.orchestra.run.smithi033.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 128 377 0.112 -3.271 1.230 2023-10-30T14:01:39.572 INFO:teuthology.orchestra.run.smithi033.stdout:+hv03.front.sepi 74.6.168.72 3 u 56 128 377 0.086 -2.686 1.774 2023-10-30T14:01:39.572 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T14:01:39.574 INFO:teuthology.orchestra.run.smithi031.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:01:39.574 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================== 2023-10-30T14:01:39.574 INFO:teuthology.orchestra.run.smithi031.stdout:*hv01.front.sepi 44.4.53.2 2 u 57 64 377 0.079 -4.738 0.709 2023-10-30T14:01:39.575 INFO:teuthology.orchestra.run.smithi031.stdout:+hv02.front.sepi 63.231.80.2 3 u 52 64 377 0.088 -1.348 0.577 2023-10-30T14:01:39.575 INFO:teuthology.orchestra.run.smithi031.stdout:+hv03.front.sepi 74.6.168.72 3 u 1 64 377 0.086 -2.113 0.293 2023-10-30T14:01:39.575 INFO:teuthology.orchestra.run.smithi031.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T14:01:39.575 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T14:01:39.585 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T14:01:39.585 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T14:01:39.595 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T14:01:39.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T14:01:39.613 INFO:teuthology.task.internal:Duration was 2365.005389 seconds 2023-10-30T14:01:39.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T14:01:39.623 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T14:01:39.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T14:01:39.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T14:01:39.739 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T14:01:39.739 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-10-30T14:01:39.741 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T14:01:39.765 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2023-10-30T14:01:39.766 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T14:01:39.776 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T14:01:39.776 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T14:01:39.815 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T14:01:39.851 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T14:01:39.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T14:01:39.899 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T14:01:39.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T14:01:39.987 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T14:01:39.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T14:01:40.011 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T14:01:40.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T14:01:40.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:01:40.055 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:01:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-10-30T14:01:40.078 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2023-10-30T14:01:40.105 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:01:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:01:40.140 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:01:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:01:40.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T14:01:40.163 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T14:01:40.164 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/remote/smithi031 2023-10-30T14:01:40.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T14:01:40.209 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046/remote/smithi033 2023-10-30T14:01:40.210 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T14:01:40.229 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T14:01:40.229 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T14:01:40.251 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T14:01:40.275 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T14:01:40.287 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T14:01:40.287 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T14:01:40.296 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T14:01:40.297 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T14:01:40.303 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T14:01:40.309 INFO:teuthology.orchestra.run.smithi031.stdout: 657973 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 14:01 /home/ubuntu/cephtest 2023-10-30T14:01:40.324 INFO:teuthology.orchestra.run.smithi033.stdout: 524291 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 14:01 /home/ubuntu/cephtest 2023-10-30T14:01:40.325 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T14:01:40.335 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T14:01:40.357 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T14:01:40.387 DEBUG:teuthology.nuke:shortname: smithi031 2023-10-30T14:01:40.387 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T14:01:40.409 DEBUG:teuthology.nuke:shortname: smithi033 2023-10-30T14:01:40.410 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T14:01:40.413 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:13:36.472144', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVLU+jSgOeJeW7UG/jbVLLJdv0luZt06WSrUBK/zLu/CZnGWK3S1K+PNeK43Y3u0/bIANzRVtLcV7EpzeK5vxA='} 2023-10-30T14:01:40.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441046', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:13:36.473675', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGasMAKSbdhk/Ddh2VZPS5CPuVlwKIWRX7nVm+pw5RdOWvQBxabSh4+BgLboPQkMclKOfNZsf1BkjclIvpQzn0M='} 2023-10-30T14:01:40.460 INFO:teuthology.orchestra.console.smithi031:Power off 2023-10-30T14:01:40.461 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T14:01:40.486 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-10-30T14:01:40.487 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:01:40.501 INFO:teuthology.orchestra.console.smithi033:Power off 2023-10-30T14:01:40.501 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T14:01:40.511 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-30T14:01:40.526 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2023-10-30T14:01:40.527 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:01:40.550 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-10-30T14:01:44.513 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:01:44.551 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:01:44.623 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-30T14:01:44.661 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-10-30T14:01:48.625 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:01:48.662 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:01:48.735 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-30T14:01:48.774 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-10-30T14:01:48.837 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-10-30T14:01:52.778 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:01:52.888 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2023-10-30T14:01:52.989 INFO:teuthology.orchestra.console.smithi033:Power off completed 2023-10-30T14:01:53.096 INFO:teuthology.run:Summary data: client.0-kernel-sha1: eacf00cc51e204f401b4d5d4f84825a68f373c74 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 2365.005388736725 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi031 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=671698dbf0a642c1bda1c3ffdfdd3cdc status: fail success: false 2023-10-30T14:01:53.096 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T14:01:53.251 INFO:teuthology.run:FAIL